Percentiles and Tail Latency

In plain words

If you time a thousand requests and sort them, the p99 is the 990th: 99% of requests were at least this fast. The p50 is the median, the middle one. The average is neither, and it hides the thing users complain about — the slow ones. The slowest 1% is called the tail, and in a big system the tail is not rare at all, because one user's page is built from dozens of requests and any one of them can be the slow one.

Percentiles, not averages

half of requests are faster — 'typical'

p50

1 in 20 is slower than this

p95

1 in 100 — what a busy user hits daily

p99

1 in 1,000 — a big customer hits it hourly

p99.9

A user who loads 100 pages a day meets your p99 every day. A customer whose script makes a million calls a day lives at your p99.9. The average describes nobody in particular.

The tail at scale

A request that fans out to N services and waits for all of them is as slow as the slowest. If each service is slow with probability p, the request meets at least one slow reply with probability 1 − (1 − p)ᴺ.

Where tails come from

• Garbage collection pauses, JIT warm-up, a page fault
• Queueing — a burst arrived just before you; see utilisation and queueing
• Cache misses — 1% of reads go to disk, and disk is 100× memory
• Noisy neighbours — another tenant on the same host is busy
• Retries and timeouts — a request that waited 2 s for a timeout, then succeeded on the retry, counts as 2 s
• The network — a lost packet is a retransmit timeout of 200 ms+

Most are not bugs. They are the normal behaviour of machines, which is why the tail cannot be fixed only by making code faster.

Cutting the tail

async function hedged<T>(call: () => Promise<T>, hedgeAfterMs = 50): Promise<T> {
  const first = call();
  const second = new Promise<T>((resolve, reject) => {
    const t = setTimeout(() => call().then(resolve, reject), hedgeAfterMs);
    first.finally(() => clearTimeout(t));       // first answered in time: never send the hedge
  });
  return Promise.race([first, second]);         // whichever comes back first
}

Where it goes wrong

• Measuring p99 per service and assuming it composes. It does not. The request's p99 is far worse than any one service's. Measure end to end.
• Hedging writes. Two copies of "charge the card" is two charges. Hedge only idempotent requests.
• Averaging percentiles. The mean of ten servers' p99s is not the fleet's p99. Compute percentiles from the raw distribution, or use a histogram that merges.
• Setting the timeout at the average. A 50 ms timeout on a 20 ms service with a 400 ms tail turns every tail event into an error. Set it from the tail you are willing to wait for.

Take this with you

• The one idea: averages hide the tail, and fan-out turns a rare tail per service into a common one per request.
• In an interview, report percentiles, do the 1 − (1 − p)ᴺ arithmetic, and name hedged requests for idempotent reads.
• At work, find your widest fan-out and measure its p99 end to end. That is the number to work on, not any single service's.