reference / tail-latency · 5-min read
Tail Latency
Tail latency is the slow end of response times — p99 and p99.9. Why averages hide it, why fan-out amplifies it, and how to cut it.
Quick facts
| Measured as | p99 · p99.9 · max |
|---|---|
| Averages | Hide it — a few slow calls barely move the mean |
| Fan-out effect | P(slow) = 1 − 0.99^N |
| Usual causes | Queueing, GC pauses, lock contention, background work |
| Usual fixes | Hedged requests, bounded queues, load shedding |
Fan-out turns p99 into the common case
Chance a request waits on at least one p99-slow backend.
| 1 backend | 1.0% |
|---|---|
| 5 backends | 4.9% |
| 10 backends | 9.6% |
| 50 backends | 39.5% |
| 100 backends | 63.4% |
Why averages lie
Latency is skewed: most calls are fast, a few are very slow. The mean sits near the fast bulk, so a service can look healthy while 1 in 100 users waits ten times longer. Report percentiles, read from a histogram, and never average percentiles across hosts.
Measuring it honestly
Many load generators wait for a slow reply before sending the next request, so they stop sampling exactly when the system stalls. This is coordinated omission. Drive load at a fixed rate and record from the intended send time.
Cutting the tail
Most tail comes from queueing and pauses, not slow code. Keep queues short so waits can't build, shed load before saturation, move background work off the request path, and send a backup request when the first one is slow.
FAQs
Is p99 enough to watch?
Depends on volume. At 1 million requests a day, the slowest 1% is 10,000 requests. High-traffic or fan-out services usually watch p99.9 too.
What is a hedged request?
If a reply hasn't arrived by about the p95 time, send the same request to another replica and use whichever answers first. It cuts the tail for a few percent of extra load.
Should alerts use p99 or the average?
Percentiles over a time window, compared against an SLO. An average can stay flat while the tail doubles.