Notesfoundations🌱 seedling

Latency Percentiles

Also: Percentiles · p99 · p999 · Tail Latency · Response Time Percentiles · Coordinated Omission

Response time is a distribution, so describe it with percentiles: p50 for the typical request, p95/p99/p99.9 for the tail. The mean is pulled up by outliers and describes no actual user.

Why it exists

The same request varies wildly: context switches, GC pauses, packet loss and retransmits, cache misses. An average hides how many users waited how long, and the slowest requests often belong to the most valuable users (the ones with the most data).

How it works

  • Sort response times; the pN value is the point that N% of requests beat. p50 is the median.
  • Response time = what the client sees (service time + queueing + network). Service time = active processing. Latency = time spent waiting; often used loosely for response time.
  • Measure on the client: head-of-line blocking means the server can see a fast request while the client waited behind slow ones.
  • SLOs and SLAs are written as percentiles: Amazon's Dynamo-era example was 300 ms at p99.9 at 500 req/s peak.
  • Aggregate by merging histograms (HdrHistogram, t-digest, DDSketch, Prometheus histograms), then read percentiles off the merged result.

Trade-offs

Gains Costs
shows what users actually experienced needs histograms, not a single counter
higher percentiles catch the problems that matter p99.99 is noisy and costly to optimize; diminishing returns

In practice

  • Coordinated omission: a closed-loop load tester stops sending while the server stalls, so the stall barely shows up. wrk2 sends at a constant rate and times from the intended send time.

Pitfalls

  • Averaging percentiles across servers or time windows. The mean of p99s is not the p99.
  • Reporting the mean, or a single p99 without watching how the whole distribution moves.

Recall

These cards are also in your review deck, scheduled with spaced repetition.