Notesfoundations🌱 seedling
Little's Law
Also: Littles Law · L = λW · Queueing Theory
In a stable system, the average number of requests in flight equals arrival rate times average time in the system: L = λW. It turns throughput and latency into the concurrency (threads, connections, workers) you need.
Why it exists
Capacity planning needs a link between "requests per second", "milliseconds per request" and "how many things are happening at once". Little's Law is that link, and it holds for any stable queueing system, whatever the arrival or service distribution.
How it works
- L = λW, all long-run averages. 2,000 req/s × 0.05 s = 100 requests in flight.
- Rearranged: max throughput = concurrency limit ÷ latency. A pool of 100 connections at 50 ms caps you at 2,000 req/s.
- Utilization ρ = λ ÷ capacity. For one server with random arrivals (M/M/1), mean time in system W = S ÷ (1 − ρ): 2× service time at 50% busy, 10× at 90%, 20× at 95%.
- So latency is non-linear in load: near saturation a small rise in λ causes a large rise in W, and by Little's Law a large rise in L (queues and memory).
Trade-offs
| Gains | Costs |
|---|---|
| quick, distribution-free sizing of pools and fleets | averages only: says nothing about the tail |
| explains why running hot is dangerous | real arrivals are burstier and non-stationary |
In practice
- AWS Lambda sizes concurrency as requests per second × average duration: 5,000 req/s at 200 ms needs 1,000 concurrent executions, the default regional account limit.
- Rising p99 with a flat mean is an early warning that utilization is near the knee of the curve.
Pitfalls
- Feedback loops break the "independent" assumption: slow responses cause timeouts and retries, raising λ and pushing W up further.
- Running at 90%+ utilization to save money leaves no headroom for spikes or a lost machine.
Connections
- Part of
- Scalability
Recall
These cards are also in your review deck, scheduled with spaced repetition.