Notesfoundations🌱 seedling
Load Shedding
Also: Admission Control · Prioritized Load Shedding · Goodput
Load shedding is a server deliberately rejecting excess work, early and cheaply, so it can spend its capacity on requests it can actually finish. It keeps goodput (useful, timely responses) near capacity instead of letting it collapse.
Why it exists
Past capacity, an unprotected server doesn't level off: queues grow, latency passes clients' timeouts, and it ends up 100% busy producing answers nobody waits for. Once median latency reaches the client timeout, availability is already 50%. When there's more work than capacity something won't get done; shedding decides what, on purpose.
How it works
- Detect overload from concurrency, queue length or age, or CPU, not raw request rate: requests differ in cost.
- Reject fast and cheap (HTTP 503, gRPC
RESOURCE_EXHAUSTED) before doing real work. - Shed by priority: background and prefetch traffic first, user-facing and critical traffic last. Never shed the load balancer's health check.
- Drop doomed work: a request that already waited past its deadline is wasted; check before starting.
- Keep queues short, and under overload prefer serving the newest requests (LIFO), since the oldest are most likely abandoned.
- Clients can shed too: Google's adaptive throttling rejects locally with probability
max(0, (requests − K·accepts)/(requests + 1)), K = 2.
Trade-offs
| Gains | Costs |
|---|---|
| goodput stays near capacity | some requests fail outright |
| server stays alive, no crash cascade | needs priorities and good overload signals |
| fast errors let clients react | misleadingly good latency metrics while shedding |
In practice
- Netflix (2024): playback requests a user waits on get guaranteed capacity; prefetch gets the rest. During a 12× prefetch spike, user-initiated availability stayed above 99.4%.
- Google: every request carries a criticality (
CRITICAL_PLUS,CRITICAL,SHEDDABLE_PLUS,SHEDDABLE). - Uber Cinnamon: a PID controller sets the rejection rate automatically, across 768 priority levels.
Pitfalls
- Shedding health checks, so the load balancer marks the server dead and moves its load onto others.
- Never load-testing past capacity, so nobody knows whether goodput plateaus or collapses.
Connections
- Part of
- Foundations
- Contrasts with
- Backpressure
Recall
These cards are also in your review deck, scheduled with spaced repetition.