Notesfoundations🌱 seedling
Cascading Failure
Also: Cascade · Domino Failure · Death Spiral
A cascading failure is one where a failure in one component raises the load on, or starves, the components around it, so they fail too, and the failure spreads through the system faster than it can be contained.
Why it exists
Redundancy assumes survivors can absorb a failed component's work. When they can't, each failure makes the next more likely.
How it works
- Load redistribution: one server in a pool dies; its share moves to the survivors, pushing them over capacity, and they die in turn.
- Slow dependencies: callers block waiting, run out of threads, and stop answering, including health checks (Timeout).
- Resource exhaustion: CPU starvation slows every request, which raises concurrency and memory; memory pressure kills tasks or shrinks caches, which overloads backends. Garbage-collected runtimes can enter a GC death spiral.
- Restarts that don't stick: the first server back receives the entire load and falls over again; cold caches make it worse.
Trade-offs
| Defence | Cost |
|---|---|
| Load Shedding on every server | some requests fail instead of all |
| Bulkheads and cells | idle reserved capacity |
| headroom (run well below capacity) | more machines |
| gradual traffic ramp after recovery | slower recovery |
In practice
- Amazon Kinesis, Nov 2020: adding front-end capacity pushed every server past an OS thread limit (each server kept a thread per peer). The outage spread to Cognito, CloudWatch, Lambda and EventBridge; servers could be brought back only a few hundred per hour.
- The fix included bigger servers (fewer peers) and cellularizing the fleet.
Pitfalls
- Health checks that fail under load: the load balancer removes busy servers and makes the survivors busier.
- Treating the first crash as the root cause, when the real problem is that the survivors had no way to refuse the excess.
Connections
- Part of
- Foundations
- Learn first
- Fault Tolerance
Recall
These cards are also in your review deck, scheduled with spaced repetition.