Notesfoundations🌱 seedling

Metastable Failure

Also: Metastable State · Congestive Collapse · Retry Storm · Sustaining Effect

A metastable failure is a bad state that a temporary trigger pushes a system into and that persists after the trigger is gone, because a feedback loop in the system (the sustaining effect, usually retries) keeps load above capacity.

Why it exists

Systems are usually run efficiently, above the load from which they could recover on their own. They're fine until a trigger arrives, and then they can't climb back.

How it works

  • Stable: load low enough to recover from a hiccup unaided.
  • Vulnerable: healthy, but above an invisible threshold. Not overloaded; just not safe against a trigger.
  • Metastable: a trigger (load spike, capacity loss, deploy) causes timeouts; a sustaining effect (retries, cache misses that can't be refilled, slow error paths) keeps load above capacity, so goodput stays near zero.
  • Example: a database fine up to 300 qps, an app retrying once after a 1 s timeout, running at 280 qps. A 10 s network blip makes retries double load to 560 qps, which sustains the timeouts. Recovery needs load below 150 qps or throttled retries.
  • Escape requires a strong push: shed load hard, disable retries, or restart with traffic held back.

Trade-offs

Defence Cost
retry budgets, jittered backoff slower recovery from genuine blips
Load Shedding and admission control some requests fail during spikes
running below the vulnerable threshold idle capacity
emergency levers (kill switches for retries) must be built and drilled in advance

In practice

  • A 2022 study of 22 metastable failures across 11 organisations found retry policy was the sustaining effect in over half, and load shedding the most common mitigation, used in over half. At least 4 of 15 major AWS outages in a decade were metastable.
  • AWS us-east-1, Oct 2025: after DynamoDB DNS was restored, EC2's lease manager kept timing out and re-queuing work, "a state of congestive collapse", until engineers throttled incoming work and restarted hosts.

Pitfalls

  • Blaming the trigger in the postmortem. The root cause is the sustaining loop; it will meet another trigger.
  • Waiting for recovery after the trigger is fixed. It won't come on its own.

Recall

These cards are also in your review deck, scheduled with spaced repetition.