Notesfoundations🌱 seedling

Human Error

Also: Operator Error · Configuration Error

Operators are the leading cause of outages, mostly through configuration mistakes. You can't make humans perfect, but you can make the system forgiving: hard to get wrong, quick to undo, limited in blast radius.

Why it exists

A study of three large internet services found operator error was the leading cause of outages in two of them, mostly configuration mistakes, while hardware caused only 4–25%. Changes to a running system are where people break things.

How it works

  • Make the right thing easy: good abstractions, APIs and admin tools. Too restrictive and people work around them.
  • Sandboxes: non-production environments with real data, where mistakes are safe.
  • Test at every level: unit, integration and manual tests. Automation covers rare corner cases.
  • Fast recovery: one-step config rollback, gradual Canary Releases, and tools to recompute bad data.
  • Detailed monitoring (telemetry): early warning, and evidence for diagnosis.
  • Training and good management.
  • Blameless Postmortems: ask how the system allowed it, not who did it.

Trade-offs

Gains Costs
a mistake hits one canary, not everyone slower deploys, rollout tooling

In practice

  • Amazon S3, Feb 2017: a mistyped input to a routine command removed far more servers than intended, and us-east-1 was down for hours.
  • Facebook, Oct 2021: a command meant to assess backbone capacity disconnected every data centre; Facebook was unreachable for nearly six hours.

Pitfalls

  • Blaming individuals. That teaches people to hide mistakes instead of fixing the system.
  • Pushing changes everywhere at once: a single typo becomes a total outage.

Recall

These cards are also in your review deck, scheduled with spaced repetition.