NotesSystem
Chaos Monkey
Netflix's tool that randomly terminates production instances during working hours, forcing every service to tolerate instance loss.
Design choices
- Chaos Engineering: random instance termination in production, not staging. Netflix introduced it publicly in 2011 as the first of its "Simian Army".
- Business hours only: it schedules kills on weekdays inside a configured window (9:00 to 15:00 by default), so engineers are at their desks when something breaks.
- Per-group frequency: each app opts in and sets a mean time between terminations (default two days, minimum one) per app, stack or cluster. A minimum gap stops it killing the same group twice in a day.
- Opt-outs: account, region and stack exceptions keep it away from things that genuinely can't lose a node yet.
- Narrow scope: since version 2 (2016) it only terminates instances and relies on Spinnaker, Netflix's deployment platform, to find them. Latency, zone and region faults moved to other tools.
Notable trade-offs
- Real (if controlled) production risk, in exchange for recovery paths that are proven every day.
- It only tests one fault, instance loss. Slow dependencies, bad config and correlated software bugs need other experiments.
- Tied to Spinnaker, so adopting it outside that ecosystem means reimplementing it.
Connections
Recall
These cards are also in your review deck, scheduled with spaced repetition.