Notesfoundations🌱 seedling
Fault Tolerance
Also: Fault-Tolerant · Resilient · Fault vs Failure
A fault is one component deviating from its spec. A failure is the whole system no longer serving users. Fault tolerance is designing so faults don't become failures.
Why it exists
You can't make the probability of a fault zero. So reliability comes from containing faults, not from preventing them all.
How it works
- Decide which specific faults you'll tolerate (e.g. "any one machine can die"). No system tolerates every fault.
- Put boundaries where faults must stop: Redundancy, health checks plus a Load Balancer, isolation, Timeouts and Circuit Breakers.
- Recover fast: automatic failover, restarts, rollbacks.
- Keep the mechanisms exercised. Tolerance code that never runs rots, which is why Chaos Engineering exists.
Trade-offs
| Gains | Costs |
|---|---|
| faults become non-events for users | more machines, more moving parts, harder testing |
In practice
- Load balancer + N servers: one server crashes and traffic routes around it.
- Rolling Upgrades: a system that tolerates losing a node can take nodes down on purpose to patch them.
Pitfalls
- Using "fault" and "failure" interchangeably. The distinction is the design goal.
- Claiming "fault-tolerant" without naming which faults.
- Redundancy against correlated faults (see Software Fault) doesn't tolerate anything.
Connections
- Part of
- Reliability
- Builds on this
- Cascading FailureHardware FaultHuman ErrorPartial FailureRolling UpgradeSoftware Fault
- Related
- Bulkhead
- In real systems
- Chaos Monkey
- Mentioned
- Load BalancerTimeoutCircuit Breaker
Recall
These cards are also in your review deck, scheduled with spaced repetition.