Notesdistributed🌱 seedling
Partial Failure
Also: Partial Failures · Nondeterministic Failure
In a distributed system some parts can be broken while others work, unpredictably, and a node often can't tell which. One computer either works or crashes; many computers on a network fail partially and nondeterministically.
Why it exists
A single machine is built to be deterministic: when hardware is faulty it prefers a total crash (kernel panic) to a wrong answer. Connect machines with a network and that property disappears. The same operation can succeed, fail, or half-succeed, and the caller may never learn which.
How it works
- A node knows only its local state and the messages that reach it. Whether a peer is alive, whether a write landed, what time it is: all inferences.
- Three sources dominate: an Unreliable Network (loss, delay, reordering, Network Partitions), unreliable clocks (Clock Skew), and Process Pauses.
- A remote call that times out is unknown, not failed: the work may have happened. Retries therefore need Idempotency.
- The design stance: assume any component can fail at any moment, detect it by suspicion (Failure Detection), and make decisions by Quorum rather than by one node's view.
Trade-offs
| Approach | Gains | Costs |
|---|---|---|
| treat failure as normal (timeouts, retries, quorums) | keeps serving through faults | complexity in every code path |
| one big reliable machine (HPC style: checkpoint, stop on fault) | simple reasoning | the whole job stops for any fault; no rolling upgrades |
In practice
- Jepsen analyses repeatedly find databases losing acknowledged writes under partial failures their designers didn't test: partitions, pauses, skewed clocks.
- A 2018 study of 136 partition failures found 80% catastrophic, most reproducible with three nodes.
Pitfalls
- Testing only clean crashes. Real faults are slow nodes, one-way links and half-failed switches.
- Treating a timeout as proof the operation didn't happen.
Connections
- Part of
- Distributed Systems
- Learn first
- Fault Tolerance
- Builds on this
- Byzantine FaultConsensusFLP ImpossibilityHappens-Before
Recall
These cards are also in your review deck, scheduled with spaced repetition.