Notesdistributed🌱 seedling

Partial Failure

Also: Partial Failures · Nondeterministic Failure

In a distributed system some parts can be broken while others work, unpredictably, and a node often can't tell which. One computer either works or crashes; many computers on a network fail partially and nondeterministically.

Why it exists

A single machine is built to be deterministic: when hardware is faulty it prefers a total crash (kernel panic) to a wrong answer. Connect machines with a network and that property disappears. The same operation can succeed, fail, or half-succeed, and the caller may never learn which.

How it works

  • A node knows only its local state and the messages that reach it. Whether a peer is alive, whether a write landed, what time it is: all inferences.
  • Three sources dominate: an Unreliable Network (loss, delay, reordering, Network Partitions), unreliable clocks (Clock Skew), and Process Pauses.
  • A remote call that times out is unknown, not failed: the work may have happened. Retries therefore need Idempotency.
  • The design stance: assume any component can fail at any moment, detect it by suspicion (Failure Detection), and make decisions by Quorum rather than by one node's view.

Trade-offs

Approach Gains Costs
treat failure as normal (timeouts, retries, quorums) keeps serving through faults complexity in every code path
one big reliable machine (HPC style: checkpoint, stop on fault) simple reasoning the whole job stops for any fault; no rolling upgrades

In practice

  • Jepsen analyses repeatedly find databases losing acknowledged writes under partial failures their designers didn't test: partitions, pauses, skewed clocks.
  • A 2018 study of 136 partition failures found 80% catastrophic, most reproducible with three nodes.

Pitfalls

  • Testing only clean crashes. Real faults are slow nodes, one-way links and half-failed switches.
  • Treating a timeout as proof the operation didn't happen.

Recall

These cards are also in your review deck, scheduled with spaced repetition.