Notesdistributed🌱 seedling
Network Partition
Also: Netsplit · Partial Partition · Partial Network Partition · Simplex Partition
A network fault that cuts some nodes off from others while all of them keep running. To each side, the other side looks crashed, so a system must ensure at most one side keeps making decisions.
Why it exists
Switches, links, NICs and routing configs fail; one top-of-rack switch isolates a whole rack. The nodes are healthy, so they keep acting on whatever they can see.
How it works
- Complete partition: two groups, no traffic between them.
- Partial partition: A and B can't talk, but C talks to both, so nodes disagree about who is up.
- Simplex partition: traffic flows one way only (a node's heartbeats get out, requests don't get in).
- Safe handling: only a side holding a majority (Quorum) may elect a leader or commit writes; the minority side stops writing or serves stale reads explicitly. Without this you get Split Brain.
- After healing, nodes must reconcile: step down from stale leadership, replay or discard divergent writes.
Trade-offs
| During a partition | Gains | Costs |
|---|---|---|
| minority refuses writes (CP) | no divergent histories | unavailable on the minority side |
| both sides accept writes (AP) | available everywhere | conflicts to resolve later (Write Conflict) |
In practice
- Partition-failure study (OSDI 2018): 136 failures in 25 systems; 80% catastrophic (data loss most common), 88% triggerable by isolating one node, 29% caused by partial partitions, leader election the most affected mechanism.
- Cloudflare, November 2020: a half-failed switch split one etcd node from the leader while a third node reached both; repeated elections left etcd read-only for six minutes and the API impaired for 6 h 33 min.
Pitfalls
- Testing only clean, complete partitions. Partial and one-way partitions find the bugs.
- Two-node clusters: neither side can form a majority.
Connections
- Part of
- Unreliable Network
- Builds on this
- CAP Theorem
- In real systems
- Jepsen
- Mentioned
- Write Conflict
Recall
These cards are also in your review deck, scheduled with spaced repetition.