Notesfoundations🌱 seedling
Bulkhead
Also: Bulkhead Pattern · Cell-Based Architecture · Shuffle Sharding · Resource Isolation
A bulkhead partitions resources (threads, connections, servers, whole cells) so that one overloaded dependency or tenant can exhaust only its own share, not everything. It limits blast radius rather than preventing the fault.
Why it exists
Shared pools couple unrelated work. If every outbound call shares one thread pool, a single slow dependency fills it and checkout dies because recommendations are slow. The name comes from ships' hulls, divided so one breach doesn't sink the ship.
How it works
- In a process: a separate thread or connection pool, or concurrency limit, per dependency or per request class.
- In a fleet: cells, independent copies of a service, each serving a slice of customers. A bad deploy or poison request takes out one cell.
- Shuffle sharding: give each customer a random small subset of workers. Two customers rarely share the whole subset, so one customer's flood or poison request affects almost no one else.
Trade-offs
| Gains | Costs |
|---|---|
| failures stay inside one compartment | reserved capacity sits idle |
| noisy neighbours can't starve others | more pools, cells and config to run |
| smaller blast radius per change | partitions too small can't absorb normal bursts |
In practice
- Route 53: 2,048 virtual name servers, each domain assigned a shuffle shard of four; about 730 billion possible shards, and no two customer domains share more than two name servers.
- Amazon Kinesis (2020 outage remediation): accelerating cellularization of the front-end fleet so one failure can't drag down the rest.
Pitfalls
- Partitioning the servers but sharing one database or cache behind them: the bulkhead leaks.
- Sizing pools from averages: a pool that's right on average rejects every burst.
Connections
- Part of
- Foundations
Recall
These cards are also in your review deck, scheduled with spaced repetition.