Notesfoundations🌱 seedling
Hardware Fault
Also: Hardware Failure · MTTF · Mean Time To Failure
Disks, RAM, power supplies and cables fail. Each one is rare, but across a fleet they're a steady background rate. Hardware faults are mostly random and independent, which is why redundancy works against them.
Why it exists
Datasheets quote a hard disk MTTF (mean time to failure) of 1–1.5 million hours, over a century. In the field, 2–4% of disks fail each year (large fleets in 2024 still saw about 1.5%), an MTTF of a few decades. At 3.65% a year, 10,000 disks lose about one disk per day. At scale, rare becomes routine.
How it works
- Component redundancy: RAID, dual power supplies, hot-swappable CPUs, batteries and diesel generators. When a part dies, its twin takes over.
- Machine-level tolerance: as data volumes grew and cloud VMs can vanish without warning, systems moved to tolerating the loss of whole machines in software.
- A bonus of machine-level tolerance is Rolling Upgrades: patch one node at a time with no downtime.
Trade-offs
| Gains | Costs |
|---|---|
| single-part failures become non-events | extra hardware; more machines to run |
In practice
- RAID 1 mirrors every write to two disks; reads continue from the survivor while the dead disk is swapped.
- Cloud platforms favour flexibility over single-VM reliability, so instances can disappear.
Pitfalls
- Trusting datasheet MTTF. Field failure rates run several times higher than datasheets imply.
- Assuming independence always holds. A bad batch, a hot rack or a shared power feed correlates failures.
- Relying only on hardware redundancy at fleet scale. The number of machines multiplies the fault rate.
Connections
- Part of
- Reliability
- Learn first
- Fault Tolerance
- Related
- RedundancySoftware Fault
- Mentioned
- Rolling Upgrade
Recall
These cards are also in your review deck, scheduled with spaced repetition.