Notesfoundations🌱 seedling

Hardware Fault

Also: Hardware Failure · MTTF · Mean Time To Failure

Disks, RAM, power supplies and cables fail. Each one is rare, but across a fleet they're a steady background rate. Hardware faults are mostly random and independent, which is why redundancy works against them.

Why it exists

Datasheets quote a hard disk MTTF (mean time to failure) of 1–1.5 million hours, over a century. In the field, 2–4% of disks fail each year (large fleets in 2024 still saw about 1.5%), an MTTF of a few decades. At 3.65% a year, 10,000 disks lose about one disk per day. At scale, rare becomes routine.

How it works

  • Component redundancy: RAID, dual power supplies, hot-swappable CPUs, batteries and diesel generators. When a part dies, its twin takes over.
  • Machine-level tolerance: as data volumes grew and cloud VMs can vanish without warning, systems moved to tolerating the loss of whole machines in software.
  • A bonus of machine-level tolerance is Rolling Upgrades: patch one node at a time with no downtime.

Trade-offs

Gains Costs
single-part failures become non-events extra hardware; more machines to run

In practice

  • RAID 1 mirrors every write to two disks; reads continue from the survivor while the dead disk is swapped.
  • Cloud platforms favour flexibility over single-VM reliability, so instances can disappear.

Pitfalls

  • Trusting datasheet MTTF. Field failure rates run several times higher than datasheets imply.
  • Assuming independence always holds. A bad batch, a hot rack or a shared power feed correlates failures.
  • Relying only on hardware redundancy at fleet scale. The number of machines multiplies the fault rate.

Recall

These cards are also in your review deck, scheduled with spaced repetition.