Foundations · 1 of 7

Reliability

Things will break. A reliable system keeps working anyway. What that means, where faults come from, and how good systems stop a fault from becoming a failure.

≈ 25 min

The whole idea

Press ← → to step, or Space to play and pause, whenever a diagram has focus.

What reliable means

Ask people what they expect from software and you get roughly this list. It does what you asked. It copes when you use it in ways nobody planned, including by mistake. It’s fast enough under the load it actually gets. And it doesn’t let the wrong people in.

Reliability means all of that stays true when things go wrong. That’s when, not if: disks die, code has bugs, people mistype commands. A reliable system is one that keeps working correctly anyway.

The things that go wrong are faults. A system that copes with them is fault-tolerant, or resilient. “Fault-tolerant” never means tolerant of every fault. If a black hole swallowed the Earth, you’d need servers in space. It always means tolerant of specific, named kinds of fault.

Faults vs failures

People use these two words as if they meant the same thing. They don’t, and the difference is the whole subject.

  • A fault is one component deviating from its spec: a disk returning errors, a process crashing, a network link dropping packets.
  • A failure is the system as a whole no longer providing the service its users need.

You can’t make the probability of a fault zero. What you can do is design so that faults don’t turn into failures.

Same fault, two outcomes

One serverTwo servers + load balancer
The fault is identical on both sides. Only the design differs.

A surprising consequence follows: it can make sense to raise the rate of faults on purpose, so the containment walls get tested all the time. We’ll come back to that. The main exception is security. If an attacker has read your data, no failover undoes it, so for security you prevent faults rather than tolerate them.

Hardware faults

One machine rarely breaks. Thousands of machines break all the time. At scale, hardware faults stop being events and become a steady background rate.

10,000disks in a fairly ordinary storage cluster
2–4%of hard disks fail each year in large field studies
≈ 1 / daydisks to replace, at the top of that range

Datasheets promise a hard disk a mean time to failure (MTTF) of over a century. Fleets see worse: field studies measure 2–4% of disks failing each year, an MTTF of a few decades, and Backblaze’s fleet of nearly 300,000 drives still ran at 1.57% in 2024. That sounds reassuring until you own ten thousand of them. At 3.65% a year, ten thousand disks lose about 365 a year: one disk every day. RAM, power supplies and network cables add their own steady trickle.

A month in the life of a cluster

Day 0of 30
0disks failed
day 1failures per dayday 30day 1failures per dayday 30

healthy disk failed disk · each square is one disk

10,000 disks, 30 days, at a 3.65% annual failure rate. The failures are spread unevenly, the way real ones are, but average out to about one a day.

The good news is that hardware faults are mostly random and independent. One disk dying says little about its neighbour.Mostly. A bad manufacturing batch, a hot rack or a shared power feed can correlate them. That’s what makes the classic defence work: redundancy. RAID keeps copies of data on several disks. Servers have dual power supplies and hot-swappable CPUs. Data centres have batteries and diesel generators. When one part dies, its twin takes over while the broken one is replaced.

Redundancy at the component level

RAID 1 mirroring: the server never stops, because a second copy was always there.

For a long time that was enough: a single well-built machine could run for years. Two things changed. Applications grew to need many machines, which multiplies the fault rate. And cloud platforms favour flexibility over single-machine reliability, so a virtual machine can disappear with no warning at all.

So systems moved to tolerating the loss of entire machines, in software, on top of the hardware redundancy. There’s a lovely side effect: if you can lose any one machine, you can also take one down on purpose. That means you can patch the whole fleet one node at a time with no planned downtime, which is called a rolling upgrade.

Losing a machine, on purpose

A rolling upgrade is fault tolerance used as an operations tool.

Software faults

Hardware faults are random. Software faults are systematic. The same bug lives in every copy of the code, so when its trigger arrives, it strikes everywhere at once. That makes software faults far more dangerous: they are correlated, and correlation defeats redundancy.

When every replica fails together

The leap second of June 30, 2012. Replicas don't help when they all share the bug.

The leap second is the textbook case: a kernel bug made many applications on Linux hang or spin at the same instant, and sites including Reddit had a very bad night. At Mozilla, Java services (Hadoop, HBase, Elasticsearch) pinned their CPUs at midnight UTC, and the fix, resetting the system clock, went out to every machine through Puppet that night. The other classic shapes:

  • A process that runs away with a shared resource: CPU, memory, disk space or network bandwidth.
  • A dependency that slows down, stops answering, or starts returning garbage.
  • A cascading failure: a small fault in one component triggers a fault in the next, which triggers another.

How a slow dependency takes everything down

Slow is often worse than dead: a dead service fails fast, while a slow one ties up every caller.

There’s no single fix for software faults. They usually hide behind an assumption about the environment that is true almost all the time, until one day it isn’t. What helps is a pile of small disciplines:

  • Think carefully about assumptions and about how components interact.
  • Test thoroughly.
  • Isolate processes, and let them crash and restart cleanly.
  • Measure and monitor how the system behaves in production.

A system can also check itself. A message queue should emit exactly as many messages as it took in. If it continually checks that, it can raise an alarm the moment the numbers disagree.

Human error

People design, build and run these systems, and people make mistakes, including careful, well-meaning experts. A 2003 study of three large internet services found that operator error was the leading cause of outages in two of them, and that most of those operator errors were configuration mistakes. Hardware, on servers or in the network, caused between 4% and 25% of the outages.

#1cause of outages in two of the three services: operator error
4–25%share of outages caused by hardware

One typo, two rollouts

The mistake is the same both times. A canary rollout limits how far it can spread.

Real outages look exactly like this. In February 2017, an input to a routine command was entered incorrectly and removed far more servers than intended, taking Amazon S3 in us-east-1 down for hours, and a big slice of the internet with it.In October 2021, a command meant to check backbone capacity instead disconnected all of Facebook’s data centres. By Cloudflare’s measurements Facebook was unreachable from about 15:40 to 21:28 UTC, nearly six hours. It was a human action again, and a tooling gap.

You can’t make people perfect, but you can make the system forgiving. The best setups combine all of these:

  • Make the right thing easy. Good abstractions, APIs and admin tools steer people away from mistakes. Too restrictive and people route around them, so it’s a balance.
  • Give people sandboxes. Separate the places where people make the most mistakes from the places where mistakes cause failures: non-production environments, with real data, where exploring is safe.
  • Test at every level, from unit tests to whole-system integration tests to manual checks. Automated tests reach the corner cases that normal operation rarely hits.
  • Make recovery fast. Config should roll back in one step. New code should roll out gradually, as canaries, so a bug reaches a few users rather than all of them. Keep tools that can recompute data if a computation turns out to have been wrong.
  • Monitor in detail. Performance metrics and error rates (telemetry) give early warning, and they’re the evidence you need when diagnosing.
  • Invest in people. Good training and management matter as much as any tool.

Breaking things on purpose

Many serious bugs live in error-handling code: failover paths, retries, recovery. A 2014 study of 198 user-reported failures in Cassandra, HBase, HDFS, Hadoop MapReduce and Redis found that 92% of the catastrophic ones came from mishandling an error the software had already detected. That code rarely runs, so it quietly rots. A failover that has never been exercised probably doesn’t work.

So deliberately cause faults, in production, while the team is watching. Netflix’s Chaos Monkey became famous for randomly terminating production instances in the middle of the working day, with engineers standing by. The practice is now called chaos engineering: state what “healthy” looks like as a measurable steady state, inject a real-world fault, and check whether the steady state holds, starting with a small blast radius.

Chaos engineering

If the system is truly fault-tolerant, this is a non-event. If it isn't, better to learn it at 2 p.m. than at 3 a.m.

How much reliability?

Reliability isn’t just for power stations and air-traffic control. Bugs in business software cost people’s time, and worse if they get figures wrong. Outages of a shop cost revenue and reputation. Even “unimportant” apps carry responsibility: imagine a parent who keeps every photo of their kids in your app, and the database gets corrupted.

But reliability has costs, and sometimes you trade it away on purpose: a prototype for an unproven market, or a service with razor-thin margins. That’s a legitimate choice, as long as you know you’re making it.

Technique Protects against What it costs
Component redundancy (RAID, dual PSUs) one part dying extra hardware
Multi-machine fault tolerance a whole machine or VM vanishing software complexity, more machines
Rolling upgrades planned downtime requires the row above
Timeouts, circuit breakers, load shedding slow dependencies cascading tuning; can reject requests that would have succeeded
Canary rollouts + fast rollback a bad change hitting everyone at once slower deploys, tooling
Chaos engineering recovery paths that silently don’t work controlled, real risk in production
Monitoring and telemetry not knowing what’s wrong instrumentation and storage

Pitfalls

  • “We have replicas, so we’re reliable.” Replicas protect against independent faults, like one disk or one machine. They do nothing against a bug or a bad config that hits every replica at once.
  • “Slow is better than down.” It’s usually worse: a slow dependency holds callers hostage and cascades. A timeout turns slow into a fast, contained failure.
  • “Fault-tolerant means nothing breaks.” It means named faults are contained. Write down which ones.
  • Chasing zero faults. You can’t get there. Design for containment and fast recovery instead.
  • Trusting the failover you’ve never run. Assume an untested recovery path is broken until a drill proves otherwise.

Recall

Go deeper

Keep going

The atomic notes for this lesson live in the knowledge graph:

Next on the map: Scalability (what happens as load grows) and Maintainability (keeping the system easy to work on for years).

Done with this lesson?

Mark it learned and the concepts it teaches will light up in your galaxy.

Next in FoundationsScalability