Notesfoundations🌱 seedling

Software Fault

Also: Software Error · Systematic Fault · Correlated Failure

Software faults are systematic: the same bug is in every copy of the code, so one trigger hits every replica at once. Correlated faults defeat redundancy, which makes them more dangerous than hardware faults.

Why it exists

Bugs hide behind assumptions about the environment that are true almost always, until they aren't.

How it works

Common shapes:

  • A bug triggered by a particular input or event. The 2012 leap second (23:59:60) hit a Linux kernel bug, and many applications hung or spun at 100% CPU at the same moment.
  • A runaway process exhausting a shared resource: CPU, memory, disk or bandwidth.
  • A dependency that slows down, stops responding, or returns corrupted data.
  • A Cascading Failure: one fault triggers the next. A slow dependency ties up callers' threads until they stop answering too.

Trade-offs

Mitigation Cost
thorough testing, process isolation, crash-and-restart engineering time
Timeouts, Circuit Breakers, Load Shedding tuning; may reject requests that would have succeeded
self-checks (e.g. a queue verifies messages in = messages out) instrumentation

In practice

  • Production monitoring and measurement catch what tests miss. Systems that can check their own guarantees can alarm on a discrepancy.
  • In a study of Cassandra, HBase, HDFS, MapReduce and Redis, 92% of catastrophic failures came from mishandling errors the code had already detected. Error-handling paths deserve their own tests.

Pitfalls

  • "We have replicas" doesn't help: every replica shares the bug.
  • Slow is worse than down. Dead fails fast; slow holds callers hostage. Timeouts turn slow into fast failure.

Recall

These cards are also in your review deck, scheduled with spaced repetition.