Notesfoundations🌱 seedling

Operability

Also: Operational Excellence · Toil · Making Life Easy for Operations

Operability is how easy a system makes it for the people running it to keep it healthy: routine tasks are easy and automated, unusual ones are possible, and behaviour is visible and predictable.

Why it exists

Good operations can keep imperfect software running; even excellent software fails under bad operations. Much of running a service is routine (deploys, scaling, failover, patching), and if the system makes that hard, people make mistakes (Human Error).

How it works

  • Visibility: health, metrics and traces are easy to find (Observability).
  • Automation of routine tasks, integrated with standard tools.
  • No single machine needs hand-holding: any node can be drained for maintenance.
  • Predictable behaviour: good defaults, clear docs, an obvious operating model.
  • Self-healing with an override: automation handles the normal case, a human can take control.
  • Kill toil: manual, repetitive, automatable work with no lasting value that grows linearly with the service.

Trade-offs

Gains Costs
fewer operator mistakes, faster incident response building and maintaining tooling
ops time freed for engineering automation can act surprisingly if it isn't observable

In practice

  • Google SRE caps toil at 50% of each engineer's time; the rest must go to engineering that removes toil permanently.

Pitfalls

  • Automation nobody can see into or override: it turns small surprises into big incidents.
  • Runbooks that encode manual steps forever instead of becoming code.

Recall

These cards are also in your review deck, scheduled with spaced repetition.