Maintainability & architecture trade-offs
Most of a system's cost arrives after launch. What makes software easy to run, understand and change for years, how changes ship safely, and when to own the hardware.
The whole idea
Where the cost goes
Writing the first version of a system is the cheap part. The expensive part is everything after: fixing bugs, keeping it running, investigating failures, adapting it to new platforms and new requirements, paying down the mistakes, and adding the features nobody knew they needed on launch day. A successful system can live for decades, long after the people who wrote it have moved on.
Engineers mostly dislike that work, especially on legacy systems: code built on assumptions that are no longer true, by people who are no longer around, that is still quietly making money. But every system you build today is tomorrow’s legacy system. Maintainability is designing so the people who inherit it, including future you, have an easier time.
It rests on three principles:
- Operability: make it easy for the people running the system to keep it healthy.
- Simplicity: make it easy for new engineers to understand, by removing complexity that doesn’t need to be there.
- Evolvability: make it easy to change, as requirements inevitably do.
Those principles also decide how changes reach production, which is where most outages come from. Google’s SRE team found that roughly 70% of outages are caused by changes to a live system. So this lesson also covers how changes ship: rolling upgrades, canaries and learning from incidents.
Operability
Good operations can often keep a mediocre system running. Even the best software can’t run reliably with bad operations. Operability is about making the routine tasks easy and the unusual ones possible. A good platform lets its operators:
- See what the system is doing: metrics, logs, traces and health that are easy to find.
- Automate the routine and integrate with standard tools, without hiding what’s happening underneath.
- Avoid single points that need hand-holding: any one machine can be taken down for maintenance.
- Predict behaviour: good defaults, clear documentation, an obvious operating model (“if I do X, Y happens”).
- Override the defaults when they’re wrong, with a self-healing system that still lets a human take control.
The enemy is toil: manual, repetitive, automatable work that has no lasting value and grows linearly with the service. Google’s SRE organization aims to keep toil below 50% of each engineer’s time, so the rest goes to engineering that removes toil for good.Automation cuts both ways. A system that heals itself is great until its automation does something surprising at 3 a.m. and the operator can’t tell why. Automate the routine, and keep the automation observable and overridable.
Observability
The first thing an operator needs is to see inside. Monitoring answers questions you thought of in advance: is CPU high, is the error rate above its threshold? Observability is the ability to answer questions you didn’t think of, about a system you can only see from the outside, without shipping new code to ask them.
Dashboards vs traces
The classic description is three kinds of telemetry:
- Metrics: cheap numeric time series, good for dashboards and alerts, but pre-aggregated, so the detail is gone.
- Logs: a record of events, as detailed as you make them.
- Traces: one request’s path through every service, as a tree of timed spans. Google’s Dapper paper (2010) described the design most tracing systems still follow.
The modern push is toward wide, structured events: one record per unit of work, carrying dozens or hundreds of fields, including high-cardinality ones like user ID, build number or feature flags. Charity Majors calls this “observability 2.0”: a single source of truth from which metrics and traces can be derived, instead of three disconnected tools. You control cost by sampling events, not by throwing away the fields that make them useful.
The plumbing has standardized. OpenTelemetry, formed in 2019 by merging OpenTracing and OpenCensus, gives one set of APIs, SDKs and a wire protocol for traces, metrics and logs, and it is widely supported by vendors and open-source backends. It became a CNCF graduated project in May 2026, the second most active CNCF project after Kubernetes. Instrument once with OpenTelemetry and you can change backends without touching application code.
Simplicity and accidental complexity
Small projects can be wonderfully simple. As they grow they get complex, and complexity slows everyone down. A system mired in it is sometimes called a big ball of mud. The symptoms are familiar: tangled dependencies, inconsistent naming, hacks for performance, special cases for one customer, and modules that reach into each other’s internals.
Fred Brooks’s 1986 essay No Silver Bullet split the difficulty of software into two kinds. Essential complexity is inherent in the problem. Accidental complexity comes from our tools and choices. His argument was that no single technique would give a tenfold productivity gain within a decade, because accidental complexity was already a minority. Moseley and Marks’s 2006 paper Out of the Tar Pit disagreed: they define essential complexity strictly, as what users see as the problem, and argue that most complexity in real systems is accidental, with mutable state as its biggest source, followed by code volume and explicit control flow.
Coupling sets the blast radius
The main tool for removing accidental complexity is abstraction: a clean interface that hides an implementation behind something simpler to use. SQL hides on-disk structures, concurrency control and crash recovery. A programming language hides machine code. A good internal API hides a team’s tables. Each lets many people reuse one good implementation, and improvements to it benefit everyone.
Evolvability
Requirements never stop changing: new features, new regulations, new platforms, growth, and the things you learn about your users. Evolvability is how easily you can adapt the system to that. It follows from simplicity, since code you understand is code you can change, but some choices help directly:
- Prefer reversible decisions. Amazon’s 2015 shareholder letter describes one-way doors (consequential and hard to undo, so decide carefully) and two-way doors (reversible, so decide fast). Most decisions are two-way doors; architecture should keep it that way where it can.
- Change things in small, compatible steps. The expand/contract pattern replaces a breaking change with several safe ones: add the new column or field, write both, migrate readers, then remove the old one.
- Replace systems incrementally. The strangler fig approach routes traffic piece by piece from an old system to a new one, instead of a big-bang rewrite.
- Keep data formats evolvable. Schemas that tolerate unknown fields and new versions make every later step easier. That is the subject of the Encoding lesson.
Irreversibility makes change hard. Anything that can’t be rolled back, such as a destructive data migration or a public API you can’t version, deserves the one-way-door treatment.
Rolling upgrades
If the system tolerates losing any one machine, you can upgrade it one machine at a time with no downtime: a rolling upgrade. Kubernetes Deployments do this by default, replacing pods gradually with at most 25% unavailable and at most 25% extra at any moment, and declaring the rollout stuck if it makes no progress for 10 minutes. The catch is what happens during the rollout.
Two versions at once
A rolling upgrade means old and new code run side by side, and read each other’s data and messages. That requires:
- Backward compatibility: new code can read data written by old code.
- Forward compatibility: old code can read data written by new code, and preserves fields it doesn’t understand. This is the harder one, and the one that bites.
Kubernetes itself is upgraded this way, so it publishes a version skew policy: API servers in a cluster must be within one minor version of each other, and a kubelet may be up to three minor versions older than the API server, but never newer. Every system that rolls needs a rule like that, written down.
The same discipline makes rollback safe. If v2 writes data that v1 can’t handle, rolling back mid-incident makes the incident worse.
Canary releases
Rolling upgrades limit downtime. Canary releases limit blast radius. Google’s SRE workbook defines canarying as a partial, time-limited deployment of a change, plus an evaluation that decides whether to continue. The part that gets the change is the canary; the rest is the control.
Canary analysis
Netflix and Google open-sourced Kayenta in 2018 to automate this. It runs three clusters: production, a fresh baseline on the current version, and a canary on the new one, the last two the same size, each taking a small slice of traffic. The fresh baseline matters: comparing a new canary to long-running production confuses “new code” with “just restarted”. Each metric is compared statistically (a Mann–Whitney U test) and marked Pass, High or Low; the share of metrics that pass is the score, and the pipeline continues or rolls back on it.
Good canaries share a few properties:
- Judge on user-facing signals: error rates and latency (SLIs), not noisy resource metrics like CPU.
- Get enough traffic and time to be representative. A canary at 3 a.m. won’t see peak-load bugs.
- Widen in waves: one host, one zone, one region, everywhere, with a bake time at each.
- Canary everything that changes behaviour: binaries, but also config, feature flags and data pushes. Feature flags separate deploying code from releasing it, so a feature can be canaried and turned off without a redeploy.
Blameless postmortems
When something does go wrong, what happens next decides whether it happens again.
Fix the system, not the person
A postmortem is a written record of an incident: its impact, the actions taken to mitigate it, the contributing causes and the follow-up work to stop it recurring. Google writes one for user-visible downtime beyond a threshold, any data loss, any on-call intervention such as a rollback, slow resolution, or a monitoring failure.
Blameless means it focuses on the contributing causes without indicting any individual or team, and it assumes everyone acted with good intentions on the information they had. John Allspaw’s 2012 post at Etsy made the case from safety science: if engineers are punished for mistakes, they stop explaining them, and the organization loses exactly the information it needs. As Google’s SRE book puts it, you can’t “fix” people, but you can fix systems and processes so they support people making good decisions.
Architecture trade-offs
Maintainability shows up most in the big structural choices, and the fashionable answer isn’t always the maintainable one.
Microservices vs monolith. Splitting a system into independently deployable services lets teams ship without coordinating, scale parts separately and isolate failures. It also turns function calls into network calls, adds partial failure, and multiplies the number of things to deploy and observe. In July 2018 Segment described collapsing more than 140 destination microservices back into a single service, because managing them had become “a huge tax” on a small team; afterwards one engineer could make in minutes a change that used to need over 140 deployments. In March 2023 Prime Video’s video-quality monitoring team moved from serverless components orchestrated by AWS Step Functions to a single process and cut infrastructure cost by over 90%. The distributed design was paying per Step Functions state transition, several per second of video, and for a high volume of S3 calls to pass frames between components; in one process, both disappeared.
Neither story says microservices are wrong. Both say the boundary should follow team and scaling needs, not fashion. A well-structured modular monolith (clear module interfaces, one deployable) gets many of the benefits with few of the costs.
| Choice | Gains | Costs |
|---|---|---|
| Monolith (modular) | one deploy, function calls, easy refactors across modules | teams coordinate releases; scales as one unit |
| Microservices | independent deploys, isolation, per-service scaling | network calls, partial failure, many things to run and observe |
| Managed services | someone else runs it; fast to adopt | less control, harder debugging, vendor lock-in |
| Self-hosted | control, cost at steady scale | you own the operations, upgrades and on-call |
Cloud vs self-hosting
For anything a business does, there’s a build-or-buy choice. In infrastructure it’s usually: run it yourself, or rent it from a cloud provider as a managed service. The right answer depends on how predictable your load is and how good you are at operations.
What you pay for
The cloud’s real advantages are speed and elasticity: no hardware lead time, capacity that follows the load, and managed databases and queues someone else operates. Its costs are a higher unit price, less control and visibility (when a managed service misbehaves, you can’t log into it), and dependence on the provider’s own outages.
37signals, maker of Basecamp and HEY, is the best-documented counterexample. In October 2022 David Heinemeier Hansson argued that renting computers is mostly a bad deal for medium-sized companies with stable growth, and that the cloud shines at the two extremes: very simple, low-traffic apps, and wildly unpredictable load.
By October 2024 they put the savings at almost $2 million a year with the same operations team, and projected well over $10 million over five years once storage moved too. The honest caveats are theirs: the workload is steady, they already had data-centre space, and running hardware is still real work.
In the real world
Three of the largest outages of 2024–2025 share one shape: a change that reached everything at once.
CrowdStrike, 19 July 2024. At 04:09 UTC a Rapid Response Content update was pushed to Windows hosts running the Falcon sensor, and every one online before it was reverted could receive it. A bug in the content validator had let a malformed template instance through; the sensor read out of bounds and Windows crashed. The update was reverted at 05:27, but many crashed machines needed hands-on repair. This content type had not been staged. CrowdStrike’s fix was a canary and then gradually larger portions of the fleet, with customers choosing when updates land.
Google Cloud, 12 June 2025. A new quota-check feature in Service Control, the binary behind API policy checks, shipped region by region on 29 May, but the code path that failed was never exercised during that rollout, and it had neither error handling nor a feature flag. On 12 June a policy change with blank fields was written to Spanner and replicated globally within seconds; every region hit a null pointer and crash-looped. In large regions, restarting tasks stampeded the database because they lacked randomized exponential backoff. The action items: feature-flag critical changes (off by default), fail open, and propagate global data incrementally.
Cloudflare, 18 November 2025. The same lesson for generated config:
The config fast path
The pattern in all three: careful staging for code, and a fast path for config or data that skipped it. Canary everything that changes behaviour, validate what your own systems generate as if a stranger wrote it, and keep a global kill switch that works when the system is on fire.
Pitfalls
- “We’ll clean it up later.” Accidental complexity compounds. Later is more expensive, and usually never.
- Dashboards as observability. Pre-aggregated metrics answer the questions you predicted. Keep raw, high-cardinality events for the ones you didn’t.
- Assuming one version at a time. During a rolling upgrade or a rollback, two versions read each other’s data. Test the mix, not just the end state.
- Canaries that can’t fail. No traffic, no comparison baseline, or nobody watching the result means it’s just a slow deploy.
- Config is code. Flags, policy data and generated files cause outages too. Give them the same staging, validation and rollback.
- Blame. It feels like accountability and works like a gag order. Fix the tool that let the mistake through.
- Architecture by fashion. Microservices, serverless and the cloud are trade-offs, not upgrades. Pick for your team size, load shape and operational skill.
Recall
Go deeper
- Google SRE book, “Postmortem Culture: Learning from Failure”: when to write one, what blameless means, and how reviews keep them honest.
- Google SRE workbook, “Canarying Releases”: canary vs control, choosing metrics, sizing and duration.
- Michael Graff & Chris Sanden, “Automated Canary Analysis at Netflix with Kayenta” (2018): the baseline/canary design and how the score is computed.
- Ben Moseley & Peter Marks, “Out of the Tar Pit” (2006): essential vs accidental complexity, and why state is the main culprit.
- Charity Majors, “There Is Only One Key Difference Between Observability 1.0 and 2.0” (2024): wide structured events as the single source of truth.
- Cloudflare, “Cloudflare outage on November 18, 2025” and Google Cloud’s 12 June 2025 incident report: two exemplary postmortems about config and data that took a fast path past staging. Cloudflare’s follow-up, “Code Orange: Fail Small” (2025), is the plan to stage config like code.
- David Heinemeier Hansson, “Our cloud exit savings will now top ten million over five years” (2024): the numbers behind a well-documented exit, with its caveats.
- Kubernetes, “Version Skew Policy”: what a real compatibility contract for rolling upgrades looks like.
Keep going
The atomic notes for this lesson live in the knowledge graph:
Next on the map: Load Balancing and Timeouts, Retries & Overload for the machinery behind safe rollouts, and Encoding & Schema Evolution for the compatibility rules that make rolling upgrades work.