Streaming Systems as Architecture
Put an ordered, replayable log at the centre, and treat every database, index and cache as a view of it. What that buys you, what it costs, and how to stay correct without distributed transactions.
The whole idea
Data integration
Every earlier lesson picked one tool for one job: a B-tree for point reads, an inverted index for search, a column store for analytics, a cache for hot keys. Real applications need all of them at once, over the same data. No single product is best at everything, so the real design question is how to keep many specialised stores consistent with each other.
The previous lesson gave the vocabulary. One system of record holds the truth. Every other store is derived data, computed from it. Change data capture and event logs move the changes, in order. This lesson takes the idea to its conclusion: if the log carries every change in a fixed order, the log is the real system of record, and every database around it is a cache of it.
The core reason this works is an old principle. Jay Kreps put it this way in his 2013 essay on logs: “If two identical, deterministic processes begin in the same state and get the same inputs in the same order, they will produce the same output and end in the same state.” That’s total order broadcast applied to a whole company’s data instead of to three replicas.
Unbundling the database
A single database already does all of this internally. Martin Kleppmann’s 2014 talk, Turning the database inside-out, pointed out that replication, secondary indexing, caching and materialized views are all derived data that a database maintains from its own log.
The database, taken apart
When you run CREATE INDEX, the database backfills the index from a consistent snapshot, then applies every later write to it, transactionally. That’s exactly snapshot plus stream, built into the engine. Unbundling means doing the same thing across systems:
- The log (Kafka, Pulsar, Kinesis, or a database’s own WAL exposed through CDC) becomes the shared write path.
- Each store (Postgres, Elasticsearch, Redis, a warehouse) is a specialised view that follows it.
- Stream processors (Flink, Kafka Streams, Materialize) replace the internal code that kept indexes and views current. In effect, they’re triggers that run across the whole company.
Kafka Streams formalises the result as stream–table duality: a stream is the changelog of a table, and a table is a snapshot of the latest value per key in a stream. Kreps’ version is a bank: the log is the list of credits and debits, and the table is the current balances.
There’s a second way to combine many stores. A federated system, like Trino or Postgres foreign data wrappers, unifies reads: one query language over many back ends. Unbundling unifies writes: one ordered log feeding many back ends. Federation answers “how do I query it all?” and unbundling answers “how do I keep it all consistent?” Most large shops end up with both.
Lambda architecture
Before logs could be kept for long, there was a split. In 2011 Nathan Marz argued that a data system is really “Query = Function(All Data)”, and that the way to compute it reliably is to keep all the raw data, immutable, and recompute views from scratch in batch. Batch is hours behind, so a second, realtime layer handles the recent events, approximately if need be. The design was later named the Lambda architecture.
Two paths, or one
Its insight still holds: immutable input plus recomputation makes bugs recoverable. If a job was wrong, fix it and rerun it over the raw data. Nothing is lost, because the raw data was never overwritten.
The cost is the second codebase. In 2014 Jay Kreps described maintaining code that must produce the same result in two complex distributed systems as “exactly as painful as it seems like it would be”. The batch job and the streaming job are written against different frameworks with different semantics, and the two must agree, forever. The query-time merge is its own source of bugs.
Kappa architecture
Kreps’ alternative was, in his words, “stupidly simple”: keep the input in a log that retains as much history as you would ever want to reprocess, and use a single stream processing job for everything. When the logic changes, reprocess by replaying the log. He named it with a shrug: “Maybe we could call this the Kappa Architecture, though it may be too simple of an idea to merit a Greek letter.”
Reprocessing by replay
The recipe has four steps. Retain the log (to reprocess 30 days, keep 30 days). Start a second instance of the job from the beginning, writing to a new table. When it has caught up, switch the application to the new table. Then stop the old job and delete the old table. This is the same shape as a rolling upgrade or a blue–green deploy, applied to derived data. The old and new views coexist, so you can compare them before switching.
Kappa only became practical as its preconditions arrived:
- Cheap, long retention. Kafka’s tiered storage (KIP-405) moves old segments to object storage. It shipped as early access in 3.6 (2023) and became production-ready in 3.9 (November 2024).
- Engines that treat batch as bounded streams. Apache Beam and Flink run the same pipeline over a bounded or an unbounded source. Flink 2.0 (March 2025) moved state to remote storage, so a job’s state can grow past its local disks.
- Deterministic jobs. Replay only reproduces results if processing depends on event time and the input, not on wall clocks or lookups to mutable services.
The end-to-end argument
Logs, offsets and exactly-once processing can make each hop reliable. They can’t make the whole reliable by themselves. That’s the end-to-end argument, from Saltzer, Reed and Clark’s 1984 paper:
The function in question can completely and correctly be implemented only with the knowledge and help of the application standing at the end points of the communication system. Therefore, providing that questioned function as a feature of the communication system itself is not possible.
Their example was a careful file transfer. Disks, buffers, memory, the network and host crashes can each corrupt it, so only a final check by the application (read the file back and compare checksums) proves the copy is right. They also told a “too-real” story from MIT. Every network hop checksummed its packets, but a gateway corrupted data while copying it between its own buffers, where no checksum covered it, swapping a pair of bytes about once in every million. Source files were damaged, and their owners had to repair them by hand from old listings.
Duplicates only the ends can see
The same logic applies to streams. Kafka’s exactly-once semantics keep a stream processor from double-counting inside Kafka. They can’t stop a user from submitting the same form twice, or a client from retrying after a lost response. The paper said it in 1984: duplicate suppression “must be accomplished by the application itself with knowledge of how to detect its own duplicates.” In practice that means an operation ID generated at the true start (the client), carried through every hop, and enforced as a unique constraint at the end. Stripe’s idempotency keys are exactly this.
Enforcing constraints
Some rules can’t be checked after the fact: a username must be unique, and an account may not go negative. In a single database you’d use a unique index or a serializable transaction. Across an unbundled system, the log itself can do it.
Uniqueness without locks
Route every request that touches a given constraint to the same log partition, for example by hashing the username. A single-threaded processor reads that partition and decides each request in order. The first claim wins, the second is rejected, and the result goes out on an output stream the client waits on. This is actual serial execution on a log, and it’s consensus in disguise: deciding a total order is what the log’s leader already did.
It scales the way logs scale, by partitioning, as long as each constraint fits in one partition. A request that spans partitions, like a transfer between two accounts on different partitions, needs one more idea.
Integrity vs timeliness
“Consistency” bundles two properties that are worth separating:
- Timeliness: readers see up-to-date state. Violating it is temporary: a lagging replica or index will catch up. That’s eventual consistency.
- Integrity: nothing is lost, nothing is applied twice, and derived data never contradicts the source. Violating it is permanent: a double-charged card or a lost credit doesn’t heal by itself.
Distributed transactions buy both at once, at the cost of coordination and availability. A log-based design can give up a little timeliness and still guarantee integrity.
Integrity now, timeliness eventually
The transfer is written to the log once, as a single message with a request ID. That append is the atomic commit point, so there’s nothing left to coordinate. A processor derives a debit and a credit, both tagged with the request ID, and sends them to the partitions that own each account. Every step is deterministic, and every consumer deduplicates by request ID, so a crash and replay changes nothing. A reader can briefly see the debit without the credit. That’s a timeliness violation, and it heals. Integrity holds throughout, with no two-phase commit.
This is how much of the real world already works. Helland and Campbell’s Building on Quicksand (2009) observed that all computing comes down to “memories, guesses, and apologies”, and that “every business includes apologies”. Airlines overbook and compensate. Warehouses sell stock that turns out to be damaged. Many constraints can be checked late, and a violation fixed with an apology or a compensating saga step, as long as the business allows it.
Bailis et al. (VLDB 2015) made that precise with invariant confluence, the condition under which operations can run without coordination and still preserve an invariant. Ten of TPC-C’s twelve invariants qualify. Uniqueness of a value the client chooses doesn’t, and neither does “balance never below zero” under concurrent withdrawals. So coordinate only for those, and let everything else flow.
Trust, but verify
Every layer has bugs, including the database, the broker and your stream jobs. The end-to-end argument applies to storage too: the only proof that derived data is right is to check it. Logs make this practical. Because derivations are deterministic functions of an immutable input, you can:
- Recompute a view from the log and diff it against the live one, continuously or on a schedule.
- Run the old and new versions of a job side by side during reprocessing, and compare their outputs before switching.
- Reconcile counts end to end (rows committed in the source vs events published vs rows in each sink), and alert on drift.
A mutable database that overwrites in place gives you nothing to check against. An event log is its own audit trail.
Trade-offs
| Lambda | Kappa | Unbundled + end-to-end checks | |
|---|---|---|---|
| Codebases | two (batch and streaming), plus a merge | one | one per derived view |
| Reprocessing | rerun the batch job over the master dataset | replay the retained log into a new table | same as Kappa |
| Needs | a data lake and two engines | long log retention, replay-capable jobs | request IDs and dedup everywhere |
| Weak spot | two implementations drifting apart | replay throughput and retention cost | cross-store reads are not a snapshot |
And the property trade-off underneath all of them:
| Distributed transaction | Log + idempotent processing | |
|---|---|---|
| Integrity | yes | yes, with request IDs and deterministic steps |
| Timeliness | immediate | eventual, typically seconds |
| Coordination | every write, across systems | only to append to one partition |
| Failure of one store | blocks or aborts writes | that view lags, everything else proceeds |
In the real world
- Uber (2020) described a production Kappa design for its streaming analytics. Lambda “requires maintaining two disparate codebases”, so Uber backfills by running the same Spark Streaming job with a Hive table as its source. The connector reads two hours of data per trigger, and nine days (about 10 TB) were replayed on 75 cores and 1.2 TB of memory.
- LinkedIn moved Who Viewed Your Profile off Lambda to a Samza nearline pipeline (2020), keeping only a small offline copy job to work around an OLAP-store limit. That halved most measures of development time and cut maintenance overhead by more than half. In 2023 it unified a standardisation pipeline (mapping job titles and skills to IDs for 900 million profiles) in Apache Beam, running the same code as a stream and as a batch backfill. The backfill dropped from seven hours to 25 minutes.
- Kafka with tiered storage makes “retain everything” an option. KIP-405 names the goal outright: make Kafka “one true source of data”.
- Streaming databases like Materialize and RisingWave package the unbundled idea back into one product: SQL views maintained incrementally from Kafka or CDC streams.
Pitfalls
- “Exactly-once” in the broker means exactly-once for the user. It covers reads and writes inside the streaming system. A client retry or a non-idempotent sink still duplicates. Carry an operation ID end to end.
- Non-deterministic jobs. A job that calls the wall clock, a random number generator or a mutable external service gives different results on replay, and Kappa reprocessing quietly diverges.
- Retention shorter than the replay you need. You can only reprocess what the log still has. Size retention, or a compacted snapshot plus a log, for the oldest view you might need to rebuild.
- Reading across views as if they were one snapshot. Two derived stores at different offsets can disagree. Read from one view, or track offsets and wait.
- Unbundling where one database would do. A single Postgres with a couple of indexes is a perfectly good unbundled-in-one-box system. Pull the log out when you have several consumers that really need it.
Recall
Go deeper
- Martin Kleppmann, “Turning the database inside-out with Apache Samza” (2015): the talk transcript that framed databases as derived data maintained from a log.
- Jay Kreps, “The Log: What every software engineer should know about real-time data’s unifying abstraction” (2013) (archived): the essay behind Kafka and the stream–table duality.
- Jay Kreps, “Questioning the Lambda Architecture” (2014): the critique, and the four-step reprocessing recipe now called Kappa.
- Nathan Marz, “How to beat the CAP theorem” (2011): the original batch-plus-realtime argument, and why immutability makes bugs recoverable.
- Saltzer, Reed & Clark, “End-to-End Arguments in System Design” (1984): twelve pages that still settle arguments about where reliability belongs.
- Helland & Campbell, “Building on Quicksand” (CIDR 2009): memories, guesses and apologies, and why businesses already run on eventual integrity.
- Bailis et al., “Coordination Avoidance in Database Systems” (VLDB 2015): exactly which invariants need coordination, and which don’t.
- Uber, “Designing a Production-Ready Kappa Architecture for Timely Data Stream Processing” (2020): what reprocessing by replay takes in practice.
Keep going
The atomic notes for this lesson live in the knowledge graph:
This is the last lesson on the map. From here, revisit the foundations: Replication and Consensus are the same log ideas at the scale of a few machines, and Batch Processing is the bounded case of everything here.