Change Data Capture & Event Sourcing
Your database already writes down every change, in order. Turn that log into a stream and every other system can follow it, or make the log of events the database itself.
The whole idea
Derived data
No real application lives in one database. An order written to Postgres also needs to show up in the search index, invalidate a cache entry, land in the warehouse for tomorrow’s report and feed a fraud model. Each of those systems holds a copy of the same facts, reshaped for a different kind of read.
It helps to sort every store into one of two kinds:
- The system of record holds the authoritative version of the data. When a new fact arrives it is written here first, and if anything disagrees with it, it wins.
- Derived data is everything computed from it: caches, search indexes, materialized views, read replicas, denormalised tables, aggregates in a warehouse, a trained model. Lose it and you can rebuild it from the source.
The distinction isn’t about the technology. Postgres can be either one, and so can Kafka. It’s about which way the data flows. Once you draw that arrow, the hard question becomes how derived stores stay in step with the source, in the right order, and without losing anything.Derived data is redundant by definition, which is exactly what normalization tries to avoid. That’s fine, as long as everyone knows which copy is the truth.
The dual-write problem
The obvious approach is to have the application update everything itself: write the database, then write the search index, then delete the cache key. It works in a demo, then fails in production in two ways.
Two writers, two stores, two orders
The first is a race. Two clients write the same key at nearly the same time. Nothing makes the database and the index receive those writes in the same order, so each store keeps a different “last” value. There’s no error to alert on, and the difference is permanent.
The second is partial failure. The database write commits and the index write times out. Now one store has the change and the other doesn’t, and there’s no transaction spanning the two to roll back. Retrying helps with the timeout, but a crash between the two writes loses the second one entirely. The transactional outbox exists for exactly this.
Change data capture
Every database already keeps that ordered record. To survive crashes, it writes each change to its write-ahead log before applying it. To feed its replicas, it ships a replication log of row changes in commit order. Change data capture (CDC) reads that log and publishes it as a stream that any system can consume. The search index becomes a follower of the database, just like a replica.
One order, applied everywhere
Log-based CDC beats the older alternatives on every axis that matters:
| Approach | How it finds changes | What it misses |
|---|---|---|
| Polling | query rows WHERE updated_at > last run | hard deletes, every intermediate state, rows whose clock is skewed |
| Triggers | a trigger writes each change to a side table | nothing, but every write pays for it inside the transaction |
| Log-based | tail the WAL, binlog or oplog | nothing; no load on the write path |
The details depend on the database. Postgres gained logical decoding and replication slots in 9.4 (2014), and its built-in pgoutput plugin, available since version 10, turns WAL records into row changes. MySQL CDC reads the binlog, which has to be in ROW format with full row images. MongoDB exposes its oplog as change streams, with resume tokens so a consumer can continue where it stopped.
Debezium is the standard open-source CDC platform. It runs as a set of Kafka Connect source connectors (Postgres, MySQL, MariaDB, MongoDB, SQL Server, Oracle, Db2, Cassandra and more), and also as Debezium Server, which writes straight to Kinesis, Pub/Sub, Pulsar or NATS with no Kafka in the way. Each change becomes an event with a fixed envelope:
{
"before": { "id": 42, "status": "pending" },
"after": { "id": 42, "status": "paid" },
"source": { "connector": "postgresql", "table": "orders", "lsn": 24023128, "txId": 555, "ts_ms": 1731900000120 },
"op": "u",
"ts_ms": 1731900000431
}
op is c, u or d for create, update and delete, and r for a row read during a snapshot. The gap between source.ts_ms (when the database committed) and ts_ms (when Debezium processed it) is your CDC lag. Events are keyed by primary key, so every change to one row lands in one Kafka partition, in order. After a delete, Debezium also emits a tombstone, a message with the same key and a null value. On a topic with log compaction, that lets Kafka eventually drop every record for the row, while a compacted topic still holds the latest state of every row that exists. Such a topic is the table.
The initial snapshot
The log only reaches back so far. Postgres recycles WAL and MySQL purges old binlogs, so a new consumer can’t build a full copy of the table from the log alone. It needs a snapshot of the current state, followed by the stream of changes from exactly that point.
The classic way is to hold a lock, or a long repeatable-read transaction, while dumping the whole table. On a big table that takes hours, and the log can’t be consumed while it runs. Netflix’s DBLog (2019) found a better way: interleave small chunks of the table with the live log, using watermarks written into the log itself to tell which value is newer.
Snapshot and stream, interleaved
The trick works because both watermarks are ordinary writes, so they appear in the log in commit order. The chunk’s SELECT ran somewhere between them. Any key that changes in that window has a log event at least as new as the chunk’s copy, so the chunk row is dropped and the log wins. Every other chunk row is safe to emit at the high watermark. Debezium adopted the same algorithm as incremental snapshots (1.6, 2021). You trigger one by inserting a row into a signalling table, and it reads in chunks of 1,024 rows by default. That makes snapshotting a table a routine operation, not a migration project.
Replication slots
A CDC connector reading Postgres holds a replication slot, which records how far the consumer has confirmed. Postgres keeps every WAL segment a slot still needs. That’s what makes CDC lossless, and it’s also the most common way CDC takes down a database.
A stalled consumer, a full disk
max_slot_wal_keep_size defaults to −1, which means unlimited. Set a cap and alert on retained WAL per slot (compare confirmed_flush_lsn in pg_replication_slots with the current WAL position). On a database with little traffic on the captured tables, enable Debezium’s heartbeat, so the slot keeps advancing even when nothing changes. Since Postgres 17, logical slots can also fail over to a standby, so a primary failover no longer forces every CDC consumer to re-snapshot.
Event sourcing
CDC takes a database built around current state and recovers the history from its log. Event sourcing turns that around: the history is the database. Every change to application state is stored as an immutable event in an append-only log, and current state is whatever you get by replaying those events.
State is a fold over events
- #1AccountOpened
Two words do a lot of work here. A command is a request (“withdraw 500”), and it can be rejected. The service validates it against the current state, and only if it passes does it record an event, a fact in the past tense (“Withdrew 30”). Events are never updated or deleted. A mistake is corrected by a new, compensating event, the way an accountant posts a reversing entry instead of erasing the ledger.
The CDC events above describe rows: “status changed from pending to paid”. An event-sourced log describes intent: “OrderPaid, by card, with this authorisation”. That’s both the appeal and the cost. You get:
- A complete audit trail for free: why every value is what it is.
- New read models from old data. A new projection replays the log from the start, including history nobody knew would matter.
- Time travel and debugging. Rebuild the state as of any point, or replay production events against a fixed handler.
You pay with three problems that never go away:
- Event schema evolution. Events are stored forever, so every handler must read every old version. Teams upcast old events to the current shape on read, which is schema evolution with no option to rewrite history.
- Replay side effects. A projection that sends emails must not send them again on replay, so external calls have to be wrapped and switched off during rebuilds.
- Deletion. An immutable log and the right to be forgotten conflict. The usual answer is crypto-shredding: encrypt personal data per subject and delete the key.
Long streams are made cheap again with snapshots: persist the state every N events, and replay only from the latest snapshot.
CQRS
Event sourcing nearly forces a second idea on you, because “replay every event for this account” is a poor way to answer “show the 20 most recent orders over $100”. Command Query Responsibility Segregation (CQRS), a name coined by Greg Young around 2010, uses one model to change data and a different model, or several, to read it.
Write one model, read another
The read model is a materialized view of the event log, maintained by a projector. Each one can live in whatever store suits its queries: Postgres for a list screen, Elasticsearch for search, Redis for a counter. The price is eventual consistency: a user who saves and immediately reloads can see the old state, which you then have to paper over with read-your-writes tricks such as reading from the write model for a few seconds, or waiting for the projection to reach the event’s position.
CQRS doesn’t require event sourcing. A normal Postgres database with CDC feeding an Elasticsearch read model is CQRS too, and probably the most common form of it.
Materialized views
Every derived store in this lesson is a materialized view: the stored result of a query over a source, kept up to date as the source changes. The interesting part is how it’s kept up to date.
Recompute, or apply the delta
- #1shipped
- #2pending
- #3shipped
- #4cancelled
- #5shipped
- #6pending
- #7pending
- #8shipped
- pending3
- shipped4
- cancelled1
A Postgres materialized view is the simple kind. REFRESH MATERIALIZED VIEW re-runs the query and completely replaces the contents. The CONCURRENTLY option avoids blocking readers, but it needs a unique index on the view, and it still recomputes everything. Fine for a nightly report, useless for anything that must be fresh.
Incremental view maintenance (IVM) treats every change as a delta and computes only the delta of the result. Sums and counts are easy. Joins, distinct counts and MIN or MAX need indexed state, because deleting the current maximum means finding the next one. That state is what IVM engines are built around:
- Materialize is built on Timely and Differential Dataflow, whose operators turn input diffs into output diffs. It keeps SQL views correct under arbitrary inserts, updates and deletes.
- RisingWave is Postgres wire-compatible and keeps its streaming state in object storage.
- Feldera implements DBSP, a formal theory of incremental computation that won a best research paper award at VLDB 2023.
- Stream processors like Flink and Kafka Streams do the same thing in code: a stateful operator is an incrementally maintained view.
Point any of these at a Debezium topic and you get a view that’s seconds behind Postgres.
Trade-offs
| Change data capture | Event sourcing | |
|---|---|---|
| Source of truth | mutable tables; the log is a by-product | the immutable event log itself |
| Events describe | row changes (before / after) | domain intent (OrderPaid, FeeRefunded) |
| Adoption | bolt onto an existing database, no app changes | a different way of writing the application |
| Coupling | consumers see your table schema, an internal detail | consumers see events you designed as a contract |
| History | only as far back as log retention plus snapshots | everything, forever |
| Hard parts | slots, snapshots, schema changes leaking downstream | event versioning, replay side effects, deletion |
The two blend. The transactional outbox is the usual bridge: the application writes designed domain events to an outbox table in the same transaction as its state, and CDC carries them out. Debezium’s outbox event router reads columns named id, aggregatetype, aggregateid, type and payload, routes each row to a topic such as outbox.event.order, keys it by aggregate ID to keep each aggregate’s events in order, and puts the event ID in a header so consumers can deduplicate.
In the real world
- Shopify (2021) captures its sharded monolith (100+ independent MySQL databases) with roughly 150 Debezium connectors on 12 Kubernetes pods. Over Black Friday 2020 it averaged about 65,000 records per second, with spikes to 100,000, and kept over 400 TB of CDC data in Kafka. Kafka Streams merges per-shard topics into compacted per-table topics. It replaced Longboat, a query-based batch extractor that ran at most hourly and couldn’t see hard deletes.
- Netflix DBLog (2019) has captured MySQL and PostgreSQL in production since 2018 for Delta, Netflix’s data synchronisation platform. Its watermark snapshots acquire no table locks, and you can pause and resume a snapshot at any time.
- LinkedIn Databus, first built in 2005 (its current design in production since 2011) and open-sourced in 2013, split CDC into a relay, which pulls committed changes from Oracle into circular event buffers held in memory (optionally memory-mapped to a file), and a bootstrap service that serves a snapshot to new consumers, or a consolidated set of changes to ones that have fallen behind the relay’s window, which then catch up from the relay. That’s the same snapshot-then-stream shape every CDC system still has.
- Airbnb SpinalTap (2018) captures changes from MySQL binlogs, DynamoDB and an in-house store. It promises at-least-once delivery and per-row commit ordering, and fences off old leaders with epochs to avoid split brain.
- Event stores are a category of their own: EventStoreDB (renamed KurrentDB with its 25.0 release in 2025, when the company rebranded as Kurrent) and Axon Server for Java’s Axon Framework. Most event-sourced systems, though, run on Postgres or Kafka.
Pitfalls
- Publishing your table schema as an API. Raw CDC topics expose column names, types and internal states. Renaming a column breaks consumers you’ve never met. Use an outbox, or a stream processor that maps rows to stable events, as the contract.
- An abandoned replication slot. Decommissioning a connector without dropping its slot pins WAL until the disk fills. Monitor slots like you monitor disk.
- Treating CDC as exactly once. Connectors restart and redeliver. Consumers must apply events idempotently, for example as upserts keyed by primary key or with an LSN check.
- Assuming order across rows. Order is per key (per partition). A transaction that touches two tables arrives as several events, possibly on several topics. Debezium can emit transaction boundary metadata if consumers need it.
- Event sourcing everything. Fowler’s warning about CQRS applies doubly. Use it for the core domain where history and audit matter, not for CRUD settings pages.
- Projections that send emails. Replaying a log will re-run side effects unless they’re explicitly disabled during a rebuild.
Recall
Go deeper
- Andreakis & Papapanagiotou, “DBLog: A Generic Change-Data-Capture Framework” (Netflix, 2019) and the paper (2020): the watermark snapshot algorithm, step by step.
- Debezium: PostgreSQL connector: the event envelope, snapshot modes, slots, heartbeats and failover slots, in one long and excellent page.
- Debezium: Outbox Event Router: how CDC and the outbox pattern combine into designed events.
- Martin & Bellemare, “Capturing Every Change From Shopify’s Sharded Monolith” (2021): CDC at real scale, with the numbers and the operational lessons.
- Botev et al., “Open-sourcing Databus” (LinkedIn, 2013): the relay and bootstrap design that shaped every CDC system since.
- Martin Fowler, “Event Sourcing” (2005) and “CQRS” (2011): the definitions, and the warnings worth heeding.
- Chris Kiehl, “Event Sourcing is Hard” (2019): a practitioner’s list of what hurt, from event versioning to UIs that 404 on fresh data.
- Budiu et al., “DBSP: Automatic Incremental View Maintenance for Rich Query Languages” (VLDB 2023): the theory of turning any query into a delta-in, delta-out circuit.
Keep going
The atomic notes for this lesson live in the knowledge graph:
Next on the map: Streaming Systems as Architecture, where the log stops being plumbing and becomes the centre of the whole design.