Stream Processing · 3 of 4

Change Data Capture & Event Sourcing

Your database already writes down every change, in order. Turn that log into a stream and every other system can follow it, or make the log of events the database itself.

≈ 25 min

The whole idea

Press ← → to step, or Space to play and pause, whenever a diagram has focus.

Derived data

No real application lives in one database. An order written to Postgres also needs to show up in the search index, invalidate a cache entry, land in the warehouse for tomorrow’s report and feed a fraud model. Each of those systems holds a copy of the same facts, reshaped for a different kind of read.

It helps to sort every store into one of two kinds:

  • The system of record holds the authoritative version of the data. When a new fact arrives it is written here first, and if anything disagrees with it, it wins.
  • Derived data is everything computed from it: caches, search indexes, materialized views, read replicas, denormalised tables, aggregates in a warehouse, a trained model. Lose it and you can rebuild it from the source.

The distinction isn’t about the technology. Postgres can be either one, and so can Kafka. It’s about which way the data flows. Once you draw that arrow, the hard question becomes how derived stores stay in step with the source, in the right order, and without losing anything.Derived data is redundant by definition, which is exactly what normalization tries to avoid. That’s fine, as long as everyone knows which copy is the truth.

The dual-write problem

The obvious approach is to have the application update everything itself: write the database, then write the search index, then delete the cache key. It works in a demo, then fails in production in two ways.

Two writers, two stores, two orders

Each write succeeds. The stores still end up different, and nothing will ever notice.

The first is a race. Two clients write the same key at nearly the same time. Nothing makes the database and the index receive those writes in the same order, so each store keeps a different “last” value. There’s no error to alert on, and the difference is permanent.

The second is partial failure. The database write commits and the index write times out. Now one store has the change and the other doesn’t, and there’s no transaction spanning the two to roll back. Retrying helps with the timeout, but a crash between the two writes loses the second one entirely. The transactional outbox exists for exactly this.

Change data capture

Every database already keeps that ordered record. To survive crashes, it writes each change to its write-ahead log before applying it. To feed its replicas, it ships a replication log of row changes in commit order. Change data capture (CDC) reads that log and publishes it as a stream that any system can consume. The search index becomes a follower of the database, just like a replica.

One order, applied everywhere

With one writer and one log, the index can be late, but never different.

Log-based CDC beats the older alternatives on every axis that matters:

Approach How it finds changes What it misses
Polling query rows WHERE updated_at > last run hard deletes, every intermediate state, rows whose clock is skewed
Triggers a trigger writes each change to a side table nothing, but every write pays for it inside the transaction
Log-based tail the WAL, binlog or oplog nothing; no load on the write path

The details depend on the database. Postgres gained logical decoding and replication slots in 9.4 (2014), and its built-in pgoutput plugin, available since version 10, turns WAL records into row changes. MySQL CDC reads the binlog, which has to be in ROW format with full row images. MongoDB exposes its oplog as change streams, with resume tokens so a consumer can continue where it stopped.

Debezium is the standard open-source CDC platform. It runs as a set of Kafka Connect source connectors (Postgres, MySQL, MariaDB, MongoDB, SQL Server, Oracle, Db2, Cassandra and more), and also as Debezium Server, which writes straight to Kinesis, Pub/Sub, Pulsar or NATS with no Kafka in the way. Each change becomes an event with a fixed envelope:

{
  "before": { "id": 42, "status": "pending" },
  "after":  { "id": 42, "status": "paid" },
  "source": { "connector": "postgresql", "table": "orders", "lsn": 24023128, "txId": 555, "ts_ms": 1731900000120 },
  "op": "u",
  "ts_ms": 1731900000431
}

op is c, u or d for create, update and delete, and r for a row read during a snapshot. The gap between source.ts_ms (when the database committed) and ts_ms (when Debezium processed it) is your CDC lag. Events are keyed by primary key, so every change to one row lands in one Kafka partition, in order. After a delete, Debezium also emits a tombstone, a message with the same key and a null value. On a topic with log compaction, that lets Kafka eventually drop every record for the row, while a compacted topic still holds the latest state of every row that exists. Such a topic is the table.

The initial snapshot

The log only reaches back so far. Postgres recycles WAL and MySQL purges old binlogs, so a new consumer can’t build a full copy of the table from the log alone. It needs a snapshot of the current state, followed by the stream of changes from exactly that point.

The classic way is to hold a lock, or a long repeatable-read transaction, while dumping the whole table. On a big table that takes hours, and the log can’t be consumed while it runs. Netflix’s DBLog (2019) found a better way: interleave small chunks of the table with the live log, using watermarks written into the log itself to tell which value is newer.

Snapshot and stream, interleaved

Tableprimary key orderk1=ak2=bk3=ck4=dk5=ek6=f
Logcommit order →k5=e
Chunkin memoryempty
Outputwhat consumers see →k5=eu
The watermark algorithm. Writes never wait, nothing is locked, and a chunk can be re-run at any time.

The trick works because both watermarks are ordinary writes, so they appear in the log in commit order. The chunk’s SELECT ran somewhere between them. Any key that changes in that window has a log event at least as new as the chunk’s copy, so the chunk row is dropped and the log wins. Every other chunk row is safe to emit at the high watermark. Debezium adopted the same algorithm as incremental snapshots (1.6, 2021). You trigger one by inserting a row into a signalling table, and it reads in chunks of 1,024 rows by default. That makes snapshotting a table a routine operation, not a migration project.

Replication slots

A CDC connector reading Postgres holds a replication slot, which records how far the consumer has confirmed. Postgres keeps every WAL segment a slot still needs. That’s what makes CDC lossless, and it’s also the most common way CDC takes down a database.

A stalled consumer, a full disk

The slot guarantees nothing is lost by keeping everything, so the database is only as healthy as its slowest slot.

max_slot_wal_keep_size defaults to −1, which means unlimited. Set a cap and alert on retained WAL per slot (compare confirmed_flush_lsn in pg_replication_slots with the current WAL position). On a database with little traffic on the captured tables, enable Debezium’s heartbeat, so the slot keeps advancing even when nothing changes. Since Postgres 17, logical slots can also fail over to a standby, so a primary failover no longer forces every CDC consumer to re-snapshot.

Event sourcing

CDC takes a database built around current state and recovers the history from its log. Event sourcing turns that around: the history is the database. Every change to application state is stored as an immutable event in an append-only log, and current state is whatever you get by replaying those events.

State is a fold over events

Commandopen account
Event log append-only
  1. #1AccountOpened
Balance projection0
Statement not built yet
    Commands can fail; events can't be undone, only followed by other events.

    Two words do a lot of work here. A command is a request (“withdraw 500”), and it can be rejected. The service validates it against the current state, and only if it passes does it record an event, a fact in the past tense (“Withdrew 30”). Events are never updated or deleted. A mistake is corrected by a new, compensating event, the way an accountant posts a reversing entry instead of erasing the ledger.

    The CDC events above describe rows: “status changed from pending to paid”. An event-sourced log describes intent: “OrderPaid, by card, with this authorisation”. That’s both the appeal and the cost. You get:

    • A complete audit trail for free: why every value is what it is.
    • New read models from old data. A new projection replays the log from the start, including history nobody knew would matter.
    • Time travel and debugging. Rebuild the state as of any point, or replay production events against a fixed handler.

    You pay with three problems that never go away:

    • Event schema evolution. Events are stored forever, so every handler must read every old version. Teams upcast old events to the current shape on read, which is schema evolution with no option to rewrite history.
    • Replay side effects. A projection that sends emails must not send them again on replay, so external calls have to be wrapped and switched off during rebuilds.
    • Deletion. An immutable log and the right to be forgotten conflict. The usual answer is crypto-shredding: encrypt personal data per subject and delete the key.

    Long streams are made cheap again with snapshots: persist the state every N events, and replay only from the latest snapshot.

    CQRS

    Event sourcing nearly forces a second idea on you, because “replay every event for this account” is a poor way to answer “show the 20 most recent orders over $100”. Command Query Responsibility Segregation (CQRS), a name coined by Greg Young around 2010, uses one model to change data and a different model, or several, to read it.

    Write one model, read another

    The read model is derived data: denormalised for one screen, and briefly behind.

    The read model is a materialized view of the event log, maintained by a projector. Each one can live in whatever store suits its queries: Postgres for a list screen, Elasticsearch for search, Redis for a counter. The price is eventual consistency: a user who saves and immediately reloads can see the old state, which you then have to paper over with read-your-writes tricks such as reading from the write model for a few seconds, or waiting for the projection to reach the event’s position.

    CQRS doesn’t require event sourcing. A normal Postgres database with CDC feeding an Elasticsearch read model is CQRS too, and probably the most common form of it.

    Materialized views

    Every derived store in this lesson is a materialized view: the stored result of a query over a source, kept up to date as the source changes. The interesting part is how it’s kept up to date.

    Recompute, or apply the delta

    orders base table
    • #1shipped
    • #2pending
    • #3shipped
    • #4cancelled
    • #5shipped
    • #6pending
    • #7pending
    • #8shipped
    count by status materialized view
    • pending3
    • shipped4
    • cancelled1
    Full refresh costs as much as the table. Incremental maintenance costs as much as the change.

    A Postgres materialized view is the simple kind. REFRESH MATERIALIZED VIEW re-runs the query and completely replaces the contents. The CONCURRENTLY option avoids blocking readers, but it needs a unique index on the view, and it still recomputes everything. Fine for a nightly report, useless for anything that must be fresh.

    Incremental view maintenance (IVM) treats every change as a delta and computes only the delta of the result. Sums and counts are easy. Joins, distinct counts and MIN or MAX need indexed state, because deleting the current maximum means finding the next one. That state is what IVM engines are built around:

    • Materialize is built on Timely and Differential Dataflow, whose operators turn input diffs into output diffs. It keeps SQL views correct under arbitrary inserts, updates and deletes.
    • RisingWave is Postgres wire-compatible and keeps its streaming state in object storage.
    • Feldera implements DBSP, a formal theory of incremental computation that won a best research paper award at VLDB 2023.
    • Stream processors like Flink and Kafka Streams do the same thing in code: a stateful operator is an incrementally maintained view.

    Point any of these at a Debezium topic and you get a view that’s seconds behind Postgres.

    Trade-offs

    Change data capture Event sourcing
    Source of truth mutable tables; the log is a by-product the immutable event log itself
    Events describe row changes (before / after) domain intent (OrderPaid, FeeRefunded)
    Adoption bolt onto an existing database, no app changes a different way of writing the application
    Coupling consumers see your table schema, an internal detail consumers see events you designed as a contract
    History only as far back as log retention plus snapshots everything, forever
    Hard parts slots, snapshots, schema changes leaking downstream event versioning, replay side effects, deletion

    The two blend. The transactional outbox is the usual bridge: the application writes designed domain events to an outbox table in the same transaction as its state, and CDC carries them out. Debezium’s outbox event router reads columns named id, aggregatetype, aggregateid, type and payload, routes each row to a topic such as outbox.event.order, keys it by aggregate ID to keep each aggregate’s events in order, and puts the event ID in a header so consumers can deduplicate.

    In the real world

    100,000records per second at peak through Shopify’s CDC pipeline, Black Friday 2020
    100+sharded MySQL databases captured by about 150 Debezium connectors
    p99 under 10 sfrom a row committing to its event landing in Kafka
    • Shopify (2021) captures its sharded monolith (100+ independent MySQL databases) with roughly 150 Debezium connectors on 12 Kubernetes pods. Over Black Friday 2020 it averaged about 65,000 records per second, with spikes to 100,000, and kept over 400 TB of CDC data in Kafka. Kafka Streams merges per-shard topics into compacted per-table topics. It replaced Longboat, a query-based batch extractor that ran at most hourly and couldn’t see hard deletes.
    • Netflix DBLog (2019) has captured MySQL and PostgreSQL in production since 2018 for Delta, Netflix’s data synchronisation platform. Its watermark snapshots acquire no table locks, and you can pause and resume a snapshot at any time.
    • LinkedIn Databus, first built in 2005 (its current design in production since 2011) and open-sourced in 2013, split CDC into a relay, which pulls committed changes from Oracle into circular event buffers held in memory (optionally memory-mapped to a file), and a bootstrap service that serves a snapshot to new consumers, or a consolidated set of changes to ones that have fallen behind the relay’s window, which then catch up from the relay. That’s the same snapshot-then-stream shape every CDC system still has.
    • Airbnb SpinalTap (2018) captures changes from MySQL binlogs, DynamoDB and an in-house store. It promises at-least-once delivery and per-row commit ordering, and fences off old leaders with epochs to avoid split brain.
    • Event stores are a category of their own: EventStoreDB (renamed KurrentDB with its 25.0 release in 2025, when the company rebranded as Kurrent) and Axon Server for Java’s Axon Framework. Most event-sourced systems, though, run on Postgres or Kafka.

    Pitfalls

    • Publishing your table schema as an API. Raw CDC topics expose column names, types and internal states. Renaming a column breaks consumers you’ve never met. Use an outbox, or a stream processor that maps rows to stable events, as the contract.
    • An abandoned replication slot. Decommissioning a connector without dropping its slot pins WAL until the disk fills. Monitor slots like you monitor disk.
    • Treating CDC as exactly once. Connectors restart and redeliver. Consumers must apply events idempotently, for example as upserts keyed by primary key or with an LSN check.
    • Assuming order across rows. Order is per key (per partition). A transaction that touches two tables arrives as several events, possibly on several topics. Debezium can emit transaction boundary metadata if consumers need it.
    • Event sourcing everything. Fowler’s warning about CQRS applies doubly. Use it for the core domain where history and audit matter, not for CRUD settings pages.
    • Projections that send emails. Replaying a log will re-run side effects unless they’re explicitly disabled during a rebuild.

    Recall

    Go deeper

    Keep going

    The atomic notes for this lesson live in the knowledge graph:

    Next on the map: Streaming Systems as Architecture, where the log stops being plumbing and becomes the centre of the whole design.

    Done with this lesson?

    Mark it learned and the concepts it teaches will light up in your galaxy.

    Next in Stream ProcessingStreaming Systems as Architecture