Notesstream🌱 seedling

Change Data Capture

Also: CDC · Log-Based CDC · Logical Decoding · Change Stream

Read a database's own replication log and publish every committed row change, in commit order, as a stream. Downstream stores then follow the database the way a replica does, with no dual writes.

Also in: Change Data Capture & Event Sourcing › The initial snapshot

Why it exists

Applications that write a database and then an index or cache themselves suffer races (writes arrive in different orders) and partial failures. CDC makes the database the only writer and derives the rest from its order.

How it works

  • Log-based: tail the WAL (Postgres logical decoding with pgoutput, from a replication slot), the binlog (MySQL, in ROW format) or the oplog (MongoDB change streams). No load on the write path, and it sees deletes and every intermediate state.
  • Events carry before and after row images, an operation (create, update, delete, snapshot read) and the source position (LSN, binlog offset).
  • Keyed by primary key, so every change to one row stays ordered in one partition. A delete is followed by a tombstone so Log Compaction can drop the key.
  • Snapshot plus stream: the log doesn't go back to the beginning, so a new consumer first gets the current table, then the changes from that exact point. Watermark-based chunked snapshots interleave the two without locks.

Trade-offs

Gains Costs
one writer, one order: no dual-write races consumers see your internal table schema
no changes to application code replication slots, snapshots and lag to operate
replayable, so new consumers can bootstrap at-least-once: consumers must be idempotent

In practice

  • Shopify captures 100+ MySQL shards with about 150 Debezium connectors, peaking near 100,000 records per second (2021). It replaced an hourly updated_at poller that missed hard deletes.

Pitfalls

  • A stopped consumer's replication slot retains WAL until the primary's disk fills. Cap it with max_slot_wal_keep_size and alert on slot lag. On Postgres 17+, enable slot failover so a primary failover doesn't force a re-snapshot.
  • Treating raw CDC topics as a public contract: a column rename breaks consumers. Publish designed events through a Transactional Outbox.

Recall

These cards are also in your review deck, scheduled with spaced repetition.