NotesSystem

Debezium

Also: Debezium Server · Debezium Engine

The standard open-source change data capture platform: connectors that read database logs (WAL, binlog, oplog, redo log) and emit every row change as an ordered event, usually into Kafka. Its defining choice is log-based capture with a snapshot-then-stream bootstrap.

Also in: Change Data Capture & Event Sourcing › In the real world

Design choices

  • Change Data Capture, log-based: Postgres through logical decoding (pgoutput) and a replication slot. MySQL and MariaDB through the binlog (ROW format, full row images). Also MongoDB, SQL Server, Oracle, Db2, Cassandra and others.
  • Event envelope: before, after, source (connector, table, LSN or offset, transaction ID, commit time), op (c, u, d, r for snapshot reads, t for truncate) and ts_ms. Commit time vs processing time gives the lag.
  • Keyed by primary key, so changes to one row stay ordered in one partition. Deletes are followed by a tombstone so Log Compaction can remove the row.
  • Snapshots: snapshot.mode sets the initial behaviour (initial, no_data, when_needed, and more). Incremental snapshots (since 1.6) interleave primary-key chunks of 1,024 rows by default with the live log, using watermark signals, with no locks. They're triggered by writing to a signalling table.
  • Transactional Outbox: the outbox event router routes outbox rows by aggregatetype to topics like outbox.event.order, keys them by aggregateid, and puts the event id in a header for deduplication.
  • Runtimes: Kafka Connect connectors; Debezium Server, which writes to Kinesis, Pub/Sub, Pulsar, NATS, Redis Streams and other sinks without Kafka; and the embedded Debezium Engine Java library.
  • Postgres operations: a heartbeat lets slots advance on quiet databases. Since Postgres 17, failover slots survive a primary failover.

Notable trade-offs

  • At-least-once delivery: restarts redeliver recent events, so consumers must be idempotent.
  • Raw CDC events expose the source schema, and schema changes flow straight to consumers unless you route through an outbox or a mapping layer.
  • An unconsumed replication slot retains WAL on the source, so a stopped connector can fill the primary's disk.