Notesstream🌱 seedling
Log Compaction
Also: Compacted Topic · Tombstone · cleanup.policy=compact · Key-Based Retention
Retention by key instead of by age: the broker keeps at least the latest record for every key and discards older ones, so replaying the topic from offset 0 always rebuilds the current table. A null-valued tombstone deletes a key.
Why it exists
Time-based retention suits event streams but breaks streams of state changes: delete by age and you lose keys that simply haven't changed lately. Keeping everything forever wastes space on superseded values.
How it works
- With cleanup.policy=compact, a background cleaner rewrites old segments, keeping only the newest record per key. The active segment is never cleaned.
- Survivors keep their original offsets (gaps appear) and their relative order.
- A record with a null value is a tombstone. It is kept for delete.retention.ms (24 h by default) so consumers see the delete, then removed.
- A segment becomes eligible once its dirty share passes min.cleanable.dirty.ratio (0.5 by default).
- The idea is the same as LSM Compaction, applied to a log: newest version wins, tombstones expire.
Trade-offs
| Gains | Costs |
|---|---|
| bounded size for unbounded history of state | no history of intermediate values |
| any consumer can bootstrap the full state | cleaner I/O; space reclaimed lazily |
| deletes propagate as tombstones | consumers lagging more than delete.retention.ms may miss deletes |
In practice
- Kafka stores consumer offsets in a compacted topic. Kafka Streams backs state stores with compacted changelogs, and Kafka Connect keeps configs and offsets in compacted topics.
- CDC: a compacted topic of row changes keyed by primary key is a replayable copy of the table.
Pitfalls
- Compaction is not immediate: old values linger until the cleaner runs, so don't rely on it alone for Right to Erasure.
- Records without keys can't be compacted.
Connections
Recall
These cards are also in your review deck, scheduled with spaced repetition.