Notesstream🌱 seedling

Consumer Offset

Also: Offset · Committed Offset · Consumer Position · Offset Commit · Consumer Lag

A log consumer's entire progress is one number per partition: the offset of the next record to read. The committed offset is the saved copy a restart resumes from, and where you commit it relative to your side effects decides the delivery guarantee.

Why it exists

A log broker keeps no per-message delivery state, so consumers must remember how far they've read. The offset makes that memory tiny and makes rewinding (replay) trivial.

How it works

  • Position: the next offset the consumer will fetch. It advances on every poll.
  • Committed offset: the position saved durably. After a crash or a rebalance, the new owner of the partition starts here.
  • Kafka stores commits in the compacted internal topic __consumer_offsets (50 partitions by default).
  • Auto-commit (on by default, every 5 s) commits what the previous poll returned, not what your code has finished.
  • Lag = log end offset − committed offset. It's the main health metric for a consumer.
  • Offsets of a group with no members expire after offsets.retention.minutes (7 days). Then auto.offset.reset (default latest) picks the restart point.

Trade-offs

Commit timing Result
before processing at-most-once: a crash skips records
after processing at-least-once: a crash replays records
atomically with the output exactly-once (Kafka transactions, or offsets stored with the result)

In practice

  • Kafka: commitSync / commitAsync, or sendOffsetsToTransaction to commit offsets inside a producer transaction.
  • Storing offsets in the same database transaction as the results is a classic way to get exactly-once into an external store.

Pitfalls

  • Auto-commit combined with handing records to a thread pool loses records on a crash.
  • auto.offset.reset=latest silently skips a backlog when offsets expire; use earliest where completeness matters.
  • Health checks that restart consumers whose offset stops moving can cause rebalance loops.

Recall

These cards are also in your review deck, scheduled with spaced repetition.