Notesstream🌱 seedling

Consumer Group

Also: Consumer Group Rebalance · Cooperative Rebalancing · Static Membership · Share Group · KIP-848

Consumers sharing a group ID split a topic's partitions so that each partition has exactly one owner in the group. A rebalance reassigns partitions as members join, leave or die, and how it's done decides whether the whole group pauses.

Why it exists

One consumer can't keep up with a busy topic, but per-key order requires one reader per partition. A group gives parallelism across partitions and exclusivity within each. Separate groups each receive every record.

How it works

  • A broker-side group coordinator tracks members through heartbeats and assigns partitions.
  • Eager rebalance (original): every member revokes all partitions, rejoins, and waits for the new assignment. The whole group stops.
  • Cooperative (Kafka 2.4, KIP-429): members keep unchanged partitions and give up only those that move.
  • Static membership (Kafka 2.3, KIP-345): a stable group.instance.id lets a restarted member reclaim its partitions with no rebalance.
  • KIP-848 (GA in Kafka 4.0, opt-in via group.protocol=consumer): the coordinator computes assignments and pushes changes through heartbeats, with no group-wide barrier.
  • Members are removed after session.timeout.ms (45 s) without heartbeats, or after max.poll.interval.ms (5 min) without polling.

Trade-offs

Gains Costs
horizontal scaling with per-partition order parallelism ≤ partition count; extras idle
automatic failover of partitions rebalances pause work; duplicates replay from last commit
independent groups, each at its own offset tuning timeouts trades detection speed for false evictions

In practice

  • Share groups (KIP-932, production-ready in Kafka 4.2) add queue semantics: per-record acks, delivery-count limits, and more consumers than partitions, at the cost of ordering.
  • Uber built a consumer proxy with per-message acks and DLQs after hitting rebalance storms and the partition cap.

Pitfalls

  • A batch that takes longer than max.poll.interval.ms gets the member evicted, and the batch is redone elsewhere, possibly in a loop.
  • Rolling restarts of large groups under the eager protocol cause back-to-back rebalances.
  • Restarting consumers because their offset stopped moving can itself trigger endless rebalances.

Recall

These cards are also in your review deck, scheduled with spaced repetition.