NotesSystem

Apache Kafka

Also: Kafka · KRaft

A distributed, partitioned, replicated commit log, built at LinkedIn (2011). Records stay after they are read, and consumers are just offsets, which makes replay, fan-out and stream processing cheap.

Also in: Message Brokers & Logs › In the real world

Design choices

  • Log-Based Message Broker: topics split into partitions. Each is an append-only log with per-partition order; keys pick partitions by hash.
  • Leader-Based Replication: each partition has a leader and followers. Producers default to acks=all, so a write is acknowledged once all in-sync replicas have it.
  • Metadata by Raft (KRaft): production-ready in 3.3 (2022). Kafka 4.0 (March 2025) removed ZooKeeper entirely.
  • Consumer Groups: eager, then cooperative (2.4), then KIP-848's broker-driven protocol (GA in 4.0). Share groups (KIP-932, production-ready in 4.2) add queue semantics.
  • Consumer Offsets live in the compacted __consumer_offsets topic.
  • Exactly-Once Semantics: idempotent producer and transactions (KIP-98, 0.11, 2017). Idempotence is on by default since 3.0.
  • Retention: by time (7 days by default) or by key via Log Compaction. Tiered storage (KIP-405, production-ready in 3.9) moves old segments to object storage.

Notable trade-offs

  • Order only within a partition, and consumer parallelism capped by partition count (except in share groups).
  • A poison record blocks its partition. Dead-lettering is the application's job.
  • Exactly-once covers read-process-write within Kafka, not external side effects.
  • Operating it means planning partitions, replication and retention. Hence managed and object-storage-based variants (WarpStream; diskless topics, KIP-1150, accepted in 2026).

Scale

  • LinkedIn, 2019: 100+ clusters, 4,000+ brokers, 7 million partitions, 7 trillion messages a day. By 2025: over 32 trillion records (17 PB) a day.