Notesstream🌱 seedling

Log-Based Message Broker

Also: Log-Based Broker · Event Log · Distributed Log · Commit Log · Event Streaming Platform · Topic Partition

A broker that stores messages as an append-only, ordered, partitioned log and keeps them until retention removes them, whether or not anyone has read them. Consumers are just positions in the log, so reading is cheap, fan-out is free and history can be replayed.

Also in: Message Brokers & Logs › Partitions and ordering

Why it exists

Queue brokers forget a message once it's acked: no replay, no second reader, no ordering guarantees after redelivery. Databases already had a durable ordered log inside them (the Write-Ahead Log, the Replication Log). A log broker makes that structure the product.

How it works

  • A topic is split into partitions. Each partition is an append-only log, replicated with Leader-Based Replication, and every record gets the next offset.
  • The producer chooses the partition, usually by Hash Partitioning the record key, so all records for a key share a partition and stay in order.
  • Reading changes nothing on the broker. Each reader stores a Consumer Offset.
  • Parallelism comes from partitions: a Consumer Group assigns each partition to one member.
  • Records leave by retention (Kafka default 7 days), not by consumption, or by Log Compaction for keyed state.

Trade-offs

Gains Costs
replay from any retained offset; new readers start at 0 order only per partition, never across
many consumer groups at no extra copy parallelism capped by partition count
high throughput (sequential I/O, no per-message state) a slow or poison record blocks its partition

In practice

  • Kafka at LinkedIn (2019): 100+ clusters, 4,000+ brokers, 7 million partitions, 7 trillion messages a day.
  • Pulsar separates stateless brokers from BookKeeper storage, and its shared subscriptions add queue semantics on top.

Pitfalls

  • Assuming global ordering across partitions.
  • Adding partitions changes hash(key) mod n, so a key's new records move to a different partition from its old ones.
  • A hot key creates a hot partition that one consumer must handle alone.

Recall

These cards are also in your review deck, scheduled with spaced repetition.