Stream Processing · 1 of 4

Message Brokers & Logs

Put a durable buffer between the service that says something happened and the services that care. Queues hand out work and forget it; logs remember everything in order, and every reader keeps its own place.

≈ 25 min

The whole idea

Press ← → to step, or Space to play and pause, whenever a diagram has focus.

Why put a broker in the middle

When an order is placed, a lot has to happen: charge the card, send the email, update search, feed the analytics warehouse, adjust the recommendations. The obvious design is for the orders service to call each of those services in turn. It works on day one and gets worse every month after.

A direct call couples three things that have no business being coupled. Availability: if email is down, orders fail. Speed: orders runs at the pace of its slowest dependency. Knowledge: every new consumer means a code change in the producer.

Direct calls vs a broker

The broker turns 'email must be up for orders to work' into 'email must eventually catch up'.

A message broker is a server built to sit in that gap. The producer hands it a message and gets an acknowledgement once the broker has stored it durably. Consumers pull messages when they are ready. The broker absorbs bursts (a natural form of buffering against overload), survives consumer outages, and lets you add a consumer without touching the producer.

The price is that the producer no longer knows what happened. Sending is asynchronous: “order placed” has been recorded, but nobody has emailed anyone yet. Anything that needs an answer, like “was the card approved?”, is still a request. A broker is for facts that other services react to, and for work that can happen later.

Queues and logs

Brokers come in two families, and the difference shapes everything else.

A queue (RabbitMQ’s classic and quorum queues, Amazon SQS, ActiveMQ) is a to-do list. Each message goes to one consumer, which processes it and acknowledges it, and then the broker deletes it. Consumers compete for messages, so adding workers adds throughput. The broker tracks the state of every message: ready, delivered but unacked, or gone.

A classic queue

Acks delete messages, and redelivery happens one message at a time, so order is the first thing to go.

The details are where real systems differ:

  • RabbitMQ pushes messages to consumers, up to a per-channel prefetch limit of unacked deliveries. Its docs suggest 100–300 as the usual sweet spot. Unacked messages are requeued when a consumer’s channel closes.
  • SQS hides a received message for a visibility timeout (30 s by default, 12 h at most). If the consumer doesn’t delete it in time, it reappears for someone else. Standard queues are at-least-once with best-effort ordering. FIFO queues add per-group ordering and a 5-minute deduplication window.

A log (Kafka, Pulsar, Amazon Kinesis, RabbitMQ Streams) works the other way. Messages are appended to an ordered file and stay there until retention removes them, whether anyone has read them or not. A consumer doesn’t delete anything. It just remembers how far it has read.

Queue Log
Unit the broker tracks every message’s state one offset per consumer group per partition
Reading a message removes it (after ack) changes nothing
Replay old messages no yes, from any retained offset
New consumer sees only new messages can start from the beginning
Ordering best effort; lost on redelivery strict, per partition
Parallelism add consumers freely capped by partition count
Slow message others carry on blocks its partition

The log

Jay Kreps described the log as “perhaps the simplest possible storage abstraction… an append-only, totally-ordered sequence of records ordered by time.” The same structure already sits at the heart of databases, as the write-ahead log and the replication stream. A log-based broker takes that internal structure and makes it the product.

Three consequences follow from “reading doesn’t delete”:

  1. Reading is cheap. The broker keeps no per-message delivery state, only a number per reader. Kafka’s 2011 paper credited this, plus serving data with the sendfile system call, for consuming about 4× faster than ActiveMQ and RabbitMQ in its benchmark.
  2. Fan-out is free. Ten consumer groups reading the same topic don’t make ten copies. Each is a pointer.
  3. History is replayable. A bug in a consumer? Fix it, rewind its offset, reprocess. A new search index? Start it at offset 0. That’s what the hero diagram shows, and it makes the log a source of truth that derived systems can be rebuilt from.

What you give up is per-message freedom. A log consumer reads strictly in order, so it cannot skip one slow message and come back to it. Throughput comes from partitions instead of from more consumers on one stream.

Partitions and ordering

One ordered log on one machine can’t scale. So a Kafka topic is split into partitions. Each partition is an independent log, replicated across brokers using leader-based replication, and every record gets the next offset in its partition.

A partitioned log

P0ana, diana0di1ana2ana3di4P1bo, evebo0bo1eve2bo3P2cy, faycy0cy1fay2cy3Each cell: key on top, offset below. Write position is always the end of the log.
Order exists within a partition, never across them. The key decides which partition, and so which order, a record joins.

The producer picks the partition. With a key, it hashes the key, so every record for user:ana lands in the same partition and stays in order. Without a key, Kafka spreads records across partitions in sticky batches (KIP-480 in 2.4, refined by KIP-794 in 3.3 because the first version sent more records to slower brokers).

That makes the key one of the most important decisions in a streaming design:

  • Choose it by what must stay ordered. All events for one account, one order, one device. Events for different keys have no order relative to each other, and nothing downstream may assume they do.
  • Watch for skew. A key that dominates traffic becomes a hot partition, and one consumer must keep up with all of it. Hash partitioning spreads keys, not load.
  • Don’t change the partition count casually. Adding partitions changes hash(key) mod n, so a key’s new events land somewhere else while its old ones stay put. Per-key order breaks across the change.

Consumer offsets

A log consumer’s entire state is a set of numbers: for each partition it reads, the offset of the next record it needs. Two of those numbers matter, and they are easy to confuse.

  • The position is the next offset the consumer will fetch. It moves every time poll() returns records.
  • The committed offset is the position the consumer has saved. After a crash or a rebalance, whoever takes over the partition resumes from here.

Kafka stores committed offsets in an internal, compacted topic, __consumer_offsets (50 partitions by default). The consumer auto-commits by default, every 5 seconds, which commits whatever was last returned by poll(), whether your code has finished with it or not. A group that has had no live members for longer than offsets.retention.minutes (7 days) loses its committed offsets, and then auto.offset.reset decides where it restarts: latest by default, which silently skips the backlog.

Because the committed offset is the consumer’s memory, where you commit relative to your side effects decides the delivery guarantee. More on that under Delivery semantics.

Consumer groups and rebalancing

Consumers that share a group.id form a consumer group and split the partitions between them. Each partition has exactly one owner in the group, which keeps order intact. Different groups are independent: billing and analytics each get every record.

Membership changes all the time: deploys, crashes, autoscaling, a long GC pause. Each change triggers a rebalance that reassigns partitions. How a group rebalances has been Kafka’s most-revised design.

Rebalancing a consumer group

Steady state6/6 partitions being consumed
C1P0P1
C2P2P3
C3P4P5
Unassignednone
Eager rebalances stop everything. Incremental ones move only what has to move.
  • Eager (the original protocol). Every member revokes all its partitions, rejoins, waits for a leader member to compute a new assignment, and resumes. KIP-429 called it what it is: “stop-the-world.” In a big group, a rolling restart can trigger rebalance after rebalance.
  • Static membership (KIP-345, Kafka 2.3). Give each instance a stable group.instance.id, and a restart that returns within the session timeout keeps its partitions with no rebalance at all.
  • Incremental cooperative (KIP-429, Kafka 2.4). Members keep the partitions they will still own and give up only the ones that move, over two rounds.
  • The new consumer protocol (KIP-848, GA in Kafka 4.0). The broker’s group coordinator computes the assignment and delivers changes through each member’s heartbeat. There’s no group-wide barrier, and members whose assignment didn’t change never notice. It’s opt-in on the client: group.protocol=consumer.

Failure detection is the other half. A member that misses heartbeats for session.timeout.ms (45 s by default) is removed. One that doesn’t call poll() within max.poll.interval.ms (5 min) is assumed stuck and removed too. That second one bites when a batch takes longer than expected to process.

Delivery semantics

Everything in a distributed system can fail between two steps, and the broker can’t tell “the consumer is slow” from “the consumer died”. So any message may be delivered more than once, or, if you commit early, not at all.

Where you commit decides the guarantee

Process, then commitat-least-oncerunning0✓1✓2✓345678committedpositionCommit, then processat-most-oncerunning0✓1✓2✓345678committedposition
The same crash, two commit orders. Neither is exactly-once on its own.
  • At-most-once: commit, then process. A crash loses whatever was committed but not yet done. That’s acceptable for metrics sampling, and wrong for payments.
  • At-least-once: process, then commit. A crash replays work. This is the default for almost every real system, so consumers must be idempotent. Upsert by a stable event ID, keep a processed-IDs table updated in the same transaction as the effect, or make the effect naturally idempotent.
  • Exactly-once: the effect and the offset commit happen atomically, or the replay is harmless. Inside Kafka this is real. The idempotent producer (KIP-98, Kafka 0.11, 2017) gives each producer an ID and numbers its records per partition, so a retried send isn’t appended twice. Transactions let a read-process-write job commit its output records and its input offsets as one atomic unit. Since Kafka 3.0, producers default to acks=all and idempotence on. Once the effect leaves Kafka (an email, a call to a payment API), you’re back to idempotency.

The producer side has the same problem in miniature. A producer whose send times out doesn’t know whether the broker stored the record. Retrying without idempotence can append a duplicate. Not retrying can lose it.

Poison messages and dead letter queues

Some messages will never succeed: a malformed payload, a schema the code doesn’t understand, a reference to a row that was deleted. Retrying them forever is worse than useless.

Poison message

In a log, one message that always fails stalls everything behind it in its partition.

A dead letter queue is where such messages go after a bounded number of attempts, so the stream can move on and a human can look later. Every broker has a version of it:

  • SQS: a redrive policy moves a message to the DLQ after maxReceiveCount receives, and DLQ redrive moves messages back once the bug is fixed.
  • RabbitMQ: a dead-letter exchange receives messages that were rejected without requeue, expired by TTL, overflowed a length limit, or exceeded a quorum queue’s delivery limit. Since RabbitMQ 4.0 that limit defaults to 20.
  • Kafka: consumers do it themselves (publish to an orders.DLQ topic, then commit past the record). Kafka Connect has a built-in DLQ for sink connectors, and share groups gained one in Kafka 4.4.

Separate permanent failures (bad data: dead-letter it) from transient ones (database down: back off and retry, ideally with backpressure on the whole consumer). Dead-lettering a message because a dependency blipped just moves your outage into a queue nobody reads.

Log compaction

Time-based retention (retention.ms, 7 days by default) is right for event streams. For a stream of state changes it’s wrong: delete old records by age and you lose keys that simply haven’t changed lately.

Log compaction

Topic user-settingskey = user, value = theme or language
offset 0ana=darkoffset 1eve=enoffset 2bo=enoffset 3ana=lightoffset 4cy=froffset 5bo=deoffset 6ana=darkoffset 7cy=nulloffset 8bo=es
Consumer’s table built by reading the topic from offset 0
not built yet
Compaction keeps the latest record per key, so the topic can always be replayed into the current table.

With cleanup.policy=compact, a background cleaner rewrites old segments and keeps only the latest record for each key. Survivors keep their offsets and order. A record with a null value is a tombstone: it marks the key deleted and is itself removed after delete.retention.ms (24 h by default). The guarantee is that a consumer that stays caught up sees every change, and a consumer starting from offset 0 always ends with the current value of every key. One that lags by more than delete.retention.ms can miss a delete.

That makes a compacted topic a table you can replay, which is why Kafka uses one for consumer offsets, Kafka Streams for its state-store changelogs, and Kafka Connect for its configs. It’s also how change data capture streams stay bootstrappable.

Trade-offs

Choice You get You pay
Queue (RabbitMQ, SQS) per-message acks, easy scaling, no partitions to plan, delays and priorities no replay; ordering weak or bounded
Log (Kafka, Pulsar, Kinesis) replay, cheap fan-out, strict per-key order, high throughput parallelism capped by partitions; head-of-line blocking; key design matters
Many partitions more parallelism more files, more replication traffic, longer recovery
Long retention replay and rebuilds from history storage (tiered storage helps)
At-least-once + idempotent consumers simple, robust dedup logic in every consumer
Kafka transactions exactly-once within Kafka latency, complexity; stops at Kafka’s edge

In the real world

7 trillionmessages a day through LinkedIn's Kafka in 2019, on 4,000+ brokers
32 trillionrecords a day through LinkedIn's Kafka by 2025, across 150 clusters
33,000jobs per second at peak in Slack's job queue, 2017
  • Kafka at LinkedIn. Kafka was built at LinkedIn and described at NetDB 2011. By 2019 it ran over 100 clusters, 100,000 topics and 7 million partitions, and handled 7 trillion messages a day. In 2025 LinkedIn reported over 32 trillion records (17 PB) a day across 150 clusters, and introduced Northguard, a log system that replicates per segment and shards its metadata as Raft-backed state machines, to get past the limits of very large Kafka clusters.
  • Kafka itself, 2022–2026. KRaft replaced ZooKeeper with a Raft-based metadata quorum. It became production-ready in 3.3 (2022), and Kafka 4.0 (March 2025) dropped ZooKeeper entirely. Tiered storage (KIP-405), which moves old segments to object storage, became production-ready in 3.9 (November 2024). KIP-848 rebalancing went GA in 4.0, and share groups in 4.2. Diskless topics written straight to object storage (KIP-1150) were accepted in March 2026 but haven’t shipped.
  • Slack’s job queue (2017). Slack ran 1.4 billion jobs a day, peaking at 33,000 a second, on Redis. When slow database writes backed jobs up, Redis hit its memory limit, and because dequeuing also needed memory, the queue jammed until engineers intervened by hand. The fix put Kafka in front of Redis as a durable buffer: 16 brokers, 32 partitions per topic, replication factor 3, 2-day retention.
  • Uber’s consumer proxy (2021). Hundreds of Uber services consume Kafka as a work queue. Poison pills blocked partitions, parallelism was capped by partition count, and rolling restarts set off “rebalance storms”. Uber built a proxy that consumes on their behalf, tracks acks per message, commits out of order and dead-letters failures. In short, queue semantics on a log, years before share groups.
  • RabbitMQ 4.0 (2024). Classic mirrored queues, deprecated for three years, were removed in favour of Raft-based quorum queues (since 3.8). RabbitMQ Streams (since 3.9) add a replayable log alongside queues.
  • Pulsar separates serving from storage: brokers are stateless, and data lives in Apache BookKeeper. A topic can be consumed as a stream (exclusive or failover subscriptions) or as a queue (shared or key-shared subscriptions, with per-message acks).

Pitfalls

  • Assuming global order. Kafka orders records within a partition only. Two events for different keys can be consumed in either order.
  • Auto-commit plus async processing. Committing what poll() returned is not committing what you finished. Hand records to a thread pool and a crash loses them.
  • Non-idempotent consumers. At-least-once is the norm, so duplicates are a when, not an if. Exactly-once in Kafka does not extend to your database or your email provider.
  • Health checks on offsets. Cloudflare (2023) restarted consumers whose committed offset wasn’t advancing. Every restart caused a rebalance, which stalled offsets again, which caused more restarts.
  • Slow batches and max.poll.interval.ms. A consumer that takes longer than the interval to process a batch is kicked out, its partitions move, the batch is redone elsewhere, and it can loop.
  • A DLQ with no alert. It’s still data loss, just with extra steps.

Recall

Go deeper

Keep going

The atomic notes for this lesson live in the knowledge graph:

Next on the map: Stream Processing (computing over these logs continuously: windows, watermarks, state and exactly-once) and Change Data Capture & Event Sourcing (turning a database into a log, and a log into a database).

Done with this lesson?

Mark it learned and the concepts it teaches will light up in your galaxy.

Next in Stream ProcessingStream Processing