Message Brokers & Logs
Put a durable buffer between the service that says something happened and the services that care. Queues hand out work and forget it; logs remember everything in order, and every reader keeps its own place.
The whole idea
Why put a broker in the middle
When an order is placed, a lot has to happen: charge the card, send the email, update search, feed the analytics warehouse, adjust the recommendations. The obvious design is for the orders service to call each of those services in turn. It works on day one and gets worse every month after.
A direct call couples three things that have no business being coupled. Availability: if email is down, orders fail. Speed: orders runs at the pace of its slowest dependency. Knowledge: every new consumer means a code change in the producer.
Direct calls vs a broker
A message broker is a server built to sit in that gap. The producer hands it a message and gets an acknowledgement once the broker has stored it durably. Consumers pull messages when they are ready. The broker absorbs bursts (a natural form of buffering against overload), survives consumer outages, and lets you add a consumer without touching the producer.
The price is that the producer no longer knows what happened. Sending is asynchronous: “order placed” has been recorded, but nobody has emailed anyone yet. Anything that needs an answer, like “was the card approved?”, is still a request. A broker is for facts that other services react to, and for work that can happen later.
Queues and logs
Brokers come in two families, and the difference shapes everything else.
A queue (RabbitMQ’s classic and quorum queues, Amazon SQS, ActiveMQ) is a to-do list. Each message goes to one consumer, which processes it and acknowledges it, and then the broker deletes it. Consumers compete for messages, so adding workers adds throughput. The broker tracks the state of every message: ready, delivered but unacked, or gone.
A classic queue
The details are where real systems differ:
- RabbitMQ pushes messages to consumers, up to a per-channel prefetch limit of unacked deliveries. Its docs suggest 100–300 as the usual sweet spot. Unacked messages are requeued when a consumer’s channel closes.
- SQS hides a received message for a visibility timeout (30 s by default, 12 h at most). If the consumer doesn’t delete it in time, it reappears for someone else. Standard queues are at-least-once with best-effort ordering. FIFO queues add per-group ordering and a 5-minute deduplication window.
A log (Kafka, Pulsar, Amazon Kinesis, RabbitMQ Streams) works the other way. Messages are appended to an ordered file and stay there until retention removes them, whether anyone has read them or not. A consumer doesn’t delete anything. It just remembers how far it has read.
| Queue | Log | |
|---|---|---|
| Unit the broker tracks | every message’s state | one offset per consumer group per partition |
| Reading a message | removes it (after ack) | changes nothing |
| Replay old messages | no | yes, from any retained offset |
| New consumer | sees only new messages | can start from the beginning |
| Ordering | best effort; lost on redelivery | strict, per partition |
| Parallelism | add consumers freely | capped by partition count |
| Slow message | others carry on | blocks its partition |
The log
Jay Kreps described the log as “perhaps the simplest possible storage abstraction… an append-only, totally-ordered sequence of records ordered by time.” The same structure already sits at the heart of databases, as the write-ahead log and the replication stream. A log-based broker takes that internal structure and makes it the product.
Three consequences follow from “reading doesn’t delete”:
- Reading is cheap. The broker keeps no per-message delivery state, only a number per reader. Kafka’s 2011
paper credited this, plus serving data with the
sendfilesystem call, for consuming about 4× faster than ActiveMQ and RabbitMQ in its benchmark. - Fan-out is free. Ten consumer groups reading the same topic don’t make ten copies. Each is a pointer.
- History is replayable. A bug in a consumer? Fix it, rewind its offset, reprocess. A new search index? Start it at offset 0. That’s what the hero diagram shows, and it makes the log a source of truth that derived systems can be rebuilt from.
What you give up is per-message freedom. A log consumer reads strictly in order, so it cannot skip one slow message and come back to it. Throughput comes from partitions instead of from more consumers on one stream.
Partitions and ordering
One ordered log on one machine can’t scale. So a Kafka topic is split into partitions. Each partition is an independent log, replicated across brokers using leader-based replication, and every record gets the next offset in its partition.
A partitioned log
The producer picks the partition. With a key, it hashes the key, so every record for user:ana lands in the
same partition and stays in order. Without a key, Kafka spreads records across partitions in sticky batches
(KIP-480 in 2.4, refined by KIP-794 in 3.3 because the first version sent more records to slower brokers).
That makes the key one of the most important decisions in a streaming design:
- Choose it by what must stay ordered. All events for one account, one order, one device. Events for different keys have no order relative to each other, and nothing downstream may assume they do.
- Watch for skew. A key that dominates traffic becomes a hot partition, and one consumer must keep up with all of it. Hash partitioning spreads keys, not load.
- Don’t change the partition count casually. Adding partitions changes
hash(key) mod n, so a key’s new events land somewhere else while its old ones stay put. Per-key order breaks across the change.
Consumer offsets
A log consumer’s entire state is a set of numbers: for each partition it reads, the offset of the next record it needs. Two of those numbers matter, and they are easy to confuse.
- The position is the next offset the consumer will fetch. It moves every time
poll()returns records. - The committed offset is the position the consumer has saved. After a crash or a rebalance, whoever takes over the partition resumes from here.
Kafka stores committed offsets in an internal, compacted topic, __consumer_offsets (50 partitions by
default). The consumer auto-commits by default, every 5 seconds, which commits whatever was last
returned by poll(), whether your code has finished with it or not. A group that has had no live members for
longer than offsets.retention.minutes (7 days) loses its committed offsets, and then auto.offset.reset decides where it restarts:
latest by default, which silently skips the backlog.
Because the committed offset is the consumer’s memory, where you commit relative to your side effects decides the delivery guarantee. More on that under Delivery semantics.
Consumer groups and rebalancing
Consumers that share a group.id form a consumer group and split the partitions between them. Each
partition has exactly one owner in the group, which keeps order intact. Different groups are independent:
billing and analytics each get every record.
Membership changes all the time: deploys, crashes, autoscaling, a long GC pause. Each change triggers a rebalance that reassigns partitions. How a group rebalances has been Kafka’s most-revised design.
Rebalancing a consumer group
- Eager (the original protocol). Every member revokes all its partitions, rejoins, waits for a leader member to compute a new assignment, and resumes. KIP-429 called it what it is: “stop-the-world.” In a big group, a rolling restart can trigger rebalance after rebalance.
- Static membership (KIP-345, Kafka 2.3). Give each instance a stable
group.instance.id, and a restart that returns within the session timeout keeps its partitions with no rebalance at all. - Incremental cooperative (KIP-429, Kafka 2.4). Members keep the partitions they will still own and give up only the ones that move, over two rounds.
- The new consumer protocol (KIP-848, GA in Kafka 4.0). The broker’s group coordinator computes the
assignment and delivers changes through each member’s heartbeat. There’s no group-wide barrier, and members
whose assignment didn’t change never notice. It’s opt-in on the client:
group.protocol=consumer.
Failure detection is the other half. A member that misses heartbeats for session.timeout.ms (45 s by
default) is removed. One that doesn’t call poll() within max.poll.interval.ms (5 min) is assumed stuck and
removed too. That second one bites when a batch takes longer than expected to process.
Delivery semantics
Everything in a distributed system can fail between two steps, and the broker can’t tell “the consumer is slow” from “the consumer died”. So any message may be delivered more than once, or, if you commit early, not at all.
Where you commit decides the guarantee
- At-most-once: commit, then process. A crash loses whatever was committed but not yet done. That’s acceptable for metrics sampling, and wrong for payments.
- At-least-once: process, then commit. A crash replays work. This is the default for almost every real system, so consumers must be idempotent. Upsert by a stable event ID, keep a processed-IDs table updated in the same transaction as the effect, or make the effect naturally idempotent.
- Exactly-once: the effect and the offset commit happen atomically, or the replay is harmless. Inside
Kafka this is real. The idempotent producer (KIP-98, Kafka 0.11, 2017) gives each producer an ID and
numbers its records per partition, so a retried send isn’t appended twice. Transactions let a
read-process-write job commit its output records and its input offsets as one atomic unit. Since Kafka 3.0,
producers default to
acks=alland idempotence on. Once the effect leaves Kafka (an email, a call to a payment API), you’re back to idempotency.
The producer side has the same problem in miniature. A producer whose send times out doesn’t know whether the broker stored the record. Retrying without idempotence can append a duplicate. Not retrying can lose it.
Poison messages and dead letter queues
Some messages will never succeed: a malformed payload, a schema the code doesn’t understand, a reference to a row that was deleted. Retrying them forever is worse than useless.
Poison message
A dead letter queue is where such messages go after a bounded number of attempts, so the stream can move on and a human can look later. Every broker has a version of it:
- SQS: a redrive policy moves a message to the DLQ after
maxReceiveCountreceives, and DLQ redrive moves messages back once the bug is fixed. - RabbitMQ: a dead-letter exchange receives messages that were rejected without requeue, expired by TTL, overflowed a length limit, or exceeded a quorum queue’s delivery limit. Since RabbitMQ 4.0 that limit defaults to 20.
- Kafka: consumers do it themselves (publish to an
orders.DLQtopic, then commit past the record). Kafka Connect has a built-in DLQ for sink connectors, and share groups gained one in Kafka 4.4.
Separate permanent failures (bad data: dead-letter it) from transient ones (database down: back off and retry, ideally with backpressure on the whole consumer). Dead-lettering a message because a dependency blipped just moves your outage into a queue nobody reads.
Log compaction
Time-based retention (retention.ms, 7 days by default) is right for event streams. For a stream of state
changes it’s wrong: delete old records by age and you lose keys that simply haven’t changed lately.
Log compaction
With cleanup.policy=compact, a background cleaner rewrites old segments and keeps only the latest record
for each key. Survivors keep their offsets and order. A record with a null value is a tombstone: it
marks the key deleted and is itself removed after delete.retention.ms (24 h by default). The guarantee is
that a consumer that stays caught up sees every change, and a consumer starting from offset 0 always ends with
the current value of every key. One that lags by more than delete.retention.ms can miss a delete.
That makes a compacted topic a table you can replay, which is why Kafka uses one for consumer offsets, Kafka Streams for its state-store changelogs, and Kafka Connect for its configs. It’s also how change data capture streams stay bootstrappable.
Trade-offs
| Choice | You get | You pay |
|---|---|---|
| Queue (RabbitMQ, SQS) | per-message acks, easy scaling, no partitions to plan, delays and priorities | no replay; ordering weak or bounded |
| Log (Kafka, Pulsar, Kinesis) | replay, cheap fan-out, strict per-key order, high throughput | parallelism capped by partitions; head-of-line blocking; key design matters |
| Many partitions | more parallelism | more files, more replication traffic, longer recovery |
| Long retention | replay and rebuilds from history | storage (tiered storage helps) |
| At-least-once + idempotent consumers | simple, robust | dedup logic in every consumer |
| Kafka transactions | exactly-once within Kafka | latency, complexity; stops at Kafka’s edge |
In the real world
- Kafka at LinkedIn. Kafka was built at LinkedIn and described at NetDB 2011. By 2019 it ran over 100 clusters, 100,000 topics and 7 million partitions, and handled 7 trillion messages a day. In 2025 LinkedIn reported over 32 trillion records (17 PB) a day across 150 clusters, and introduced Northguard, a log system that replicates per segment and shards its metadata as Raft-backed state machines, to get past the limits of very large Kafka clusters.
- Kafka itself, 2022–2026. KRaft replaced ZooKeeper with a Raft-based metadata quorum. It became production-ready in 3.3 (2022), and Kafka 4.0 (March 2025) dropped ZooKeeper entirely. Tiered storage (KIP-405), which moves old segments to object storage, became production-ready in 3.9 (November 2024). KIP-848 rebalancing went GA in 4.0, and share groups in 4.2. Diskless topics written straight to object storage (KIP-1150) were accepted in March 2026 but haven’t shipped.
- Slack’s job queue (2017). Slack ran 1.4 billion jobs a day, peaking at 33,000 a second, on Redis. When slow database writes backed jobs up, Redis hit its memory limit, and because dequeuing also needed memory, the queue jammed until engineers intervened by hand. The fix put Kafka in front of Redis as a durable buffer: 16 brokers, 32 partitions per topic, replication factor 3, 2-day retention.
- Uber’s consumer proxy (2021). Hundreds of Uber services consume Kafka as a work queue. Poison pills blocked partitions, parallelism was capped by partition count, and rolling restarts set off “rebalance storms”. Uber built a proxy that consumes on their behalf, tracks acks per message, commits out of order and dead-letters failures. In short, queue semantics on a log, years before share groups.
- RabbitMQ 4.0 (2024). Classic mirrored queues, deprecated for three years, were removed in favour of Raft-based quorum queues (since 3.8). RabbitMQ Streams (since 3.9) add a replayable log alongside queues.
- Pulsar separates serving from storage: brokers are stateless, and data lives in Apache BookKeeper. A topic can be consumed as a stream (exclusive or failover subscriptions) or as a queue (shared or key-shared subscriptions, with per-message acks).
Pitfalls
- Assuming global order. Kafka orders records within a partition only. Two events for different keys can be consumed in either order.
- Auto-commit plus async processing. Committing what
poll()returned is not committing what you finished. Hand records to a thread pool and a crash loses them. - Non-idempotent consumers. At-least-once is the norm, so duplicates are a when, not an if. Exactly-once in Kafka does not extend to your database or your email provider.
- Health checks on offsets. Cloudflare (2023) restarted consumers whose committed offset wasn’t advancing. Every restart caused a rebalance, which stalled offsets again, which caused more restarts.
- Slow batches and max.poll.interval.ms. A consumer that takes longer than the interval to process a batch is kicked out, its partitions move, the batch is redone elsewhere, and it can loop.
- A DLQ with no alert. It’s still data loss, just with extra steps.
Recall
Go deeper
- Kreps, Narkhede & Rao, “Kafka: a Distributed Messaging System for Log Processing” (NetDB 2011): the original design, and why stateless consumers and
sendfilebeat per-message tracking. - Jay Kreps, “The Log: What every software engineer should know about real-time data’s unifying abstraction” (2013): the essay that made the log an architecture, not just a data structure.
- KIP-848: The Next Generation of the Consumer Rebalance Protocol: why the group-wide barrier had to go, and how broker-side assignment works.
- KIP-932: Queues for Kafka: share groups, per-record acknowledgement and delivery counts on top of a log.
- Neha Narkhede, “Exactly-once Semantics Are Possible: Here’s How Kafka Does It” (2017): the idempotent producer and transactions, and exactly where the guarantee ends.
- RabbitMQ docs: Consumer Acknowledgements and Publisher Confirms and Dead Letter Exchanges: the queue model done carefully.
- Uber, “Enabling Seamless Kafka Async Queuing with Consumer Proxy” (2021): what goes wrong using a log as a work queue at scale, and the fix.
- Slack, “Scaling Slack’s Job Queue” (2017): a real overload incident and why a durable log went in front of Redis.
Keep going
The atomic notes for this lesson live in the knowledge graph:
Next on the map: Stream Processing (computing over these logs continuously: windows, watermarks, state and exactly-once) and Change Data Capture & Event Sourcing (turning a database into a log, and a log into a database).