Notesencoding🌱 seedling
Message Passing
Also: Asynchronous Messaging · Async Messaging · Message-Driven Communication
In asynchronous message passing a sender hands a message to an intermediary, usually a message broker, and carries on without waiting. The broker buffers, redelivers and fans out messages, decoupling sender and recipient in time and address.
Why it exists
Direct calls need both sides up at the same moment and couple the caller to the callee's address and speed. A broker absorbs outages and bursts, and lets new consumers subscribe without the sender changing.
How it works
- The sender publishes to a queue or topic on a Message Broker and moves on; communication is one-way unless you add a reply queue.
- The broker buffers while consumers are down, redelivers unacknowledged messages, and fans out to many consumers.
- Delivery is usually at least once, so consumers see duplicates and must be idempotent: record processed message IDs atomically with their effects, or make effects naturally idempotent.
- Messages are encoded bytes: producers and consumers evolve independently, often under a Schema Registry.
- Actor frameworks (Erlang/OTP, Akka, Orleans) apply the same model inside a program, assuming messages can be lost.
Trade-offs
| Gains | Costs |
|---|---|
| sender and receiver needn't be up together | no immediate reply; results arrive later, if at all |
| absorbs bursts and outages | another system to run; backlogs can hide problems |
| easy fan-out and new consumers | duplicates and reordering to handle |
In practice
- Apache Kafka: a log-based broker; its idempotent producer (default since Kafka 3.0) removes duplicates from producer retries, not consumer reprocessing.
- RabbitMQ and SQS: queue-based brokers that delete messages once acknowledged.
Pitfalls
- Assuming exactly-once processing end to end. Delivery is at least once; consumers deduplicate.
- A consumer that republishes messages and drops fields it doesn't understand.
Connections
- Part of
- Encoding
- Learn first
- Serialization
- Contrasts with
- RPC
- In real systems
- Apache KafkaRabbitMQ
Recall
These cards are also in your review deck, scheduled with spaced repetition.