Notesstream🌱 seedling
Delivery Semantics
Also: Delivery Guarantee · At-Least-Once · At-Most-Once · Message Delivery Semantics · Idempotent Consumer
Because any step can fail between "received" and "acknowledged", a message is delivered at-most-once (may be lost), at-least-once (may be duplicated) or effectively exactly-once (duplicates are made harmless). At-least-once with idempotent consumers is the industry default.
Why it exists
A broker can't tell a slow consumer from a dead one, and a producer whose send timed out can't tell whether it was stored. Every retry that prevents loss risks a duplicate.
How it works
- At-most-once: ack or commit first, then process. A crash loses work. Fine for sampled metrics.
- At-least-once: process, then ack or commit. A crash replays work, so consumers must be idempotent.
- Exactly-once: the effect and the progress marker commit atomically (Kafka transactions, offsets stored with results), or the effect is deduplicated. See Exactly-Once Semantics.
- Producer side: retries after a timeout can append duplicates. Kafka's idempotent producer (on by default since 3.0) numbers records per partition, so the broker drops retried duplicates.
- Making consumers idempotent: upsert by event ID, keep a processed-ID table in the same transaction as the effect, or use naturally idempotent writes.
Trade-offs
| Guarantee | Gains | Costs |
|---|---|---|
| at-most-once | simplest, lowest latency | data loss on failure |
| at-least-once | no loss | duplicates; dedup in every consumer |
| exactly-once | no loss, no visible duplicates | latency, coordination; stops at system boundaries |
In practice
- SQS standard queues are at-least-once with best-effort ordering; FIFO queues dedupe within a 5-minute window.
- Kafka: acks=all plus idempotence are the producer defaults since 3.0 (fully effective from 3.0.1 / 3.2).
Pitfalls
- Believing a broker's "exactly-once" covers your database or email provider. It ends at the broker's boundary.
- Generating the idempotency key per attempt instead of per event makes deduplication useless.
Connections
- Part of
- Message Broker
- Builds on this
- Dead Letter Queue
- Related
- Consumer OffsetIdempotencyExactly-Once SemanticsRetry with BackoffTransactional OutboxEnd-to-End ArgumentMessage Passing
- In real systems
- Apache KafkaRabbitMQ
Recall
These cards are also in your review deck, scheduled with spaced repetition.