Notesstream🌱 seedling

Exactly-Once Semantics

Also: Exactly-Once · EOS · Effectively-Once · Exactly-Once Processing · Kafka Transactions · Idempotent Producer · Two-Phase Commit Sink

Records may be delivered and processed more than once, but the effect on state and output is as if each were processed once. It needs the progress marker (offsets, checkpoint) and the effect to commit atomically, or the effect to be idempotent.

Why it exists

At-least-once is the safe default, but duplicates corrupt counts, balances and billing. Deduplicating by hand in every consumer is error-prone.

How it works

  • Within Kafka: the idempotent producer deduplicates retried appends. Transactions commit output records and consumed offsets atomically. Consumers must read with isolation.level=read_committed (the default is read_uncommitted).
  • Kafka Streams: processing.guarantee=exactly_once_v2 puts offsets, state changelogs and output in one transaction per commit.
  • Flink end to end: checkpoints make state exact. A two-phase-commit sink writes into a transaction, pre-commits on the checkpoint barrier and commits when the checkpoint completes. A crash aborts open transactions.
  • Outside transactional systems: make the effect idempotent with a key derived from the event, or store offsets in the same database transaction as the result.

Trade-offs

Gains Costs
no duplicate effects, no loss output visible only per commit or checkpoint (latency)
simpler consumers downstream transactions, timeouts and fencing to operate
stops at the boundary of systems that can join the transaction

In practice

  • Flink's Kafka sink defaults to a 1-hour transaction timeout, but brokers cap transaction.max.timeout.ms at 15 minutes by default. Raise one or lower the other, and keep it above checkpoint plus recovery time, or data can be lost.
  • Kafka producers default to idempotence and acks=all since 3.0, so duplicate appends from retries are already handled; transactions are still opt-in.

Pitfalls

  • Downstream consumers left on read_uncommitted see aborted records.
  • Side effects like emails and HTTP calls aren't covered. They need their own idempotency.
  • "Exactly-once delivery" over a network is impossible. What's achievable is exactly-once effect.

Recall

These cards are also in your review deck, scheduled with spaced repetition.