Notesstream🌱 seedling

End-to-End Argument

Also: End-to-End Principle · End-to-End Arguments · End-to-End Correctness

A function like duplicate suppression or integrity checking can only be implemented completely by the application at the endpoints. Reliability in lower layers is a performance optimisation that makes failures rarer; it never replaces the end-to-end check.

Why it exists

Every hop can be reliable and the whole still wrong. Corruption can happen between hops (in a buffer, on a disk, in your code), and duplicates can be created above the layer that deduplicates (a user retrying).

How it works

  • Careful file transfer: disks, buffers, memory, the network and host crashes can each corrupt a copy, so only reading the result back and comparing an end-to-end checksum proves it.
  • Duplicate suppression: TCP removes duplicate packets within one connection, but a client retry after a lost response is a new connection. Only an operation ID generated at the origin and checked at the destination catches it.
  • In streams, the request ID travels through every hop, and the final write enforces it (a unique constraint, or a dedup table updated in the same transaction).
  • Lower-layer reliability still pays for itself by making end-to-end retries rare.

Trade-offs

Gains Costs
correctness that survives any intermediate bug IDs and dedup state at every sink
freedom to use simple, unreliable middle layers the application must reason about retries

In practice

  • Stripe idempotency keys: the client generates a unique key per operation, and the server stores it with the result and returns that result on retries.
  • Kafka exactly-once covers reads and writes inside Kafka. It doesn't make a client's retried HTTP request, or a non-transactional sink, exactly once.

Pitfalls

  • Believing "exactly-once" in the broker means exactly-once for the user.
  • Generating the dedup ID in the middle (in the app server, per attempt) instead of once at the true origin.

Recall

These cards are also in your review deck, scheduled with spaced repetition.