Modes of Dataflow
Encoded bytes travel between processes in a few distinct ways, and each one fails differently. Databases, REST and RPC, message brokers and durable workflows, and the idempotency and versioning that keep them honest.
The whole idea
Three modes of dataflow
Whenever data leaves a process for another one that doesn’t share its memory, it is encoded into bytes, and somebody else decodes it. The previous lesson was about the bytes. This one is about the journey: who encodes, who decodes, whether they run at the same time, and what happens when something fails in between. That decides which compatibility you need and which failures you have to handle.
There are three classic modes, and a fourth that has grown up in the last decade:
- Through a database. The writer encodes, and a reader decodes later, possibly much later.
- Through service calls, with REST or RPC. A client sends a request and waits for a response. Both sides must be up at once.
- Through asynchronous messages. A sender hands a message to a broker, which delivers it when the recipient is ready.
- Through a durable workflow. An engine records every step of a long-running process so that code can resume it after a crash, days or months later.
Dataflow through databases
Writing to a database is sending a message to your future self. The reader may be the same service a second later, or code that doesn’t exist yet. So databases need backward compatibility for as long as old rows exist, and, during rolling deploys, forward compatibility too, because old code will read rows that new code wrote.
The subtle failure is the one from the encoding lesson: old code reads a row, decodes it into an object that doesn’t know a new column, and writes it back without that column. ORMs and mappers make this easy to do by accident.
Migrations are where data and code meet. Rewriting a billion rows to a new schema is slow and risky, so most databases avoid it: adding a nullable column, or since PostgreSQL 11 a column with a constant default, only changes metadata, and old rows report the default when read. In effect the database does schema resolution for you, which is why “data outlives code” works at all.
REST
REST is the architectural style Roy Fielding described in his 2000 dissertation, distilled from how the web itself works. Its constraints are a client–server split, stateless requests (each carries everything needed to serve it), cacheable responses, a layered system that intermediaries can join, an optional code-on-demand, and above all a uniform interface: resources named by URLs and manipulated through a small, fixed set of methods and self-describing representations.
In practice “a REST API” usually means JSON over HTTP, with resource-shaped URLs and HTTP methods as verbs.
Its strength is everything HTTP already gives you: caches, proxies, load balancers, browsers, curl, and
decades of tooling. An OpenAPI document describes the interface and generates clients and docs.
The method semantics matter more than they look. HTTP defines safe methods (GET, HEAD, OPTIONS) that shouldn’t change anything, and idempotent ones, where repeating a request has the same effect as sending it once: PUT, DELETE and the safe methods. POST is neither, which is why “create a charge” needs extra help to retry safely.Fielding’s full style also includes hypermedia, where responses carry the links to what you can do next. Most JSON APIs skip that part.
RPC
Remote procedure calls take the opposite approach: make a call to another machine look exactly like a local function call. The idea goes back to the early 1980s (Birrell and Nelson’s 1984 paper made it practical) and resurfaced in CORBA, Java RMI, DCOM and SOAP. Most of those ended badly, and a 1994 Sun Microsystems paper, “A Note on Distributed Computing”, explained why: a remote call is fundamentally different from a local one, so hiding the difference hides the problems.
- Latency is unpredictable, from microseconds to seconds, and you can’t fix that by optimizing the caller.
- No shared memory. Arguments must be encoded and copied; pointers mean nothing on the other side.
- Concurrency becomes real: the server is handling other callers while it handles you.
- Partial failure. A local call returns or throws. A remote call can also do neither.
The third outcome
So modern RPC frameworks stopped pretending. They make the network explicit: calls take deadlines, return status codes rather than just values or exceptions, and can be asynchronous or streaming. They also lean on a schema-driven encoding with clear evolution rules, so client and server can be deployed independently.
gRPC
gRPC is Google’s open-source RPC framework, released in February 2015 as the successor to Stubby, the
internal RPC system Google had used between its services for over a decade. It runs over HTTP/2 and uses
Protocol Buffers for both the interface definition and the messages. A service is a .proto file:
service Payments {
rpc Charge(ChargeRequest) returns (ChargeReply);
rpc WatchBalance(Account) returns (stream Balance);
}
From it, gRPC generates client stubs and server skeletons in many languages, and offers four shapes of call:
Four kinds of call
Two properties stand out for a senior engineer:
- Deadlines are first class. A client says how long it will wait; the remaining time travels with the
call and can be propagated to downstream calls. When it expires, the call ends with
DEADLINE_EXCEEDED. The overload lesson covers why the default of no deadline is a trap. - Success is decided independently on each side. gRPC’s own docs say client and server “make independent and local determinations of the success of the call”. A server can finish while the client has already given up: the diagram above, as a documented guarantee.
Compatibility in RPC has a direction. Servers are usually upgraded before clients, so requests need backward compatibility (new servers read old clients’ requests) and responses need forward compatibility (old clients read new servers’ responses). Protobuf’s tag rules give you both, as long as nobody reuses a field number.
REST and RPC coexist comfortably: JSON over HTTP for public APIs, where reach and debuggability matter, and gRPC between internal services, where efficiency, streaming and generated types do.
Idempotency
Because a timeout’s outcome is unknown, every robust client retries, and every retry might be a duplicate. An operation is idempotent if doing it twice has the same effect as doing it once. “Set the address to X” and “delete order 7” are naturally idempotent. “Add $20”, “append a row” and “charge the card” are not.
The naive retry
The standard fix is an idempotency key, popularised by Stripe’s 2017 write-up. The client generates a unique key for each intent, one per purchase rather than one per HTTP attempt, and sends it with the request and with every retry. The server remembers what it did for each key.
An idempotency key
The details are where implementations go wrong:
- Record the key atomically with the effect. AWS’s guidance is that recording the client token and performing the mutation must be a single ACID operation. Otherwise a crash between the two leaves either a charge with no key (a retry charges again) or a key with no charge (the purchase never happens).
- Store the response, not just a flag. Stripe’s v1 API saves the status code and body of the first request with a given key once the endpoint starts executing, and replays them on retries, including 500 errors.
- Compare parameters. A retry with the same key but a different body is a bug. Stripe and EC2 both reject it rather than guess which one the client meant.
- Keys expire. Stripe may prune v1 keys once they’re 24 hours old, and EC2 keeps a
ClientTokenfor the life of the resource plus an interval. Retries must happen inside that window. - Concurrent duplicates need a lock or a unique constraint on the key, so two simultaneous retries can’t both run.
Message passing
Between the database and the direct call sits asynchronous message passing. The sender hands a message to a message broker (a queue such as RabbitMQ or SQS, or a log such as Kafka) and continues without waiting. Compared with calling the recipient directly, the broker:
- buffers messages when the recipient is down or overloaded, so it improves reliability;
- redelivers messages to a consumer that crashed;
- decouples the sender from the recipients’ addresses, so there’s no service discovery to get right;
- fans out one message to many consumers;
- makes the flow one-way: the sender doesn’t wait for, or even get, a reply, unless you build one with a reply queue.
A broker absorbs failure
Brokers typically deliver at least once: they redeliver anything not acknowledged, so consumers see duplicates after crashes, rebalances and timeouts. The fix is the same as for RPC: make consumers idempotent, either by recording processed message IDs in the same transaction as their effects, or by making the effect itself idempotent (an upsert keyed by order ID). Kafka’s idempotent producer, on by default since Kafka 3.0, removes duplicates caused by the producer’s own retries, but not by a consumer processing a message twice.
Messages are encoded bytes like any other, so the encoding lesson applies: producers and consumers evolve independently, a schema registry enforces compatibility, and a consumer that republishes a message must preserve fields it doesn’t know.Actor frameworks such as Erlang/OTP, Akka and Orleans use message passing even inside one program. Location transparency works better there, because actors already assume messages can be lost.
API versioning
An internal service can coordinate with its callers. A public API can’t: clients you have never met will keep sending last year’s requests for years. Additive changes, such as a new endpoint, an optional parameter or a new response field, are safe as long as clients ignore unknown fields. A breaking change needs a version.
- A major version in the URL (
/v1/,/v2/) is simple and visible, but each version is effectively a separate API to run. - Dated versions in a header let changes ship in small steps. GitHub’s REST API takes an
X-GitHub-Api-Versionheader; its first dated version was2022-11-28, and an older version stays supported for at least 24 months after a newer one ships. - Pinning plus transformation is Stripe’s design. An account is pinned to the API version current at its
first request, a
Stripe-Versionheader can override it per request, and the response passes through one small version change module per breaking change on its way out.
Walking a response back in time
Stripe’s 2017 post reported almost a hundred backward-incompatible changes handled this way over six years,
while keeping every version since 2011 working. Since the 2024-09-30.acacia release, Stripe ships monthly
versions with no breaking changes and a major release with breaking changes twice a year.
Durable execution
Many business processes outlive any single request: charge the card, create the shipment, wait three days, ask for a review. Classically you build that from a database table of states, a queue, a cron job and a great deal of retry code, and every crash between steps is a bug to find. Durable execution lets you write the process as ordinary code and makes the code itself crash-proof.
The best-known design, used by Temporal, works by event-history replay:
Crash, replay, resume
- Activities are the steps that touch the outside world. The engine retries them, so each runs at least once and must be idempotent.
- The workflow function orchestrates activities. Its progress is recorded in an append-only event history. After a crash, a new worker re-runs the code; each call that already has a recorded result returns it instantly, so the code rebuilds its state and carries on.
- Timers are durable. “Sleep for three days” is an event in the history, not a blocked thread.
- Workflow code must be deterministic. Replay compares the calls the code makes with the history, so reading the clock, generating random numbers or doing I/O directly in workflow code breaks it. The SDK provides deterministic versions that record their results.
- Changing code under running workflows needs versioning, through patch markers or by routing old workflows to workers running old code. This is schema evolution again, with the history as the data.
Temporal’s founders, Maxim Fateev and Samar Abbas, built Cadence at Uber after working on AWS’s Simple Workflow Service, and founded Temporal on a fork of it in 2019. Histories have hard limits: the Temporal service terminates a workflow whose history exceeds 51,200 events or 50 MB, after warning from 10,240 events or 10 MB, so long-lived workflows periodically continue-as-new with a fresh history.
The idea has spread fast: AWS Step Functions (workflows as JSON state machines), AWS Lambda durable functions (December 2025, checkpoint-and-replay, up to a year), Azure Durable Functions, Cloudflare Workflows (generally available in April 2025), and newer engines such as Restate and DBOS.
Workflow orchestration
Durable execution is one way to answer a design question: when a business process spans several services, who owns the process?
Choreography vs orchestration
- Choreography: services publish events and react to each other’s. It is loosely coupled, with no central dependency, but the process exists only implicitly, spread across services. Debugging “where is order 81?” or changing the sequence means touching several codebases.
- Orchestration: one component, a workflow, calls each step, tracks state, retries failures and runs compensations when it gives up, the core of the saga pattern. The process is explicit, observable and changed in one place. The cost is a central component every step depends on.
Orchestrators come in different flavours. Temporal and Step Functions coordinate long-running business processes. Airflow and its relatives schedule batch data pipelines as DAGs. Step Functions also shows the semantic choice involved: Standard workflows run up to a year with exactly-once workflow execution, while Express workflows run up to five minutes and, when started asynchronously, execute at least once, which suits only idempotent steps.
Trade-offs
| Database | REST | RPC (gRPC) | Message broker | Durable workflow | |
|---|---|---|---|---|---|
| Both sides up at once? | no | yes | yes | no | no |
| Caller waits? | no | yes | yes, or streams | no | the workflow waits, durably |
| Typical encoding | table schema | JSON + OpenAPI | Protobuf | Avro, Protobuf or JSON + registry | SDK-serialized payloads |
| Failure looks like | stale or missing rows | HTTP error, timeout | status code, deadline | redelivery, dead letters | retries, a stuck history |
| Duplicates come from | rare | client retries | client retries | redelivery | activity retries |
| Compatibility burden | backward, forever | requests backward, responses forward | same, via tags | per registry mode | code vs. recorded histories |
In the real world
- Netflix described in December 2025 how its Spinnaker deployment stack had grown its own orchestration and retry logic, yet about 4% of deployments still failed from transient cloud-operation errors. After moving those operations onto Temporal as workflows with idempotent activities, that share fell to 0.0001%.
- Stripe built both halves of this lesson into its public API: idempotency keys on every POST, with stored responses replayed on retry and parameter mismatches rejected, and date-based version pinning with version-change modules.
- AWS EC2’s
RunInstancestakes aClientToken, and the SDKs generate one if you don’t. The service stores the token with the request’s parameters, returns a semantically equivalent response to retries, and rejects a token reused with different parameters. - Google ran Stubby across its data centres for over a decade before rebuilding it on public standards as gRPC in 2015.
- GitHub moved its REST API to dated versions in a header in November 2022, with a two-year support window for old versions.
Pitfalls
- Treating a timeout as a failure. It means “unknown”. Either retry idempotently or check the outcome.
- One idempotency key per attempt instead of per intent. A key generated inside the retry loop protects nothing.
- Recording the key outside the transaction that performs the effect.
- No deadline on remote calls, so a slow dependency holds every caller (see the overload lesson).
- Clients that reject unknown fields or enum values, which turns every additive server change into a breaking one.
- Expecting a broker to give exactly-once processing end to end. Delivery is at least once; consumers deduplicate.
- Non-deterministic workflow code, or deploying changed workflow code without versioning.
- Choreography sprawl, where no one can say what the process is, or an orchestrator that grows into a god service owning everyone’s logic.
Recall
Go deeper
- Roy Fielding, “Architectural Styles and the Design of Network-based Software Architectures”, chapter 5 (2000): where REST’s constraints were defined.
- Waldo, Wyant, Wollrath & Kendall, “A Note on Distributed Computing” (1994): why a remote call can’t be made to look local.
- gRPC: Core concepts: call types, deadlines, cancellation and independent success.
- Brandur Leach, “Designing robust and predictable APIs with idempotency” (Stripe, 2017): the idempotency-key pattern from the source.
- Malcolm Featonby, “Making retries safe with idempotent APIs” (Amazon Builders’ Library): atomic token recording, parameter mismatches and late-arriving retries.
- Brandur Leach, “APIs as infrastructure: future-proofing Stripe with versioning” (2017): pinning and version-change modules.
- Temporal: Workflow definitions and determinism: what replay requires of your code, and how versioning works.
- Netflix, “How Temporal Powers Reliable Cloud Operations at Netflix” (2025): durable execution in a real deployment system.
Keep going
The atomic notes for this lesson live in the knowledge graph:
Next on the map: Message Brokers & Logs (queues, logs, consumer groups and delivery semantics) and Distributed Transactions (two-phase commit, sagas and the outbox).