Transactions · 3 of 3

Distributed Transactions

One write, two machines, and a promise that both or neither will change. How two-phase commit keeps that promise, why it blocks, how consensus fixes the blocking, and what to do instead when you can't afford it.

≈ 30 min

The whole idea

Press ← → to step, or Space to play and pause, whenever a diagram has focus.

Atomic across machines

On one database, atomicity is the easy letter of ACID. A transaction writes to a log; the commit is one durable log record; a crash before it means the transaction never happened, and a crash after it means it did. One machine, one log, one moment of truth.

Sharding breaks that. When Alice’s account is on shard 1 and Bob’s is on shard 2, a transfer is two local transactions on two machines with two logs, and nothing ties their outcomes together. The same happens without sharding: an order service and a payment service with separate databases, or a database and a message broker.

A distributed transaction is a transaction whose writes span more than one node, and whose outcome must still be atomic: every participant commits, or every participant aborts.

Without coordination

Each shard is atomic on its own. The pair isn't.

The naive approach fails in every order you try. Commit shard 1 first and a failure on shard 2 destroys money; commit shard 2 first and you create it. Undoing the first commit isn’t an option: once committed, a value is visible, and other transactions may have read it and acted on it. And the worst case isn’t a clean failure, it’s a timeout, where you don’t know whether the other side committed at all.

The requirement is precise: no participant may commit until it knows every other participant will commit. That needs a protocol.

Two-phase commit

Two-phase commit (2PC) adds a coordinator (often a component of the database or the application’s transaction manager) and splits the commit in two.

  1. Prepare. The coordinator sends prepare to every participant. Each participant does all the work that could fail: it checks constraints, takes locks, and writes its changes plus a prepare record to its log, forced to disk. Then it votes yes. A participant that can’t prepare votes no.
  2. Commit. If every vote is yes, the coordinator writes commit to its own log, forced to disk, and sends commit to everyone. If any vote is no, or a participant doesn’t answer in time, it logs abort and sends that instead.

One no aborts everyone

A yes vote gives up the right to abort. A no vote, or silence, sinks the transaction.

Two promises make it work, and both are irreversible:

  • A participant that votes yes surrenders its right to abort. Whatever happens, including its own crash and restart, it must be able to commit when told to. That’s why the prepare record has to be durable before the vote.
  • Once the coordinator logs its decision, the decision can’t change. After a crash it re-reads the log and keeps telling participants until every one has acknowledged.

The coordinator’s log write is the transaction’s single moment of truth, the same role the commit record plays on one machine. Compared with the naive approach, 2PC is atomic.

When the coordinator fails

The promises that make 2PC atomic also make it fragile. Look at the moment right after both participants vote yes.

In doubt

Two healthy participants, one dead coordinator, and no way forward.

A participant that has voted yes and hasn’t heard the decision is in doubt. It can’t abort (it promised not to), and it can’t commit (maybe the coordinator decided abort because another participant voted no). It can’t learn the answer from the other participants either: if they’re all in doubt too, nobody knows whether the coordinator wrote commit before it died. So it waits, holding its locks, for as long as the coordinator is down. Every other transaction that touches those rows waits too.

That’s what it means to call 2PC a blocking protocol. It can’t make progress through the failure of one node, the coordinator, no matter how many other nodes are healthy.

When the coordinator’s log is truly lost, the only way out is an operator, or the database itself, making a heuristic decision: commit or abort the in-doubt transaction by hand, and hope it matches what the other participants did. If it doesn’t, atomicity is gone. Postgres lists in-doubt transactions in pg_prepared_xacts and MySQL in XA RECOVER, precisely so that someone can find them.

The price of 2PC

Blocking is the dramatic failure. The everyday cost is quieter: time spent holding locks.

Where the time goes

time →One databaselocal commitworkfsyncclient acklocksTwo-phase commitcoordinator + 2 shardsworkprepare roundlogcommit roundclient acklocksCoordinator crashesafter prepareworkprepare roundin doubt, until it recovers →locksParallel commitsCockroachDBworkwrites + STAGINGresolve intentsclient acklocks
Widths are schematic. What matters is the order of events and how long the lock bars run.

Compared with a local commit, 2PC adds a network round trip for prepare, forced log writes on every participant, the coordinator’s own forced write, and a second round trip for commit. Participants hold their locks through all of it. Put the participants in different regions and each round trip is a cross-region one. Under contention, that lock hold time is what caps throughput, so a hot row touched by distributed transactions becomes a queue.

There’s also an amplification cost. Throughput and availability are now limited by the slowest and least available participant. Any participant can veto, and any participant’s outage stalls every transaction that needs it.

2PC on top of consensus

The blocking problem comes from one specific weakness: the coordinator’s decision lives on one machine. Replace that machine with a replicated group that agrees through consensus, and the decision survives the loss of any minority.

Replicate every party

Coordinator group (consensus)
Same two phases. Now no single crash can leave anyone in doubt for long.

Gray and Lamport made this precise in Consensus on Transaction Commit (2006): their Paxos Commit uses 2F+1 coordinators and keeps going as long as F+1 of them work. Classic 2PC turns out to be the special case F = 0: a single coordinator, tolerating zero faults.

This is the design of the big distributed SQL databases:

  • Spanner makes every split (shard) a Paxos group. A cross-split transaction picks one participant as coordinator. Participants record their locks and prepare results in their Paxos logs; the coordinator assigns a TrueTime timestamp and replicates the commit decision to a majority of its own group before telling anyone.
  • CockroachDB writes intents (a replicated provisional value plus a lock) through Raft, and keeps the outcome in a transaction record: PENDING, STAGING, COMMITTED or ABORTED. The transaction’s coordinator, on the gateway node, heartbeats the record; if the heartbeats stop, other transactions that run into its intents can abort it. Parallel commits write the record as STAGING alongside the final writes, so the transaction counts as committed once all of them succeed. That cuts commit latency from two rounds of consensus to one.
  • Percolator, Google’s 2010 system for incrementally updating the web search index, built cross-row transactions on Bigtable with no coordinator server at all. The client designates one written cell as the primary lock; committing means replacing that one lock with a commit record. Anyone who finds a stale lock checks the primary to roll the transaction forward or back. TiDB’s storage layer, TiKV, uses the same scheme. Percolator cut the average age of documents in Google’s search results by 50%.
Calvin (2012) takes a different route: agree on the order of transactions first, then have every node execute them deterministically in that order. Nodes never need a commit vote, because a deterministic execution can’t disagree. The catch is that a transaction must declare what it will read and write up front.

Consensus fixes blocking. It doesn’t fix the cost: each phase is now a round of consensus, locks are still held across both, and the slowest participant still sets the pace.

XA transactions

Everything above is 2PC inside one database system, where the vendor controls both ends. XA is the other kind: a 1991 X/Open standard for running 2PC across heterogeneous systems, such as two different databases, or a database and a message broker, coordinated by an external transaction manager. Java’s JTA exposes it; Postgres implements the participant side as PREPARE TRANSACTION / COMMIT PREPARED, and MySQL as XA START … XA PREPARE … XA COMMIT.

XA has every 2PC problem plus a few of its own:

  • The coordinator is often a library inside the application server. Its log is on that server’s disk. Lose the server, and the in-doubt transactions wait for it, holding locks in databases other teams depend on.
  • It’s a lowest common denominator. Participants can’t detect deadlocks across systems, and isolation across them is whatever each one provides.
  • The participant side is often a second-class feature. Postgres ships with max_prepared_transactions set to zero, which disables prepared transactions, and its docs warn that leaving one prepared holds its locks and can block VACUUM badly enough to threaten transaction-ID wraparound. MySQL supports XA only in InnoDB and warns against combining XA with replication filters or statement-based replication.

Sagas

Sometimes the participants are services you can’t, or shouldn’t, lock together: a payment provider, a warehouse system, an airline. Holding locks across them for the length of a business process isn’t possible. Sagas (Garcia-Molina and Salem, 1987) give up isolation to get rid of the locks.

A saga is a sequence of local transactions T1, T2, … Tn. Each one commits immediately in its own service. For every step that can be undone there’s a compensating transaction Ci that semantically reverses it. If step Tk fails, the saga runs Ck−1 … C1 in reverse. The outcome is either all of T1…Tn, or a prefix followed by its compensations.

Forward, then back

Each step commits immediately. Failure means running the undo steps, in reverse.

The details are where sagas succeed or fail:

  • Compensation is semantic, not physical. You don’t restore old bytes; you issue a refund, release a reservation, send an apology email. Some actions can’t be compensated at all (an email was sent, a package shipped), so order steps so the irreversible ones come last.
  • Steps and compensations must be idempotent and retryable. A saga that crashes mid-compensation resumes and retries, so “refund” will sometimes run twice. A compensation must also be able to succeed eventually: a saga stuck between “charged” and “refunded” is the worst outcome.
  • No isolation. Other transactions can see the intermediate states. A customer sees a charge and then a refund; a stock check sees inventory that is “reserved” by an order that’s about to be cancelled. The standard countermeasures are design techniques: semantic locks (an order in a PENDING state that others treat as tentative), commutative updates, and re-reading values before acting.
  • Orchestration vs choreography. An orchestrator tells each service what to do next and tracks where the saga is, which makes the flow easy to see and test. In choreography, each service reacts to the previous service’s events, with no central controller; it’s looser, but the flow lives in no single place. Durable workflow engines such as Temporal are a common way to run orchestrated sagas so they survive crashes.

The transactional outbox

Every saga step, and nearly every event-driven service, has the same small distributed transaction hidden inside it: update my database and tell the world. That’s two systems and no shared transaction, the dual-write problem.

The dual-write problem

Two writes to two systems. A crash between them breaks one or the other.

The transactional outbox removes the second write from the transaction. The service inserts the event into an outbox table in the same database, in the same local transaction as the business change. The event is committed if and only if the order is. A separate relay then publishes outbox rows to the broker, either by polling the table or, better, by tailing the database’s change log (change data capture, as Debezium does).

Outbox, relay, idempotent consumer

Atomic on the way in, at-least-once on the way out, deduplicated at the end.

The relay can always crash between publishing and recording its progress, so it will sometimes publish an event twice. Delivery is at least once, and every consumer must be idempotent: record the IDs of processed events in its own database, in the same transaction as the effects, and skip repeats. That table is sometimes called an inbox. Debezium’s outbox router puts each event’s ID in a message header for exactly this, and uses the aggregate ID (the order ID) as the Kafka message key, so all events for one order land in one partition in commit order.

The result is end-to-end atomicity built from pieces that are each simple: one local transaction, one relay, one dedup table. No coordinator, no in-doubt state, no cross-system locks.

Avoiding the problem

The cheapest distributed transaction is the one you don’t need. Pat Helland’s Life beyond Distributed Transactions (2007) argues that systems that must scale indefinitely should treat it as a design constraint:

  • Make the unit of atomicity an entity. A transaction touches one entity with one key: one customer, one order, one account. Entities can live anywhere, so they shard freely.
  • Co-locate what changes together. Choose shard keys so the rows a transaction touches share a shard. Figma groups tables with the same shard key into a colo that supports transactions on that key; DynamoDB item collections share a partition key; Vitess keyspaces put related tables on the same keyspace ID.
  • Coordinate across entities with messages, made reliable with outboxes, idempotent handlers and workflows that remember what they’ve asked of whom.

Most real systems end up with a mix: single-shard transactions for the common path, a database that does 2PC over consensus for the few cross-shard writes that need it, and sagas plus outboxes between services.

Trade-offs

Approach Atomic? Isolated? Blocks on a crash? Typical cost
Naive per-node commits no no no lost or duplicated updates
Classic 2PC yes yes (with locking) yes, on coordinator loss 2 round trips, forced writes, long lock holds
2PC over consensus yes yes no, while a majority lives every phase is a consensus round
XA across systems yes per participant yes, and the coordinator is in the app operational fragility across teams
Saga eventually no no compensations, intermediate states, idempotency
Outbox + idempotent consumers yes, for “write + publish” n/a no relay to run; duplicates to dedupe

In the real world

100actions in one DynamoDB TransactWriteItems call, within one Region
2 → 1consensus rounds per commit with CockroachDB parallel commits
0max_prepared_transactions in stock Postgres: prepared transactions off
  • DynamoDB transactions group up to 100 actions (4 MB total) across tables in one account and Region. Each item costs two underlying reads or writes, one to prepare and one to commit, and conflicting transactions are cancelled, not queued. A client token makes retries idempotent for ten minutes. Across global-table Regions there’s no atomicity: other Regions can observe partially applied transactions.
  • Vitess defaults to a best-effort MULTI mode where a cross-shard commit can succeed on some shards and fail on others. Atomic TWOPC mode is opt-in, needs semi-sync replication, adds commit latency, and still allows fractured reads: a cross-shard query can see a transaction half-committed.
  • Spanner and CockroachDB run 2PC over Paxos and Raft respectively, so cross-shard transactions are serializable by default and survive coordinator loss.
  • Kafka transactions are a 2PC-style protocol too: a transaction coordinator, a transaction log, and commit or abort markers written to every partition involved. In November 2024, Jepsen’s analysis of Bufstream found aborted reads, lost writes and torn transactions in Kafka itself when processes paused at the wrong moment (KAFKA-17754), and concluded that the transaction protocol needed revision. Kafka’s engineers expected KIP-890 to fix it; its strengthened protocol, which bumps the producer epoch on every transaction, is on by default since Kafka 4.0 (2025). Commit protocols are easy to get subtly wrong, even for mature systems.

Pitfalls

  • Treating a timeout as a failure. After a timeout, the remote side may have committed. Retry with an idempotency key, or ask; never assume.
  • Forgetting the in-doubt case. Monitor prepared transactions (pg_prepared_xacts, XA RECOVER) and alert on old ones; they hold locks.
  • “Our saga is a transaction.” It isn’t isolated. Design for readers that see intermediate states.
  • Non-idempotent compensations. They will run twice eventually.
  • Publishing from inside the transaction. Calling Kafka before COMMIT publishes events for transactions that may roll back. Use an outbox.
  • Assuming an outbox gives exactly-once. It gives at-least-once. Exactly-once effects come from idempotent consumers.
  • Cross-shard transactions on the hot path by accident. A shard key that splits rows usually written together turns every write into 2PC. Fix the key.

Recall

Go deeper

Keep going

The atomic notes for this lesson live in the knowledge graph:

Next on the map: The Trouble with Distributed Systems (why timeouts and partial failure make all of this hard) and Consensus (the machinery that makes 2PC non-blocking).

Done with this lesson?

Mark it learned and the concepts it teaches will light up in your galaxy.

Transactions: roadmapBack to the roadmap