Distributed Transactions
One write, two machines, and a promise that both or neither will change. How two-phase commit keeps that promise, why it blocks, how consensus fixes the blocking, and what to do instead when you can't afford it.
The whole idea
Atomic across machines
On one database, atomicity is the easy letter of ACID. A transaction writes to a log; the commit is one durable log record; a crash before it means the transaction never happened, and a crash after it means it did. One machine, one log, one moment of truth.
Sharding breaks that. When Alice’s account is on shard 1 and Bob’s is on shard 2, a transfer is two local transactions on two machines with two logs, and nothing ties their outcomes together. The same happens without sharding: an order service and a payment service with separate databases, or a database and a message broker.
A distributed transaction is a transaction whose writes span more than one node, and whose outcome must still be atomic: every participant commits, or every participant aborts.
Without coordination
The naive approach fails in every order you try. Commit shard 1 first and a failure on shard 2 destroys money; commit shard 2 first and you create it. Undoing the first commit isn’t an option: once committed, a value is visible, and other transactions may have read it and acted on it. And the worst case isn’t a clean failure, it’s a timeout, where you don’t know whether the other side committed at all.
The requirement is precise: no participant may commit until it knows every other participant will commit. That needs a protocol.
Two-phase commit
Two-phase commit (2PC) adds a coordinator (often a component of the database or the application’s transaction manager) and splits the commit in two.
- Prepare. The coordinator sends prepare to every participant. Each participant does all the work that could fail: it checks constraints, takes locks, and writes its changes plus a prepare record to its log, forced to disk. Then it votes yes. A participant that can’t prepare votes no.
- Commit. If every vote is yes, the coordinator writes commit to its own log, forced to disk, and sends commit to everyone. If any vote is no, or a participant doesn’t answer in time, it logs abort and sends that instead.
One no aborts everyone
Two promises make it work, and both are irreversible:
- A participant that votes yes surrenders its right to abort. Whatever happens, including its own crash and restart, it must be able to commit when told to. That’s why the prepare record has to be durable before the vote.
- Once the coordinator logs its decision, the decision can’t change. After a crash it re-reads the log and keeps telling participants until every one has acknowledged.
The coordinator’s log write is the transaction’s single moment of truth, the same role the commit record plays on one machine. Compared with the naive approach, 2PC is atomic.
When the coordinator fails
The promises that make 2PC atomic also make it fragile. Look at the moment right after both participants vote yes.
In doubt
A participant that has voted yes and hasn’t heard the decision is in doubt. It can’t abort (it promised not to), and it can’t commit (maybe the coordinator decided abort because another participant voted no). It can’t learn the answer from the other participants either: if they’re all in doubt too, nobody knows whether the coordinator wrote commit before it died. So it waits, holding its locks, for as long as the coordinator is down. Every other transaction that touches those rows waits too.
That’s what it means to call 2PC a blocking protocol. It can’t make progress through the failure of one node, the coordinator, no matter how many other nodes are healthy.
When the coordinator’s log is truly lost, the only way out is an operator, or the database itself, making a
heuristic decision: commit or abort the in-doubt transaction by hand, and hope it matches what the other
participants did. If it doesn’t, atomicity is gone. Postgres lists in-doubt transactions in
pg_prepared_xacts and MySQL in XA RECOVER, precisely so that someone can find them.
The price of 2PC
Blocking is the dramatic failure. The everyday cost is quieter: time spent holding locks.
Where the time goes
Compared with a local commit, 2PC adds a network round trip for prepare, forced log writes on every participant, the coordinator’s own forced write, and a second round trip for commit. Participants hold their locks through all of it. Put the participants in different regions and each round trip is a cross-region one. Under contention, that lock hold time is what caps throughput, so a hot row touched by distributed transactions becomes a queue.
There’s also an amplification cost. Throughput and availability are now limited by the slowest and least available participant. Any participant can veto, and any participant’s outage stalls every transaction that needs it.
2PC on top of consensus
The blocking problem comes from one specific weakness: the coordinator’s decision lives on one machine. Replace that machine with a replicated group that agrees through consensus, and the decision survives the loss of any minority.
Replicate every party
Gray and Lamport made this precise in Consensus on Transaction Commit (2006): their Paxos Commit uses 2F+1 coordinators and keeps going as long as F+1 of them work. Classic 2PC turns out to be the special case F = 0: a single coordinator, tolerating zero faults.
This is the design of the big distributed SQL databases:
- Spanner makes every split (shard) a Paxos group. A cross-split transaction picks one participant as coordinator. Participants record their locks and prepare results in their Paxos logs; the coordinator assigns a TrueTime timestamp and replicates the commit decision to a majority of its own group before telling anyone.
- CockroachDB writes intents (a replicated provisional value plus a lock) through Raft, and keeps the
outcome in a transaction record:
PENDING,STAGING,COMMITTEDorABORTED. The transaction’s coordinator, on the gateway node, heartbeats the record; if the heartbeats stop, other transactions that run into its intents can abort it. Parallel commits write the record asSTAGINGalongside the final writes, so the transaction counts as committed once all of them succeed. That cuts commit latency from two rounds of consensus to one. - Percolator, Google’s 2010 system for incrementally updating the web search index, built cross-row transactions on Bigtable with no coordinator server at all. The client designates one written cell as the primary lock; committing means replacing that one lock with a commit record. Anyone who finds a stale lock checks the primary to roll the transaction forward or back. TiDB’s storage layer, TiKV, uses the same scheme. Percolator cut the average age of documents in Google’s search results by 50%.
Consensus fixes blocking. It doesn’t fix the cost: each phase is now a round of consensus, locks are still held across both, and the slowest participant still sets the pace.
XA transactions
Everything above is 2PC inside one database system, where the vendor controls both ends. XA is the
other kind: a 1991 X/Open standard for running 2PC across heterogeneous systems, such as two different
databases, or a database and a message broker, coordinated by an external transaction manager. Java’s JTA
exposes it; Postgres implements the participant side as PREPARE TRANSACTION / COMMIT PREPARED, and MySQL
as XA START … XA PREPARE … XA COMMIT.
XA has every 2PC problem plus a few of its own:
- The coordinator is often a library inside the application server. Its log is on that server’s disk. Lose the server, and the in-doubt transactions wait for it, holding locks in databases other teams depend on.
- It’s a lowest common denominator. Participants can’t detect deadlocks across systems, and isolation across them is whatever each one provides.
- The participant side is often a second-class feature. Postgres ships with
max_prepared_transactionsset to zero, which disables prepared transactions, and its docs warn that leaving one prepared holds its locks and can blockVACUUMbadly enough to threaten transaction-ID wraparound. MySQL supports XA only in InnoDB and warns against combining XA with replication filters or statement-based replication.
Sagas
Sometimes the participants are services you can’t, or shouldn’t, lock together: a payment provider, a warehouse system, an airline. Holding locks across them for the length of a business process isn’t possible. Sagas (Garcia-Molina and Salem, 1987) give up isolation to get rid of the locks.
A saga is a sequence of local transactions T1, T2, … Tn. Each one commits immediately in its own service. For every step that can be undone there’s a compensating transaction Ci that semantically reverses it. If step Tk fails, the saga runs Ck−1 … C1 in reverse. The outcome is either all of T1…Tn, or a prefix followed by its compensations.
Forward, then back
The details are where sagas succeed or fail:
- Compensation is semantic, not physical. You don’t restore old bytes; you issue a refund, release a reservation, send an apology email. Some actions can’t be compensated at all (an email was sent, a package shipped), so order steps so the irreversible ones come last.
- Steps and compensations must be idempotent and retryable. A saga that crashes mid-compensation resumes and retries, so “refund” will sometimes run twice. A compensation must also be able to succeed eventually: a saga stuck between “charged” and “refunded” is the worst outcome.
- No isolation. Other transactions can see the intermediate states. A customer sees a charge and then a
refund; a stock check sees inventory that is “reserved” by an order that’s about to be cancelled. The
standard countermeasures are design techniques: semantic locks (an order in a
PENDINGstate that others treat as tentative), commutative updates, and re-reading values before acting. - Orchestration vs choreography. An orchestrator tells each service what to do next and tracks where the saga is, which makes the flow easy to see and test. In choreography, each service reacts to the previous service’s events, with no central controller; it’s looser, but the flow lives in no single place. Durable workflow engines such as Temporal are a common way to run orchestrated sagas so they survive crashes.
The transactional outbox
Every saga step, and nearly every event-driven service, has the same small distributed transaction hidden inside it: update my database and tell the world. That’s two systems and no shared transaction, the dual-write problem.
The dual-write problem
The transactional outbox removes the second write from the transaction. The service inserts the event into an outbox table in the same database, in the same local transaction as the business change. The event is committed if and only if the order is. A separate relay then publishes outbox rows to the broker, either by polling the table or, better, by tailing the database’s change log (change data capture, as Debezium does).
Outbox, relay, idempotent consumer
The relay can always crash between publishing and recording its progress, so it will sometimes publish an event twice. Delivery is at least once, and every consumer must be idempotent: record the IDs of processed events in its own database, in the same transaction as the effects, and skip repeats. That table is sometimes called an inbox. Debezium’s outbox router puts each event’s ID in a message header for exactly this, and uses the aggregate ID (the order ID) as the Kafka message key, so all events for one order land in one partition in commit order.
The result is end-to-end atomicity built from pieces that are each simple: one local transaction, one relay, one dedup table. No coordinator, no in-doubt state, no cross-system locks.
Avoiding the problem
The cheapest distributed transaction is the one you don’t need. Pat Helland’s Life beyond Distributed Transactions (2007) argues that systems that must scale indefinitely should treat it as a design constraint:
- Make the unit of atomicity an entity. A transaction touches one entity with one key: one customer, one order, one account. Entities can live anywhere, so they shard freely.
- Co-locate what changes together. Choose shard keys so the rows a transaction touches share a shard. Figma groups tables with the same shard key into a colo that supports transactions on that key; DynamoDB item collections share a partition key; Vitess keyspaces put related tables on the same keyspace ID.
- Coordinate across entities with messages, made reliable with outboxes, idempotent handlers and workflows that remember what they’ve asked of whom.
Most real systems end up with a mix: single-shard transactions for the common path, a database that does 2PC over consensus for the few cross-shard writes that need it, and sagas plus outboxes between services.
Trade-offs
| Approach | Atomic? | Isolated? | Blocks on a crash? | Typical cost |
|---|---|---|---|---|
| Naive per-node commits | no | no | no | lost or duplicated updates |
| Classic 2PC | yes | yes (with locking) | yes, on coordinator loss | 2 round trips, forced writes, long lock holds |
| 2PC over consensus | yes | yes | no, while a majority lives | every phase is a consensus round |
| XA across systems | yes | per participant | yes, and the coordinator is in the app | operational fragility across teams |
| Saga | eventually | no | no | compensations, intermediate states, idempotency |
| Outbox + idempotent consumers | yes, for “write + publish” | n/a | no | relay to run; duplicates to dedupe |
In the real world
- DynamoDB transactions group up to 100 actions (4 MB total) across tables in one account and Region. Each item costs two underlying reads or writes, one to prepare and one to commit, and conflicting transactions are cancelled, not queued. A client token makes retries idempotent for ten minutes. Across global-table Regions there’s no atomicity: other Regions can observe partially applied transactions.
- Vitess defaults to a best-effort
MULTImode where a cross-shard commit can succeed on some shards and fail on others. AtomicTWOPCmode is opt-in, needs semi-sync replication, adds commit latency, and still allows fractured reads: a cross-shard query can see a transaction half-committed. - Spanner and CockroachDB run 2PC over Paxos and Raft respectively, so cross-shard transactions are serializable by default and survive coordinator loss.
- Kafka transactions are a 2PC-style protocol too: a transaction coordinator, a transaction log, and commit or abort markers written to every partition involved. In November 2024, Jepsen’s analysis of Bufstream found aborted reads, lost writes and torn transactions in Kafka itself when processes paused at the wrong moment (KAFKA-17754), and concluded that the transaction protocol needed revision. Kafka’s engineers expected KIP-890 to fix it; its strengthened protocol, which bumps the producer epoch on every transaction, is on by default since Kafka 4.0 (2025). Commit protocols are easy to get subtly wrong, even for mature systems.
Pitfalls
- Treating a timeout as a failure. After a timeout, the remote side may have committed. Retry with an idempotency key, or ask; never assume.
- Forgetting the in-doubt case. Monitor prepared transactions (
pg_prepared_xacts,XA RECOVER) and alert on old ones; they hold locks. - “Our saga is a transaction.” It isn’t isolated. Design for readers that see intermediate states.
- Non-idempotent compensations. They will run twice eventually.
- Publishing from inside the transaction. Calling Kafka before
COMMITpublishes events for transactions that may roll back. Use an outbox. - Assuming an outbox gives exactly-once. It gives at-least-once. Exactly-once effects come from idempotent consumers.
- Cross-shard transactions on the hot path by accident. A shard key that splits rows usually written together turns every write into 2PC. Fix the key.
Recall
Go deeper
- Gray & Lamport, “Consensus on Transaction Commit” (ACM TODS 2006): why 2PC blocks, and Paxos Commit, of which 2PC is the zero-fault special case.
- Google Cloud, “Life of Spanner Reads & Writes”: participants, coordinator splits, Paxos-logged prepares and commit wait, step by step.
- CockroachDB docs: Transaction Layer: write intents, transaction records and parallel commits.
- Peng & Dabek, “Large-scale Incremental Processing Using Distributed Transactions and Notifications” (OSDI 2010): Percolator’s client-driven 2PC with a primary lock.
- Garcia-Molina & Salem, “Sagas” (SIGMOD 1987): the original: long-lived transactions, compensations, and why they trade isolation away.
- Pat Helland, “Life beyond Distributed Transactions: an Apostate’s Opinion” (CIDR 2007): entities, activities and messaging as the alternative at scale.
- microservices.io: Saga and Transactional Outbox: the patterns with orchestration, choreography and relay variants.
- Jepsen, “Bufstream 0.1.0” (2024): a modern reminder that commit protocols fail in subtle ways, Kafka’s included.
Keep going
The atomic notes for this lesson live in the knowledge graph:
Next on the map: The Trouble with Distributed Systems (why timeouts and partial failure make all of this hard) and Consensus (the machinery that makes 2PC non-blocking).