Notesreplication🌱 seedling
Replication Lag
Also: Replica Lag · Stale Reads · Follower Lag
How far an asynchronous follower trails the leader's log. Usually milliseconds, unboundedly more under load, and the source of stale reads that look like bugs.
Why it exists
Asynchronous followers are how leader-based systems scale reads: add replicas, spread reads. The price is that each read from a follower may reflect a slightly older state.
How it works
- A follower is at some position in the leader's log; lag is the gap, in entries, bytes or seconds.
- Stop writing and every follower catches up: Eventual Consistency. "Eventually" has no upper bound.
- Lag produces three anomalies, each with its own guarantee:
- your own write seems to vanish → Read-Your-Writes Consistency;
- data goes back in time between reads → Monotonic Reads;
- effects appear before causes across partitions → Consistent Prefix Reads.
Trade-offs
| Option | Gains | Costs |
|---|---|---|
| read from followers | read scale, locality | stale reads |
| read from leader | fresh | leader load |
| synchronous replicas | bounded staleness | write latency and availability |
In practice
- Amazon Aurora readers share the writer's storage and typically lag much less than 100 ms.
- Lag spikes come from bulk imports, long transactions, a long query on a replica (PostgreSQL may pause replay or cancel the query) and cross-region links.
- GitLab, January 2017: a load spike made the secondary lag until the primary had already discarded WAL it needed, forcing a rebuild, during which the primary's data directory was wiped by mistake.
Pitfalls
- Treating lag as constant. Monitor it and decide what happens past a threshold (stop routing reads, alert).
Connections
- Learn first
- Synchronous vs Asynchronous Replication
- Builds on this
- Consistency Model
Recall
These cards are also in your review deck, scheduled with spaced repetition.