Notesdistributed🌱 seedling
Clock Skew
Also: Clock Drift · Clock Synchronization · Time-of-Day Clock · Monotonic Clock · Leap Second · Leap Smear · NTP
Different machines' clocks disagree (skew) and wander (drift), and the time-of-day clock can jump, even backwards. A timestamp is a range, not a point, so wall-clock times are unsafe for ordering events across machines.
Also in: The Trouble with Distributed Systems › Leap seconds and monotonic clocks
Why it exists
Every machine keeps time with a quartz oscillator whose rate varies with temperature, and synchronizes it over an Unreliable Network with variable delay.
How it works
- Two clocks: the time-of-day clock (wall time; NTP may step it, backwards included) and the monotonic clock (only moves forward; only differences mean anything). Measure durations with the monotonic clock, always.
- Drift: Google's time infrastructure assumes up to 200 µs/s, about 17 s/day if never resynchronized.
- Sync accuracy: NTP over the internet usually holds tens of milliseconds, under 1 ms on a good LAN; AWS Time Sync (2023) reaches microseconds on supported EC2 instances, with error bounds via ClockBound.
- Leap seconds: Linux inserts one by stepping the wall clock back a second. A smear (24 h, noon to noon, 11.6 ppm slow, used by Google and AWS) avoids the step.
- Damage: Last Write Wins across skewed clocks drops causally later writes; leases and caches expire early or late.
Trade-offs
| Approach | Gains | Costs |
|---|---|---|
| NTP + wall-clock timestamps | cheap, human-readable | ordering wrong within the skew bound |
| explicit uncertainty (TrueTime, ClockBound) | safe ordering by waiting out the bound | special hardware or services; commit waits |
| logical clocks (Lamport Clock, Hybrid Logical Clock) | causality-safe without precise time | no real-time ordering of unrelated events |
In practice
- Cloudflare, 2017-01-01: RRDNS (Go) computed durations from the wall clock; the leap second made one negative,
rand.Int63npanicked, and about 0.2% of DNS queries were affected until 06:45 UTC. Go 1.9 then put a monotonic reading intime.Time. - The CGPM voted in October 2026 to stop inserting leap seconds from 20 May 2027. Steps from NTP corrections and VM resumes remain.
Pitfalls
end − starton the wall clock. It can be negative.- Monitoring clock offset as an afterthought; a node with a dead NTP daemon drifts silently.
Connections
Recall
These cards are also in your review deck, scheduled with spaced repetition.