Notesdistributed🌱 seedling

Clock Skew

Also: Clock Drift · Clock Synchronization · Time-of-Day Clock · Monotonic Clock · Leap Second · Leap Smear · NTP

Different machines' clocks disagree (skew) and wander (drift), and the time-of-day clock can jump, even backwards. A timestamp is a range, not a point, so wall-clock times are unsafe for ordering events across machines.

Also in: The Trouble with Distributed Systems › Leap seconds and monotonic clocks

Why it exists

Every machine keeps time with a quartz oscillator whose rate varies with temperature, and synchronizes it over an Unreliable Network with variable delay.

How it works

  • Two clocks: the time-of-day clock (wall time; NTP may step it, backwards included) and the monotonic clock (only moves forward; only differences mean anything). Measure durations with the monotonic clock, always.
  • Drift: Google's time infrastructure assumes up to 200 µs/s, about 17 s/day if never resynchronized.
  • Sync accuracy: NTP over the internet usually holds tens of milliseconds, under 1 ms on a good LAN; AWS Time Sync (2023) reaches microseconds on supported EC2 instances, with error bounds via ClockBound.
  • Leap seconds: Linux inserts one by stepping the wall clock back a second. A smear (24 h, noon to noon, 11.6 ppm slow, used by Google and AWS) avoids the step.
  • Damage: Last Write Wins across skewed clocks drops causally later writes; leases and caches expire early or late.

Trade-offs

Approach Gains Costs
NTP + wall-clock timestamps cheap, human-readable ordering wrong within the skew bound
explicit uncertainty (TrueTime, ClockBound) safe ordering by waiting out the bound special hardware or services; commit waits
logical clocks (Lamport Clock, Hybrid Logical Clock) causality-safe without precise time no real-time ordering of unrelated events

In practice

  • Cloudflare, 2017-01-01: RRDNS (Go) computed durations from the wall clock; the leap second made one negative, rand.Int63n panicked, and about 0.2% of DNS queries were affected until 06:45 UTC. Go 1.9 then put a monotonic reading in time.Time.
  • The CGPM voted in October 2026 to stop inserting leap seconds from 20 May 2027. Steps from NTP corrections and VM resumes remain.

Pitfalls

  • end − start on the wall clock. It can be negative.
  • Monitoring clock offset as an afterthought; a node with a dead NTP daemon drifts silently.

Recall

These cards are also in your review deck, scheduled with spaced repetition.