NotesSystem

Jepsen

Also: Knossos · Elle

Kyle Kingsbury's framework and series of analyses (since 2013) that test distributed databases under injected faults and check the recorded histories against the consistency models they claim.

Also in: Serializability › In the real world · Consistency Models › In the real world

Design choices

  • Black-box histories: clients run operations against a real cluster and record every invocation and result, including timeouts as "unknown".
  • Fault injection: network partitions (including partial ones), process pauses and crashes, clock skew, membership changes. See Network Partition and Clock Skew.
  • Checkers: Knossos searches for a valid linearization; Elle (2020) infers transaction dependency graphs to find isolation anomalies efficiently.
  • The consistency map: jepsen.io/consistency relates the models and marks each as totally available, sticky available, or unavailable under partition.

Notable findings

  • etcd 3.4.3 (2020): key-value operations strict serializable; locks unsafe even in healthy clusters, with about 18% of acknowledged updates lost in one lock-protected test. Fix: fence with the lock key's revision.
  • jetcd 0.8.2 (2024): the Java client retried non-idempotent requests that may have succeeded, causing lost updates; etcd itself wasn't at fault.
  • NATS 2.12.1 (2025): flushed to disk every two minutes by default while acknowledging immediately; docs claimed linearizable and always available.

Notable trade-offs

  • Testing finds bugs, it can't prove their absence: a passing run means no violation was observed under those faults.
  • Many analyses are vendor-funded, published under a stated ethics policy.