Notesdistributed🌱 seedling

Coordination Service

Also: Lock Service · Distributed Lock Service · Configuration Service · Distributed Lock

A small, consensus-replicated store (usually 3 or 5 nodes) holding the little data a fleet must agree on: who leads, current configuration, membership, locks. Applications use it instead of implementing consensus themselves.

Why it exists

Few teams should implement Paxos or Raft. A shared service gives every application linearizable writes, leases, watches and ordering through a simple API, at the cost of a dependency everyone shares.

How it works

  • Data model: a tree of small nodes (ZooKeeper znodes) or a flat key space with revisions (etcd).
  • Leases / sessions: keys that disappear when their owner stops renewing (ephemeral znodes, etcd lease-attached keys).
  • Watches: clients are notified when a key changes, instead of polling.
  • Ordering: sequential znodes and global revisions give every change a monotonically increasing number, usable as a Fencing Token.
  • Recipes: Leader Election (create a key, watchers queue behind it), locks, group membership, Service Discovery, config distribution.

Trade-offs

Gains Costs
correct agreement without writing consensus a critical shared dependency
watches instead of polling small data and low write rates only
automatic release via leases locks still need fencing at the resource

In practice

  • Chubby (Google, 2006): five replicas per cell; one master served 90,000 clients; designed for locks, most used for naming.
  • Kubernetes stores all cluster state in etcd.
  • Kafka used ZooKeeper for metadata until KRaft; Kafka 4.0 (March 2025) removed ZooKeeper mode.

Pitfalls

  • Treating its locks as mutual exclusion. Jepsen (2020) showed two holders of an etcd lock at once; the protected resource must reject stale tokens.
  • Using it as a general database: large values and high write rates hurt the whole fleet.
  • Over-trusting its uptime: Google deliberately took the global Chubby down in quarters when real failures hadn't brought it down to its SLO, so services couldn't come to depend on it never failing.

Recall

These cards are also in your review deck, scheduled with spaced repetition.