Notesdistributed🌱 seedling

Leader Election

Also: Master Election · Electing a Leader · Election Timeout

Choosing exactly one node to lead, such that every node agrees and two leaders can never act in the same term. Done properly it is consensus: a majority vote per numbered term.

Why it exists

Leader-based systems need a single writer. Promoting by timeout alone, or by script, can elect two leaders under a partition (Split Brain).

How it works

  • Terms: every election increments a term; a node seeing a higher term steps down. Old-term messages are rejected.
  • One vote per term, granted to the first eligible candidate; a majority wins. Two majorities can't coexist, so one leader per term.
  • Randomized timeouts stop candidates colliding (Raft's split votes).
  • Eligibility: in Raft, only candidates with up-to-date logs win, so no committed data is lost.
  • PreVote (ask "could I win?" first) and CheckQuorum (leader steps down without a majority) reduce disruption.
  • Outsourced: applications elect a leader via a Coordination Service by creating a key with a Lease; the next contender watches it.

Trade-offs

Choice Gains Costs
short election timeout fast failover spurious elections under load
long election timeout stable leadership longer write outages
leader leases local reads failover waits for the old lease

In practice

  • etcd: 1 s election timeout by default; guidance is at least 10× the member round trip.
  • Cloudflare (Nov 2020): a partially failed switch left etcd members with conflicting views; elections kept tying and writes were blocked.
  • Elasticsearch 7.0 (2019) replaced manually configured minimum_master_nodes with a self-managed voting configuration.

Pitfalls

  • Being elected is not proof of being leader later: a paused leader must still be fenced with a Fencing Token.
  • Partial connectivity (A sees B, B sees C, A can't see C) causes leader churn even when every node is up.

Recall

These cards are also in your review deck, scheduled with spaced repetition.