Notesreplication🌱 seedling

Failover

Also: Automatic Failover · Leader Promotion · Promotion

Promoting a follower to leader when the leader fails, and pointing clients and other followers at it. Every step (detecting, choosing, reconfiguring) has a failure mode.

Why it exists

A dead follower just restarts and catches up. A dead leader means no writes at all until someone takes over.

How it works

  1. Detect the failure, usually by a Timeout: crashed, slow and partitioned look identical from outside.
  2. Choose a new leader, ideally the most up-to-date follower, by Leader Election or a controller.
  3. Reconfigure clients to send writes to it, and make sure the old leader steps down when it returns.

Trade-offs

Timeout Effect
short fast recovery, but load spikes and network blips cause needless failovers that add load
long fewer false alarms, but longer write outages

In practice

  • Patroni (PostgreSQL) holds the leader role as a key with a TTL in etcd, Consul or ZooKeeper (30 s by default) and limits promotion to followers within maximum_lag_on_failover.
  • GitHub, September 2012: a node already known to be out of date was promoted. MySQL-generated IDs were also keys in Redis, so the stores disagreed: events showed on the wrong dashboards and 16 private repositories were briefly exposed.

Pitfalls

  • With asynchronous replication the new leader lacks the old leader's last writes; they're usually discarded, which is dangerous if caches, indexes or other databases already saw them.
  • The old leader returning and still acting as leader: Split Brain.
  • An untested failover is a failover that doesn't work.

Recall

These cards are also in your review deck, scheduled with spaced repetition.