Notesfoundations🌱 seedling
Health Check
Also: Health Checking · Outlier Detection · Passive Health Check · Fail Open · Panic Threshold
A health check decides whether a backend should get traffic right now. Active checks probe an endpoint on a timer. Passive checks (outlier detection) watch real responses. Both need guardrails so a correlated failure doesn't empty the pool.
Why it exists
Backends die, hang or break in ways the process itself can't see. Without a health signal the balancer keeps sending traffic into the hole.
How it works
- Active: probe
/healthevery interval. Remove after N consecutive failures; re-admit after M passes (hysteresis). Detection lag is about interval × N. - Shallow vs deep: a shallow check asks "can this server serve?" (process up, disk writable). A deep check also tests dependencies, so one dependency blip fails every server at once.
- Passive / outlier detection: count real errors and eject a backend for a growing period after, say, 5 consecutive 5xx. Cap ejections at a small share of the pool.
- Fail open: if too much of the pool looks unhealthy, assume the signal is wrong and send traffic to everyone. A total outage is worse than a partial one.
Trade-offs
| Choice | Gains | Costs |
|---|---|---|
| faster probes, lower thresholds | quicker removal | probe load, flapping |
| deep checks | catch broken credentials, config | correlated mass ejection |
| passive checks | test the real work | need traffic; can eject for a bad client |
In practice
- AWS ALB: defaults to a probe every 30 s, unhealthy after 2 failures, healthy after 5 passes. It fails open, routing to all targets if all are unhealthy.
- Envoy: outlier detection ejects after 5 consecutive 5xx for 30 s (longer on repeats), at most 10% of hosts. Below 50% healthy hosts (the panic threshold) it ignores health entirely.
Pitfalls
- The black hole: a server that fails fast still passes
/health, and a load-aware algorithm sends it the most traffic. Passive checks catch it. - Deep checks on the balancer turn a dependency hiccup into a full outage. Send dependency checks to monitoring instead.
Connections
- Part of
- Load Balancer
- In real systems
- Envoy
Recall
These cards are also in your review deck, scheduled with spaced repetition.