Health Checks

1Three Different Questions

"Is the service healthy?" is actually three questions with different consumers and consequences:

Load balancers' target health checks (ALB, NLB) are readiness checks by another name.

2What Each Should Check

Liveness — keep it dumb. Return 200 if the process can serve a trivial request (event loop not blocked, no deadlock). Never check dependencies in liveness: if the database is down, restarting every app instance doesn't fix the database — it turns a partial outage into a total one (crash loops, cold caches, thundering herds on recovery).

Readiness — "can I do useful work?":

• startup work finished (config loaded, caches warmed, migrations checked)
• not draining for shutdown
• critical local resources OK (connection pool can hand out a connection)
• dependencies: only hard dependencies without which every request would fail, and with care — if all instances fail readiness because a shared dependency is down, the LB has zero targets (some LBs then "fail open" and send traffic anyway). Often it's better to stay ready and return fast, meaningful errors (circuit breaker) than to disappear.

Deep health / diagnostics (/health/details, auth-protected): every dependency with latency and status — for humans and dashboards, not for automated restarts.

3Kubernetes Configuration

startupProbe:
  httpGet: { path: /livez, port: 8080 }
  periodSeconds: 5
  failureThreshold: 30          # up to 150 s to start before liveness takes over
livenessProbe:
  httpGet: { path: /livez, port: 8080 }
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3           # restarted after ~30 s of failures
readinessProbe:
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5
  failureThreshold: 2
lifecycle:
  preStop: { exec: { command: ["sleep", "10"] } }   # let endpoints update before shutdown starts
terminationGracePeriodSeconds: 40

4Graceful Shutdown Is Part of Health

SIGTERM received
  → readiness starts returning 503 (stop getting NEW traffic)
  → wait until the LB/endpoints have removed the instance (preStop sleep / a few seconds)
  → stop accepting new connections, finish in-flight requests (bounded time)
  → close DB pools, flush logs/telemetry
  → exit 0  (before terminationGracePeriodSeconds, or SIGKILL arrives)

Without this, every deploy produces a burst of 502s (system-design/system-design/08-service_discovery exercise 1).

5Implementation Tips

• Health endpoints must be cheap and fast (no heavy queries), have timeouts for anything they check, and ideally serve cached results computed in the background, so a burst of probes can't overload a dependency.
• Exclude them from auth, rate limiting, request logs (noise) and tracing sampling.
• Return machine-readable JSON ({"status":"fail","checks":{...}}) with correct HTTP codes (200 / 503).
• Expose them on the internal port only; don't leak dependency details publicly.

Under the HoodHow the Kubelet and an ALB Act on Probes▾

The kubelet on each node runs probes for its pods on the configured period. Consecutive failures reaching failureThreshold change the container's status: a liveness failure makes the kubelet kill and restart the container (with exponential back-off between restarts — the CrashLoopBackOff you see); a readiness failure flips the pod's Ready condition, the EndpointSlice controller removes the pod IP from the Service's endpoints, and kube-proxy/ingress controllers stop routing new connections to it (existing connections aren't cut). An ALB runs its own health checks from every load balancer node; a target becomes unhealthy after UnhealthyThresholdCount consecutive failures, and if all targets in a group are unhealthy, the ALB fails open and routes to all of them — a deliberate choice to avoid a total outage caused by an over-eager health check.

⚠️ Common Pitfalls

1. Dependency checks in liveness — a DB blip restarts the whole fleet.
2. Readiness that says "ready" before warm-up — the first requests time out.
3. No startup probe for slow starters — liveness kills them before they ever start.
4. Exiting immediately on SIGTERM — dropped requests every deploy.
5. Expensive health endpoints — probes become a load source.
6. Same endpoint for all three probes — different questions need different answers.

📋 Quick Reference

🎤 Interview Q&A

🛠️ Practice exercises

Each exercise is a file in this topic folder. Code exercises are self-checking: run them and make the checks pass.

✅ Check yourself

Source: obs_health_checks_theory.md · generated by tools/build_lessons.py