# Health Checks

## 1. Three Different Questions

"Is the service healthy?" is actually three questions with different consumers and consequences:

| Probe | Question | If it fails… | Typical endpoint |
|---|---|---|---|
| **liveness** | is the process stuck beyond recovery? | the orchestrator **restarts** it | `GET /livez` |
| **readiness** | should this instance receive traffic **right now**? | removed from the load balancer / Service endpoints (not restarted) | `GET /readyz` |
| **startup** | has slow initialization finished? | liveness/readiness checks don't start until it passes | `GET /startupz` (or reuse livez) |

Load balancers' **target health checks** (ALB, NLB) are readiness checks by another name.

## 2. What Each Should Check

**Liveness — keep it dumb.** Return 200 if the process can serve a trivial request (event loop not blocked, no deadlock). **Never check dependencies** in liveness: if the database is down, restarting every app instance doesn't fix the database — it turns a partial outage into a total one (crash loops, cold caches, thundering herds on recovery).

**Readiness — "can I do useful work?"**:
- startup work finished (config loaded, caches warmed, migrations checked)
- not draining for shutdown
- critical local resources OK (connection pool can hand out a connection)
- dependencies: only **hard** dependencies without which every request would fail, and with care — if *all* instances fail readiness because a shared dependency is down, the LB has zero targets (some LBs then "fail open" and send traffic anyway). Often it's better to stay ready and return fast, meaningful errors (circuit breaker) than to disappear.

**Deep health / diagnostics** (`/health/details`, auth-protected): every dependency with latency and status — for humans and dashboards, **not** for automated restarts.

## 3. Kubernetes Configuration

```yaml
startupProbe:
  httpGet: { path: /livez, port: 8080 }
  periodSeconds: 5
  failureThreshold: 30          # up to 150 s to start before liveness takes over
livenessProbe:
  httpGet: { path: /livez, port: 8080 }
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3           # restarted after ~30 s of failures
readinessProbe:
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5
  failureThreshold: 2
lifecycle:
  preStop: { exec: { command: ["sleep", "10"] } }   # let endpoints update before shutdown starts
terminationGracePeriodSeconds: 40
```

## 4. Graceful Shutdown Is Part of Health

```
SIGTERM received
  → readiness starts returning 503 (stop getting NEW traffic)
  → wait until the LB/endpoints have removed the instance (preStop sleep / a few seconds)
  → stop accepting new connections, finish in-flight requests (bounded time)
  → close DB pools, flush logs/telemetry
  → exit 0  (before terminationGracePeriodSeconds, or SIGKILL arrives)
```

Without this, every deploy produces a burst of 502s (`system-design/system-design/08-service_discovery` exercise 1).

## 5. Implementation Tips

- Health endpoints must be **cheap and fast** (no heavy queries), have **timeouts** for anything they check, and ideally serve **cached** results computed in the background, so a burst of probes can't overload a dependency.
- Exclude them from auth, rate limiting, request logs (noise) and tracing sampling.
- Return machine-readable JSON (`{"status":"fail","checks":{...}}`) with correct HTTP codes (200 / 503).
- Expose them on the internal port only; don't leak dependency details publicly.

## Under the Hood: How the Kubelet and an ALB Act on Probes

The **kubelet** on each node runs probes for its pods on the configured period. Consecutive failures reaching `failureThreshold` change the container's status: a liveness failure makes the kubelet **kill and restart** the container (with exponential back-off between restarts — the `CrashLoopBackOff` you see); a readiness failure flips the pod's `Ready` condition, the **EndpointSlice controller** removes the pod IP from the Service's endpoints, and kube-proxy/ingress controllers stop routing new connections to it (existing connections aren't cut). An **ALB** runs its own health checks from every load balancer node; a target becomes unhealthy after `UnhealthyThresholdCount` consecutive failures, and if *all* targets in a group are unhealthy, the ALB **fails open** and routes to all of them — a deliberate choice to avoid a total outage caused by an over-eager health check.

![A request handler exposes livez that only checks the process, readyz that reflects warm-up, draining and critical resources, and a protected details endpoint; the kubelet restarts on liveness failure and removes the pod from endpoints on readiness failure, and SIGTERM flips readiness before draining](diagrams/01-probe-semantics.svg)

## Common Pitfalls

1. **Dependency checks in liveness** — a DB blip restarts the whole fleet.
2. **Readiness that says "ready" before warm-up** — the first requests time out.
3. **No startup probe for slow starters** — liveness kills them before they ever start.
4. **Exiting immediately on SIGTERM** — dropped requests every deploy.
5. **Expensive health endpoints** — probes become a load source.
6. **Same endpoint for all three probes** — different questions need different answers.

## Quick Reference

| | checks | failure action |
|---|---|---|
| liveness | process alive, not deadlocked | restart |
| readiness | warmed up, not draining, can serve | remove from LB |
| startup | init finished | delays the other probes |
| details | every dependency (protected) | humans/dashboards only |