# Resilience and Reliability Patterns

**Reliability** is the probability a system does what users need, when they need it. **Resilience** is how it behaves when parts fail — and in distributed systems parts *always* fail. This topic collects the patterns into one toolbox and shows how they combine.

## 1. Design Assumptions

- Every network call can fail, be slow, or succeed without you knowing (`06-failure_handling_and_retry_strategies/`).
- Dependencies will be down at some point; your service's availability is bounded by synchronous dependencies (availability of a chain ≈ product of availabilities).
- Load will exceed capacity at some point (spikes, retries, bugs).
- Humans will deploy bad changes.

The goal: **contain** failures (blast radius), **degrade** gracefully, and **recover** automatically.

## 2. The Pattern Toolbox

| Pattern | Problem it addresses | Mechanism |
|---|---|---|
| **timeouts** | slow dependencies hold resources forever | bound every wait |
| **retries with backoff + jitter** | transient failures | bounded, idempotent, one layer |
| **circuit breaker** | repeatedly calling a dead dependency | fail fast while open; probe to recover (`distributed-systems/fundamentals/07-circuit_breaker`) |
| **bulkhead** | one dependency/tenant exhausting shared resources | separate pools/queues/concurrency limits per dependency or tenant |
| **rate limiting** | clients overloading you | token buckets per client (`backend/api-design/07-rate_limiting`) |
| **load shedding** | overload of the whole service | reject early (429/503) when queues/latency exceed thresholds; prioritize critical traffic |
| **backpressure** | fast producers, slow consumers | bounded queues, pull-based consumption (`distributed-systems/fundamentals/05-backpressure`) |
| **fallback / graceful degradation** | dependency down | cached/stale data, defaults, feature turned off |
| **redundancy & failover** | instance/AZ/region failure | N+1 instances, multi-AZ, active-active/passive (`distributed-systems/fundamentals/10-…`) |
| **idempotency** | retries and duplicates | idempotency keys, dedup, upserts |
| **queue-based load leveling** | spiky load | buffer work in a durable queue |
| **health checks & self-healing** | stuck instances | liveness/readiness, auto-replacement (`devops-cloud/observability/04-health_checks`) |
| **static stability** | control plane down | data plane keeps working on last-known config |
| **safe deployments** | bad changes | canaries, feature flags, automatic rollback (`devops-cloud/cicd/02-deployment_strategies`) |

## 3. Bulkheads

Named after ship compartments: a breach floods one compartment, not the ship.

```ts
const pools = {
  payments: new Semaphore(20),         // at most 20 concurrent calls to payments
  recommendations: new Semaphore(5),   // a slow recommender can tie up at most 5 slots
};

async function callRecommendations(userId: string) {
  if (!pools.recommendations.tryAcquire()) return FALLBACK_RECOMMENDATIONS;   // full → degrade, don't queue
  try { return await withTimeout(recoClient.get(userId), 200); }
  catch { return FALLBACK_RECOMMENDATIONS; }
  finally { pools.recommendations.release(); }
}
```

Without the bulkhead, a slow recommendation service consumes every worker/connection, and checkout (which needs payments) fails too.

## 4. Load Shedding and Prioritization

When demand exceeds capacity, **somebody** won't be served. Decide who, deliberately:

- reject early and cheaply (before parsing bodies or touching the DB),
- use **queue time** or **in-flight count** as the signal (concurrency limits like Netflix's adaptive limiters),
- prioritize: checkout > browsing > analytics beacons; authenticated > anonymous; interactive > batch,
- return `503`/`429` with `Retry-After` so well-behaved clients back off.

Serving 80% of requests well beats serving 100% of requests so slowly that they all time out (**goodput** vs throughput).

## 5. Graceful Degradation Ladder

Plan what the product does as dependencies fail:

```
all healthy          → full experience
recommendations down → show bestsellers (static/cached)
search down          → category browsing only
payments slow        → accept order, "payment processing" state, confirm asynchronously
inventory down       → allow orders for items with stale stock > threshold, reconcile later
database read replica down → serve reads from primary with stricter rate limits
```

Each step needs a feature flag / kill switch and is exercised in tests and game days.

## 6. Verifying Resilience: Chaos Engineering

Untested resilience mechanisms usually don't work. **Chaos engineering** runs controlled experiments: hypothesize steady state ("checkout success ≥ 99.5%"), inject a failure (kill pods, add 500 ms latency to payments, black-hole the cache, fail an AZ), observe, and fix what breaks. Start in staging, with small blast radius and an abort switch; graduate to production game days. Tools: Chaos Mesh, Litmus, AWS FIS, Gremlin, Toxiproxy (for local tests).

## Under the Hood: Why Resilience Mechanisms Interact

Patterns compose — and can fight. A retry policy inside a circuit breaker: should failed retries count as one failure or several? (Usually the breaker wraps the whole retried call, so persistent failure opens it quickly.) A timeout shorter than the dependency's p99 turns slow-but-successful calls into failures that trigger retries, increasing load on the slow dependency (**metastable failure**: the system stays degraded even after the original trigger disappears, because retries and timeouts sustain the overload). Load shedding and autoscaling interact: shedding hides demand from CPU-based scaling signals. The safe composition order from the caller's view is usually: **bulkhead (concurrency limit) → circuit breaker → timeout → retry (budgeted) → fallback**, with metrics on each stage so you can see which one acted.

![A call passes through a bulkhead concurrency limit, a circuit breaker, a timeout and a budgeted retry before reaching the dependency, with a fallback when any stage gives up; load shedding at the edge and a degradation ladder keep critical flows working](diagrams/01-resilience-stack.svg)

## Common Pitfalls

1. **Resilience features that are never tested** — fallbacks that throw, breakers that never open.
2. **Unbounded queues** "for resilience" — latency explodes, then memory.
3. **Shared pools** for critical and non-critical dependencies.
4. **Timeouts longer than the caller's timeout**.
5. **Retries + short timeouts on an overloaded dependency** — metastable failure.
6. **Fallbacks that hit the same failing dependency** (e.g. "fallback" to the same DB).

## Quick Reference

```
contain: bulkheads · rate limits · load shedding · backpressure
detect & stop: timeouts · circuit breakers · health checks
recover: retries (budgeted, idempotent) · failover · self-healing · safe deploys
degrade: fallbacks · degradation ladder · kill switches
verify: chaos experiments · game days
order: bulkhead → breaker → timeout → retry → fallback
```