Resilience and Reliability Patterns

Reliability is the probability a system does what users need, when they need it. Resilience is how it behaves when parts fail — and in distributed systems parts always fail. This topic collects the patterns into one toolbox and shows how they combine.

1Design Assumptions

• Every network call can fail, be slow, or succeed without you knowing (06-failure_handling_and_retry_strategies/).
• Dependencies will be down at some point; your service's availability is bounded by synchronous dependencies (availability of a chain ≈ product of availabilities).
• Load will exceed capacity at some point (spikes, retries, bugs).
• Humans will deploy bad changes.

The goal: contain failures (blast radius), degrade gracefully, and recover automatically.

2The Pattern Toolbox

3Bulkheads

Named after ship compartments: a breach floods one compartment, not the ship.

const pools = {
  payments: new Semaphore(20),         // at most 20 concurrent calls to payments
  recommendations: new Semaphore(5),   // a slow recommender can tie up at most 5 slots
};

async function callRecommendations(userId: string) {
  if (!pools.recommendations.tryAcquire()) return FALLBACK_RECOMMENDATIONS;   // full → degrade, don't queue
  try { return await withTimeout(recoClient.get(userId), 200); }
  catch { return FALLBACK_RECOMMENDATIONS; }
  finally { pools.recommendations.release(); }
}

Without the bulkhead, a slow recommendation service consumes every worker/connection, and checkout (which needs payments) fails too.

4Load Shedding and Prioritization

When demand exceeds capacity, somebody won't be served. Decide who, deliberately:

• reject early and cheaply (before parsing bodies or touching the DB),
• use queue time or in-flight count as the signal (concurrency limits like Netflix's adaptive limiters),
• prioritize: checkout > browsing > analytics beacons; authenticated > anonymous; interactive > batch,
• return 503/429 with Retry-After so well-behaved clients back off.

Serving 80% of requests well beats serving 100% of requests so slowly that they all time out (goodput vs throughput).

5Graceful Degradation Ladder

Plan what the product does as dependencies fail:

all healthy          → full experience
recommendations down → show bestsellers (static/cached)
search down          → category browsing only
payments slow        → accept order, "payment processing" state, confirm asynchronously
inventory down       → allow orders for items with stale stock > threshold, reconcile later
database read replica down → serve reads from primary with stricter rate limits

Each step needs a feature flag / kill switch and is exercised in tests and game days.

6Verifying Resilience: Chaos Engineering

Untested resilience mechanisms usually don't work. Chaos engineering runs controlled experiments: hypothesize steady state ("checkout success ≥ 99.5%"), inject a failure (kill pods, add 500 ms latency to payments, black-hole the cache, fail an AZ), observe, and fix what breaks. Start in staging, with small blast radius and an abort switch; graduate to production game days. Tools: Chaos Mesh, Litmus, AWS FIS, Gremlin, Toxiproxy (for local tests).

Under the HoodWhy Resilience Mechanisms Interact▾

Patterns compose — and can fight. A retry policy inside a circuit breaker: should failed retries count as one failure or several? (Usually the breaker wraps the whole retried call, so persistent failure opens it quickly.) A timeout shorter than the dependency's p99 turns slow-but-successful calls into failures that trigger retries, increasing load on the slow dependency (metastable failure: the system stays degraded even after the original trigger disappears, because retries and timeouts sustain the overload). Load shedding and autoscaling interact: shedding hides demand from CPU-based scaling signals. The safe composition order from the caller's view is usually: bulkhead (concurrency limit) → circuit breaker → timeout → retry (budgeted) → fallback, with metrics on each stage so you can see which one acted.

⚠️ Common Pitfalls

1. Resilience features that are never tested — fallbacks that throw, breakers that never open.
2. Unbounded queues "for resilience" — latency explodes, then memory.
3. Shared pools for critical and non-critical dependencies.
4. Timeouts longer than the caller's timeout.
5. Retries + short timeouts on an overloaded dependency — metastable failure.
6. Fallbacks that hit the same failing dependency (e.g. "fallback" to the same DB).

📋 Quick Reference

contain: bulkheads · rate limits · load shedding · backpressure
detect & stop: timeouts · circuit breakers · health checks
recover: retries (budgeted, idempotent) · failover · self-healing · safe deploys
degrade: fallbacks · degradation ladder · kill switches
verify: chaos experiments · game days
order: bulkhead → breaker → timeout → retry → fallback

🎤 Interview Q&A

🛠️ Practice exercises

Each exercise is a file in this topic folder. Code exercises are self-checking: run them and make the checks pass.

✅ Check yourself

Source: adv_resilience_and_reliability_patterns_theory.md · generated by tools/build_lessons.py