Failure Handling and Retry Strategies

distributed-systems/fundamentals/06-retry_with_exponential_backoff and 07-circuit_breaker introduced the mechanisms. This topic is the engineering discipline around them: classifying failures, choosing what to retry, budgeting retries across a system, and failing in ways that are safe for users and data.

1Classify Every Failure

Make classification explicit in code (error types or a retryable flag), not ad-hoc catch blocks.

2A Good Retry Policy

async function withRetry<T>(op: () => Promise<T>, opts = { attempts: 4, baseMs: 100, maxMs: 2000, deadlineMs: 5000 }) {
  const start = Date.now();
  for (let attempt = 1; ; attempt++) {
    try {
      return await op();
    } catch (err) {
      if (!isRetryable(err) || attempt >= opts.attempts) throw err;
      const backoff = Math.min(opts.maxMs, opts.baseMs * 2 ** (attempt - 1));
      const delay = Math.random() * backoff;                           // full jitter
      if (Date.now() - start + delay > opts.deadlineMs) throw err;     // respect the caller's deadline
      await sleep(retryAfter(err) ?? delay);                           // honour Retry-After when given
    }
  }
}

Checklist: bounded attempts, exponential backoff, jitter, only retryable errors, only idempotent operations (or with idempotency keys), respect Retry-After, an overall deadline (don't retry past the point where the caller has given up).

3Retry Amplification

Retries multiply across layers. If each of 4 layers retries 3 times (4 attempts each), one failing call at the bottom can be attempted 4⁴ = 256 times. During an overload, retries add load exactly when the system can least afford it — turning a brownout into an outage.

Mitigations:

• Retry at one layer (usually the one closest to the user or to the failing dependency) — others fail fast.
• Retry budgets: allow retries only up to e.g. 10% of request volume per client (gRPC/Envoy support this); once exhausted, fail immediately.
• Circuit breakers stop calling a dependency that's clearly down.
• Deadline propagation: pass the remaining time budget downstream (grpc-timeout, a deadline header) so no layer works on requests whose caller has already timed out.

4Timeouts Everywhere

Every network call needs a timeout — the default in many clients is infinite. Set them from the latency SLO downward: if the user-facing budget is 1 s, a downstream call might get 300 ms with one retry. Separate connect timeouts (short) from request timeouts. A missing timeout turns a slow dependency into exhausted thread/connection pools upstream.

5Failing Gracefully

• Fallbacks: cached/stale data, a default ("recommendations unavailable"), a degraded mode (read-only).
• Partial results: return what you have with an indication of what's missing.
• Queue for later: accept the request, process asynchronously (backend/frameworks/16-background_jobs), with an outbox for reliability.
• Compensate multi-step operations (sagas — distributed-systems/fundamentals/08-distributed_transactions).
• Clear errors to users with request IDs; never leak internals.

Under the HoodWhy Full Jitter Works▾

With pure exponential backoff, every client that failed at time t retries at exactly t + 100ms, t + 300ms, t + 700ms… — synchronized waves that hit the recovering server together. Full jitter picks the delay uniformly in [0, base·2ⁿ], which spreads retries evenly over the window. AWS's analysis ("Exponential Backoff and Jitter") found full jitter minimizes both total work and completion time compared with no jitter or "equal jitter" (base·2ⁿ/2 + random(0, base·2ⁿ/2)). Decorrelated jitter (delay = random(base, prev × 3), capped) is a popular alternative that avoids very short delays repeating. In all variants the key property is the same: destroy the correlation between clients' retry times.

⚠️ Common Pitfalls

1. Retrying non-idempotent operations without idempotency keys — duplicate charges.
2. Retrying permanent errors — wasted capacity, delayed failure.
3. Retries at every layer — amplification during incidents.
4. No timeouts — hung requests and exhausted pools.
5. No jitter — synchronized retry storms.
6. Swallowing errors (catch {}) — silent data loss.

📋 Quick Reference

classify: transient → retry · permanent → fail fast · unknown outcome → idempotency/reconcile · exhaustion → shed
retry: bounded · exponential · full jitter · Retry-After · deadline · idempotent only · ONE layer · budget
timeouts on every call (connect < request) · circuit breakers · fallbacks · compensations

🎤 Interview Q&A

🛠️ Practice exercises

Each exercise is a file in this topic folder. Code exercises are self-checking: run them and make the checks pass.

✅ Check yourself

Source: adv_failure_handling_and_retry_strategies_theory.md · generated by tools/build_lessons.py