Circuit Breaker

This topic assumes you've read 06-retry_with_exponential_backoff/. Retry with backoff answers "how should ONE call handle a transient failure of the thing it's calling." A circuit breaker answers a related but different question: "once a downstream dependency is CONSISTENTLY failing, how do we stop every caller from continuing to hammer it at all — retries included." Part 2 covers half-open probe mechanics in depth, combining circuit breaking with retry/backoff, and bulkheading.

1The Problem Retry Alone Doesn't Solve

Retry with backoff makes each individual failing call more polite about retrying. But if a downstream service is fully down — not flaky, just down — every caller, even with perfect backoff and jitter, is still eventually going to retry it, over and over, forever, each retry costing a connection attempt, a timeout to expire, resources on both sides. Multiply that by every caller in the system, and a downstream outage keeps consuming capacity on the calling side too, and keeps applying load to a downstream service that needs to be left alone to recover, not continuously probed by everyone.

Downstream service goes down at t=0.
Without a circuit breaker:
  Every caller keeps calling (with backoff, so not instantly, but still --
  forever) -> every call still pays a full connection/timeout cost ->
  callers' own resources (threads, connections) get tied up waiting on a
  dependency that has no chance of responding -> the FAILURE SPREADS to
  services that depend on the caller, even though those services have
  nothing to do with the original outage.

This upstream resource exhaustion — not the original outage itself — is very often what actually takes down a system during a dependency failure. A circuit breaker's job is to stop that spread by making the caller stop trying, fast, once it's clear the downstream is not going to succeed.

2The Three States

A circuit breaker wraps calls to a dependency and tracks their outcomes, moving between three states:

        failures exceed threshold
  CLOSED ─────────────────────────▶ OPEN
    ▲                                 │
    │                                 │ after a cooldown timeout
    │ probe succeeds                  ▼
    └──────────────────────── HALF-OPEN
                probe fails
              (back to OPEN)

• Closed — the normal state. Calls pass through to the real dependency. The breaker counts recent failures.
• Open — the breaker has seen enough failures to conclude the dependency is down. Calls fail immediately, without even attempting the real call — this is the entire point: stop spending resources on a call that's overwhelmingly likely to fail anyway.
• Half-open — after a cooldown period, the breaker allows a small number of "probe" calls through to test whether the dependency has recovered. Success moves back to Closed; failure moves back to Open (and resets the cooldown).

type CircuitState = "closed" | "open" | "half-open";

interface Breaker {
  state: CircuitState;
  consecutiveFailures: number;
  failureThreshold: number;   // e.g. 5 -- trip to OPEN after 5 in a row
  openedAt: number | null;    // timestamp, used to know when cooldown elapses
  cooldownMs: number;         // e.g. 30000 -- how long to stay OPEN before probing
}

3Closed → Open: Tripping the Breaker

While closed, every failed call increments a failure counter; every successful call resets it. Once consecutive (or windowed — Part 2 covers the distinction) failures cross failureThreshold, the breaker trips to open and remembers when:

function recordFailure(breaker: Breaker, now: number): void {
  breaker.consecutiveFailures++;
  if (breaker.state === "closed" && breaker.consecutiveFailures >= breaker.failureThreshold) {
    breaker.state = "open";
    breaker.openedAt = now;
  }
}

Once open, calls are rejected without attempting the real dependency at all:

function call<T>(breaker: Breaker, fn: () => T, now: number): T {
  if (breaker.state === "open") {
    if (now - (breaker.openedAt ?? 0) < breaker.cooldownMs) {
      throw new Error("circuit open — failing fast, not calling downstream");
    }
    breaker.state = "half-open"; // cooldown elapsed, allow a probe through
  }
  try {
    const result = fn();
    recordSuccess(breaker);
    return result;
  } catch (err) {
    recordFailure(breaker, now);
    throw err;
  }
}

This is the core mechanical difference from retry: retry still calls the real dependency, just with delay between attempts. An open circuit breaker doesn't call the dependency at all — it fails locally and instantly, which is exactly what stops the caller-side resource exhaustion described in Section 1.

4Half-Open: Testing Recovery Without Fully Committing

Going straight from Open back to Closed after a fixed timeout would be guessing blind — the dependency might still be down. Half-open lets through a small number of probe calls (often just one) to actually test the water:

• Probe succeeds -> the dependency is very likely back -> transition to Closed, reset the failure counter.
• Probe fails -> still down -> transition back to Open, restart the cooldown timer.

Crucially, while half-open, the breaker does not let the full flood of normal traffic through — only the limited probe volume — so a still-broken dependency isn't immediately re-hammered by every caller the moment the cooldown expires.

⚠️ Common Pitfalls

1. Treating retry and circuit breaking as interchangeable — retry handles a single call's transient blip; a circuit breaker handles a sustained failure by stopping calls altogether. They solve different problems and are normally used together (Part 2).
2. No half-open state at all (binary open/closed with a timer) — either keeps failing fast forever with no way back to closed, or flips straight back to closed and gets immediately re-tripped by full traffic against a dependency that's still down.
3. Letting the full traffic volume through during half-open — defeats the purpose of probing cautiously; a single bad probe request among thousands re-floods a barely-recovering dependency.
4. Sizing failureThreshold too low — trips the breaker (and starts failing fast for ALL callers) on normal, brief error-rate noise rather than a genuine sustained outage.
5. No cooldown, or a cooldown so short it barely differs from immediate retry — reopens the door to the dependency before it's had any real time to recover.

📋 Quick Reference

States: CLOSED (normal, counts failures) -> OPEN (fail fast, NO real call) -> HALF-OPEN (probe) -> CLOSED or OPEN

Open circuit = calls fail LOCALLY and INSTANTLY -- the defining difference from retry,
               which still calls the real dependency, just with delay

Trip condition: consecutive/windowed failures >= failureThreshold  ->  CLOSED to OPEN
Recovery test:  after cooldownMs elapses, OPEN -> HALF-OPEN, allow limited probe traffic
Probe succeeds -> HALF-OPEN -> CLOSED (reset failure count)
Probe fails    -> HALF-OPEN -> OPEN (restart cooldown)

Part 1 covered the three states and a consecutive-failure trip condition. This part covers a more realistic failure-rate-based trip condition, exactly how circuit breaking and retry-with-backoff compose together (they are not alternatives), and bulkheading as a complementary isolation pattern.

1Consecutive Failures vs a Sliding Window of Failure Rate

Part 1's consecutiveFailures >= failureThreshold trips on N failures in a row. That's simple, but has a rough edge: a single success resets the counter to zero, so a dependency failing 90% of the time but happening to succeed once every few calls (by chance, or because 10% of instances behind a load balancer are healthy) might never trip the breaker, even though callers are still failing constantly.

A sliding window failure rate is the more realistic real-world condition: trip based on the proportion of failures over the last N calls (or the last T seconds), not a strict consecutive streak:

interface WindowedBreaker {
  state: CircuitState;
  recentOutcomes: boolean[]; // true = success, false = failure; capped at windowSize
  windowSize: number;        // e.g. 20 -- look at the last 20 calls
  failureRateThreshold: number; // e.g. 0.5 -- trip if >= 50% of the window failed
  minimumCalls: number;      // e.g. 10 -- don't evaluate the rate on too few samples
  openedAt: number | null;
  cooldownMs: number;
}

function recordOutcome(breaker: WindowedBreaker, success: boolean, now: number): void {
  breaker.recentOutcomes.push(success);
  if (breaker.recentOutcomes.length > breaker.windowSize) {
    breaker.recentOutcomes.shift(); // keep only the most recent windowSize outcomes
  }
  if (breaker.state !== "closed" || breaker.recentOutcomes.length < breaker.minimumCalls) return;

  const failureCount = breaker.recentOutcomes.filter((s) => !s).length;
  const failureRate = failureCount / breaker.recentOutcomes.length;
  if (failureRate >= breaker.failureRateThreshold) {
    breaker.state = "open";
    breaker.openedAt = now;
  }
}

The minimumCalls guard matters: evaluating a failure rate on only 1-2 samples is statistically meaningless (1 failure out of 1 call is a 100% failure rate) — real breaker implementations refuse to trip until there's enough sample volume to trust the rate.

2Combining Circuit Breaking With Retry: Order Matters

These two patterns are almost always used together, layered in a specific order — retry on the inside, circuit breaker on the outside:

async function callWithResilience<T>(
  breaker: Breaker,
  fn: () => Promise<T>,
  retryOptions: RetryOptions
): Promise<T> {
  return call(breaker, () => retryWithBackoff(fn, retryOptions), Date.now());
  //     ^ circuit breaker                ^ retry, wrapped INSIDE the breaker's call
}

Why this order and not the reverse: the circuit breaker needs to see each retry-exhausted attempt as a single outcome (success or failure) to correctly judge the dependency's overall health, not each individual retry within it. If the breaker instead wrapped a single non-retrying call and retry wrapped the breaker, every one of retry's individual attempts would separately count against the breaker's trip condition — inflating the apparent failure rate and tripping the breaker on transient blips that retry, on its own, would have quietly absorbed. Layered correctly: retry absorbs brief transient blips per-call; the breaker watches the aggregate success/failure of those retried calls over time to detect a sustained outage retry alone can't fix (and once it trips, stops paying the cost of even attempting the retries).

3Bulkheading: The Complementary Isolation Pattern

A circuit breaker protects a caller from a specific failing dependency by stopping calls to it. It does nothing, by itself, to stop that dependency's slowness from consuming a shared resource pool (a thread pool, a connection pool) that other, healthy dependencies also depend on — while the breaker is still closed and probing, or during the window before it trips, calls to the slow dependency can still tie up every available thread/connection, starving calls to completely unrelated, healthy dependencies.

Bulkheading (named for a ship's watertight compartments) addresses this by giving each dependency its own isolated resource pool — its own connection pool, its own thread pool, its own concurrent-request limit — so that one dependency saturating its pool cannot starve calls to any other dependency:

Without bulkheading:                    With bulkheading:
  [shared thread pool: 50]                [pool A: 20] -> Dependency A (slow/failing)
   -> Dependency A (slow)                 [pool B: 20] -> Dependency B (healthy, unaffected)
   -> Dependency B (healthy)              [pool C: 10] -> Dependency C (healthy, unaffected)
  A's slowness can consume all 50          A's slowness can consume at most its own 20
  threads, starving B entirely             B and C are structurally protected from A

Circuit breaking and bulkheading are complementary, not redundant: bulkheading limits the blast radius of a dependency being slow (even before enough failures have accumulated to trip a breaker), while the circuit breaker stops continuing to try a dependency that's confirmed to be failing. A resilient call path to an external dependency typically has both: a bounded, isolated resource pool (bulkhead) AND a circuit breaker watching outcomes through that pool, with retry-with-backoff wrapped innermost.

⚠️ Common Pitfalls

1. Trip condition based on tiny sample sizes — a failureRateThreshold evaluated with no minimumCalls guard trips (or fails to trip) on statistical noise rather than a real sustained failure pattern.
2. Wrapping the circuit breaker INSIDE retry instead of the other way around — makes the breaker see every individual retry attempt as a separate outcome, tripping it on transient blips retry alone would have absorbed, and losing the "stop calling entirely" benefit during retries.
3. Assuming a circuit breaker alone prevents resource exhaustion — it stops calls once tripped, but does nothing about a shared thread/connection pool being saturated by a slow (not-yet-tripped) dependency; that's bulkheading's job.
4. One global breaker for all dependencies instead of one breaker per dependency — a failure in dependency A shouldn't fail-fast calls to unrelated, healthy dependency B.
5. Never resetting the sliding window after a long healthy period — stale old failures sitting in a window sized by call-count rather than time can distort the rate if traffic is bursty; time-windowed variants (or clearing on a long enough gap) address this.

📋 Quick Reference

Consecutive-failure trip: simple, but a single lucky success resets it -- can mask a
                           dependency that's failing most, but not all, of the time
Windowed failure-rate trip: trips on the PROPORTION of recent failures; always guard
                           with a minimumCalls sample-size floor before evaluating

Layering order: retry (innermost, absorbs per-call blips) -> circuit breaker (outermost,
                watches AGGREGATE outcomes, stops calling once tripped)

Bulkheading: isolated resource pool PER DEPENDENCY -- limits blast radius of one slow
             dependency starving calls to unrelated healthy ones; complements, doesn't
             replace, circuit breaking

🎤 Interview Q&A

🛠️ Practice exercises

Each exercise is a file in this topic folder. Code exercises are self-checking: run them and make the checks pass.

✅ Check yourself

Source: dist_circuit_breaker_theory_1.md, dist_circuit_breaker_theory_2.md · generated by tools/build_lessons.py