# Circuit Breaker — Part 2: Windowed Failures, Combining With Retry, and Bulkheading

Part 1 covered the three states and a consecutive-failure trip condition. This part covers a more realistic failure-rate-based trip condition, exactly how circuit breaking and retry-with-backoff compose together (they are not alternatives), and bulkheading as a complementary isolation pattern.

## 1. Consecutive Failures vs a Sliding Window of Failure Rate

Part 1's `consecutiveFailures >= failureThreshold` trips on N failures **in a row**. That's simple, but has a rough edge: a single success resets the counter to zero, so a dependency failing 90% of the time but happening to succeed once every few calls (by chance, or because 10% of instances behind a load balancer are healthy) might never trip the breaker, even though callers are still failing constantly.

A **sliding window failure rate** is the more realistic real-world condition: trip based on the *proportion* of failures over the last N calls (or the last T seconds), not a strict consecutive streak:

```ts
interface WindowedBreaker {
  state: CircuitState;
  recentOutcomes: boolean[]; // true = success, false = failure; capped at windowSize
  windowSize: number;        // e.g. 20 -- look at the last 20 calls
  failureRateThreshold: number; // e.g. 0.5 -- trip if >= 50% of the window failed
  minimumCalls: number;      // e.g. 10 -- don't evaluate the rate on too few samples
  openedAt: number | null;
  cooldownMs: number;
}

function recordOutcome(breaker: WindowedBreaker, success: boolean, now: number): void {
  breaker.recentOutcomes.push(success);
  if (breaker.recentOutcomes.length > breaker.windowSize) {
    breaker.recentOutcomes.shift(); // keep only the most recent windowSize outcomes
  }
  if (breaker.state !== "closed" || breaker.recentOutcomes.length < breaker.minimumCalls) return;

  const failureCount = breaker.recentOutcomes.filter((s) => !s).length;
  const failureRate = failureCount / breaker.recentOutcomes.length;
  if (failureRate >= breaker.failureRateThreshold) {
    breaker.state = "open";
    breaker.openedAt = now;
  }
}
```

The `minimumCalls` guard matters: evaluating a failure *rate* on only 1-2 samples is statistically meaningless (1 failure out of 1 call is a 100% failure rate) — real breaker implementations refuse to trip until there's enough sample volume to trust the rate.

## 2. Combining Circuit Breaking With Retry: Order Matters

These two patterns are almost always used **together**, layered in a specific order — retry on the *inside*, circuit breaker on the *outside*:

```ts
async function callWithResilience<T>(
  breaker: Breaker,
  fn: () => Promise<T>,
  retryOptions: RetryOptions
): Promise<T> {
  return call(breaker, () => retryWithBackoff(fn, retryOptions), Date.now());
  //     ^ circuit breaker                ^ retry, wrapped INSIDE the breaker's call
}
```

Why this order and not the reverse: the circuit breaker needs to see each **retry-exhausted attempt** as a single outcome (success or failure) to correctly judge the dependency's overall health, not each *individual* retry within it. If the breaker instead wrapped a single non-retrying call and retry wrapped the breaker, every one of retry's individual attempts would separately count against the breaker's trip condition — inflating the apparent failure rate and tripping the breaker on transient blips that retry, on its own, would have quietly absorbed. Layered correctly: retry absorbs brief transient blips per-call; the breaker watches the *aggregate* success/failure of those retried calls over time to detect a sustained outage retry alone can't fix (and once it trips, stops paying the cost of even attempting the retries).

## 3. Bulkheading: The Complementary Isolation Pattern

A circuit breaker protects a caller from a *specific failing dependency* by stopping calls to it. It does nothing, by itself, to stop that dependency's slowness from consuming a **shared resource pool** (a thread pool, a connection pool) that other, healthy dependencies also depend on — while the breaker is still closed and probing, or during the window before it trips, calls to the slow dependency can still tie up every available thread/connection, starving calls to completely unrelated, healthy dependencies.

**Bulkheading** (named for a ship's watertight compartments) addresses this by giving each dependency its **own isolated resource pool** — its own connection pool, its own thread pool, its own concurrent-request limit — so that one dependency saturating its pool cannot starve calls to any other dependency:

```
Without bulkheading:                    With bulkheading:
  [shared thread pool: 50]                [pool A: 20] -> Dependency A (slow/failing)
   -> Dependency A (slow)                 [pool B: 20] -> Dependency B (healthy, unaffected)
   -> Dependency B (healthy)              [pool C: 10] -> Dependency C (healthy, unaffected)
  A's slowness can consume all 50          A's slowness can consume at most its own 20
  threads, starving B entirely             B and C are structurally protected from A
```

Circuit breaking and bulkheading are complementary, not redundant: bulkheading limits the *blast radius* of a dependency being slow (even before enough failures have accumulated to trip a breaker), while the circuit breaker stops *continuing to try* a dependency that's confirmed to be failing. A resilient call path to an external dependency typically has both: a bounded, isolated resource pool (bulkhead) AND a circuit breaker watching outcomes through that pool, with retry-with-backoff wrapped innermost.

## Common Pitfalls

1. **Trip condition based on tiny sample sizes** — a `failureRateThreshold` evaluated with no `minimumCalls` guard trips (or fails to trip) on statistical noise rather than a real sustained failure pattern.
2. **Wrapping the circuit breaker INSIDE retry** instead of the other way around — makes the breaker see every individual retry attempt as a separate outcome, tripping it on transient blips retry alone would have absorbed, and losing the "stop calling entirely" benefit during retries.
3. **Assuming a circuit breaker alone prevents resource exhaustion** — it stops *calls* once tripped, but does nothing about a shared thread/connection pool being saturated by a slow (not-yet-tripped) dependency; that's bulkheading's job.
4. **One global breaker for all dependencies** instead of one breaker per dependency — a failure in dependency A shouldn't fail-fast calls to unrelated, healthy dependency B.
5. **Never resetting the sliding window after a long healthy period** — stale old failures sitting in a window sized by call-count rather than time can distort the rate if traffic is bursty; time-windowed variants (or clearing on a long enough gap) address this.

## Quick Reference

```
Consecutive-failure trip: simple, but a single lucky success resets it -- can mask a
                           dependency that's failing most, but not all, of the time
Windowed failure-rate trip: trips on the PROPORTION of recent failures; always guard
                           with a minimumCalls sample-size floor before evaluating

Layering order: retry (innermost, absorbs per-call blips) -> circuit breaker (outermost,
                watches AGGREGATE outcomes, stops calling once tripped)

Bulkheading: isolated resource pool PER DEPENDENCY -- limits blast radius of one slow
             dependency starving calls to unrelated healthy ones; complements, doesn't
             replace, circuit breaking
```