Fault Tolerance and High Availability

1Two Related but Distinct Goals

Fault tolerance is a system's ability to keep functioning correctly when a component fails — a server crashes, a disk dies, a process hangs — without that failure becoming a user-visible failure. High availability (HA) is the resulting property: the system, as a whole, stays reachable and responsive a very high percentage of the time. Fault tolerance is the mechanism; high availability is the outcome it produces. A system can be fault-tolerant for some failure modes and not others — the discipline here is being explicit about exactly which failures you've actually designed for.

2Redundancy: The Foundation of Fault Tolerance

You cannot tolerate the failure of a component you only have one of. Redundancy means running more than one instance of a component so that one failing doesn't take the whole capability down with it — but how those redundant instances are used matters:

Active-passive (a.k.a. primary-standby): one instance handles all traffic; one or more standbys sit idle, ready to take over.

[Primary: handling all traffic] --- replicating state --->  [Standby: idle, up to date]

Primary fails -> Standby is PROMOTED to primary -> traffic redirected to it

The standby does no useful work most of the time — it exists purely as insurance — and the failover (detecting the failure + promoting the standby + redirecting traffic) takes some nonzero time, during which the system is genuinely down.

Active-active: multiple instances all handle traffic simultaneously; if one fails, the others simply absorb its share of load.

[Instance A: handling 1/3 of traffic]
[Instance B: handling 1/3 of traffic]  <- load balanced across all three
[Instance C: handling 1/3 of traffic]

Instance B fails -> load balancer stops routing to it -> A and C absorb B's share
(no "promotion" step -- A and C were ALREADY fully capable of serving traffic)

Active-active avoids the idle-capacity waste of active-passive and typically fails over faster (no promotion step, just "stop sending traffic to the dead one"), but it requires every active instance to be capable of handling the full range of traffic correctly at all times, and — for anything stateful — requires that state to actually be shared or replicated in a way multiple simultaneously-active writers can use safely (see the CAP theorem topic for what that tradeoff costs).

3Detecting Failure: Health Checks

Redundancy only helps if the system actually notices a component has failed and reacts — otherwise traffic keeps flowing to a dead or broken instance regardless of how many healthy replicas exist elsewhere. A health check is a periodic, automated probe:

Load balancer, every 5 seconds, to each backend instance:
  GET /health -> expects 200 OK within 2 seconds

3 consecutive failures -> mark instance UNHEALTHY -> stop routing traffic to it
2 consecutive successes (after being unhealthy) -> mark HEALTHY again -> resume routing

The "N consecutive failures/successes" thresholds exist deliberately, not arbitrarily: a health check that reacts to a single failed probe is too twitchy (one dropped packet or one slow GC pause takes an otherwise-healthy instance out of rotation), while one that takes too long to react leaves traffic flowing to a genuinely dead instance for longer than necessary. A meaningful health check endpoint also needs to actually verify the instance CAN serve real traffic (e.g. check it can reach its own database connection pool), not just that the process is alive and listening — a shallow "always return 200" health check defeats the entire purpose.

4Eliminating Single Points of Failure (SPOFs)

A single point of failure is any one component whose failure takes down the whole system, regardless of how redundant everything else is. Redundancy at one layer is worthless if a different, non-redundant layer sits underneath or beside it:

[3x redundant web servers] -> [1 load balancer] -> the load balancer IS a SPOF
[3x redundant web servers] -> [2x redundant load balancers, active-passive] -> [1 database] -> the database IS a SPOF

Eliminating a SPOF means applying the same redundancy thinking to EVERY layer a request passes through, not just the layer that's easiest to scale horizontally (typically the stateless application tier) — the load balancer itself, DNS, the database, shared caches, and even things like a single power supply or a single availability zone/datacenter are all candidate SPOFs in a system that hasn't explicitly addressed them.

5Availability Math: The Nines

Availability is usually expressed as a percentage of time a system is genuinely reachable and correct, over a period (commonly a year):

Each additional nine is roughly an order-of-magnitude reduction in acceptable downtime, and in practice an order-of-magnitude increase in engineering cost and complexity to achieve — the jump from three nines to four nines typically requires eliminating SPOFs that were "good enough" at three nines, faster automated failover instead of manual intervention, and often multi-region (not just multi-instance) redundancy. Chasing more nines than a system's actual business requirements justify is a common and expensive mistake; five-nines infrastructure for an internal admin tool used by 10 employees is effort spent on the wrong problem.

6Graceful Degradation

Not every failure has to mean full unavailability. Graceful degradation means a system keeps serving its core functionality even when a non-critical dependency has failed, rather than treating every failure as total:

E-commerce product page, "recommended items" service is DOWN:
  Bad:  the whole product page fails to load (recommendations became a SPOF
        for a feature that isn't actually essential to buying the product)
  Good: the product page loads normally, WITHOUT a recommendations section
        (the core purchase flow is entirely unaffected by that dependency's failure)

This connects directly to earlier topics: a circuit breaker that's open for the recommendations service should fail fast and let the page render without that section, rather than the whole page request hanging or erroring because one non-essential dependency is down — bulkheading (previous topic) is often exactly what keeps that failure from consuming resources needed by the actually-critical parts of the request.

const [product, recs] = await Promise.all([
  getProduct(id), getRecommendations(id)   // recs down → whole page 500
]);

const product = await getProduct(id);
const recs = await recsBreaker.call(() => getRecommendations(id))
  .catch(() => []);                         // page renders without the section

⚠️ Common Pitfalls

1. Redundancy at only one layer — three redundant application servers behind one non-redundant load balancer or one non-redundant database just relocates the SPOF, it doesn't eliminate it (Section 4).
2. A health check that only verifies the process is running, not that it can actually serve real traffic — keeps routing to an instance that's "alive" but functionally broken (e.g. its database connection pool is exhausted).
3. Health check thresholds set too aggressively — reacting to a single failed probe flaps healthy instances in and out of rotation on ordinary jitter, which can itself cause instability.
4. Treating every dependency as equally critical — a non-essential dependency (recommendations, "recently viewed") being allowed to take down an entire page defeats the purpose of having redundancy anywhere else; graceful degradation requires explicitly deciding what's core vs optional.
5. Chasing availability nines disconnected from actual business need — five-nines engineering effort applied to a system where three nines would be entirely sufficient is cost without corresponding value.

📋 Quick Reference

Fault tolerance = mechanism (surviving a component failure without a user-visible outage)
High availability = the resulting outcome (system stays reachable a high % of the time)

Active-passive: standby idle until promoted -- failover has a detection+promotion gap
Active-active: all instances live, survivors just absorb the dead one's share -- faster failover

Health checks: N-consecutive-failures/successes thresholds; must verify REAL serving capability
SPOF = ANY single component whose failure takes down the whole system -- check every layer

Nines: 99% ~ 3.65 days/yr down | 99.9% ~ 8.76 hr/yr | 99.99% ~ 52.6 min/yr | 99.999% ~ 5.26 min/yr

Graceful degradation: a non-critical dependency failing should degrade a FEATURE, not the
                       whole system -- pairs directly with circuit breaking + bulkheading

🎤 Interview Q&A

🛠️ Practice exercises

Each exercise is a file in this topic folder. Code exercises are self-checking: run them and make the checks pass.

✅ Check yourself

Source: dist_fault_tolerance_and_high_availability_theory.md · generated by tools/build_lessons.py