# Exercise 2: Diagnose a Failover Incident

## Scenario

`QueueMetrics`, a job-monitoring dashboard, runs one leader and two asynchronous read replicas. An incident report from last week reads:

> 14:02:03 — Automated monitoring on Replica A reports it cannot reach the Leader (network blip between AZs). Automated failover tool promotes Replica A to leader after a 5-second detection window.
> 14:02:04 — The old Leader, which was never actually down (only briefly unreachable from Replica A's specific network path — every other client, including Replica B, remained connected to it the whole time), continues accepting writes from the application, which had not been repointed yet.
> 14:02:03 to 14:02:19 — For 16 seconds, both the old Leader and newly-promoted Replica A accept writes independently.
> 14:02:19 — Engineers notice two "leaders" and manually intervene, but by then some writes exist on one node and not the other, and reconciling them requires manual review.

Separately, the postmortem also notes: three job-completion events that were written to the old Leader at 14:02:02 (one second before the incident began) and acknowledged to the client as successful were never found on Replica A or Replica B after the incident — they simply don't exist anywhere anymore.

## Deliverables

1. **Name the failure mode for the 14:02:03-14:02:19 window** using theory section 5's terminology, and explain precisely what went wrong in the failover tool's detection/promotion logic that allowed it — which specific safeguard from theory section 5 (point 1) was missing or misconfigured?
2. **Explain the three lost job-completion events separately** from the split-brain issue — using theory section 3, explain why writes acknowledged at 14:02:02 could vanish entirely even though the split-brain incident happened a full second later, and what replication mode (if it had been used for at least one follower) would have prevented this specific loss.
3. **Propose a concrete fix to the failover detection logic** — your fix must address why "Replica A can't reach the Leader" was treated as "the Leader is down" when in fact only one network path was affected, referencing the quorum/consensus point in theory section 5.
4. **Explain what "manual review to reconcile" means concretely** in this scenario — what has to happen to the writes that exist only on the old Leader vs. only on newly-promoted Replica A before the two can be treated as one consistent dataset again?