# Exercise 1: Make a Product Page Resilient

## Scenario

The `product-page` service (Node, 20 instances, each with 50 concurrent request slots) renders a product page by calling, **in parallel**:

| Dependency | p50 | p99 | Availability | Needed for |
|---|---|---|---|---|
| `catalog` (product data) | 20 ms | 80 ms | 99.95% | essential |
| `pricing` | 15 ms | 60 ms | 99.9% | essential (can't sell without a price) |
| `inventory` | 25 ms | 120 ms | 99.5% | "In stock" badge |
| `reviews` | 60 ms | 900 ms | 99.0% | reviews section |
| `recommendations` (ML) | 120 ms | 2,500 ms | 98% | "You may also like" |

Today: no timeouts (HTTP client default: none), all calls share one connection pool of 100 per instance, the page waits for **all** five calls, and each client library retries 3 times on any error. Last Black Friday, `recommendations` slowed to 10 s per call; within 4 minutes every `product-page` instance was saturated and the whole site went down, even though `catalog` and `pricing` were healthy.

## Deliverables

1. **Explain the outage** mechanically: Little's law (in-flight = throughput × latency), shared pool exhaustion, and the role of the 3× retries.
2. **Classify** each dependency: essential vs optional, and define the **fallback** for each optional one.
3. **Timeouts**: pick a timeout per dependency (justify from p99 and the page's 300 ms p99 budget). Do you retry any of them? Where?
4. **Bulkheads**: per-dependency concurrency limits per instance (numbers), and what happens when a bulkhead is full.
5. **Circuit breakers**: which dependencies get one, with what thresholds (failure rate, minimum calls, open duration, half-open probes)?
6. **Page assembly**: rewrite the orchestration (pseudocode or TypeScript) so the page renders as soon as essentials are ready and optional parts are filled or replaced by fallbacks within the budget.
7. **Composite availability**: compute the page's availability before (all 5 required) and after (only catalog + pricing required).
8. **Chaos experiment** to prove the fix: hypothesis, injected fault, metrics, abort conditions.

## Requirements Checklist

- [ ] Outage explanation: 10 s latency × request rate ⇒ in-flight requests far exceed slots/pool; retries multiply calls to the slow dependency; shared pool starves healthy dependencies.
- [ ] Essential: catalog, pricing. Optional with fallbacks: inventory ("check availability at checkout" or cached stock), reviews (cached summary / hide), recommendations (bestsellers / hide).
- [ ] Timeouts below the page budget (e.g. catalog/pricing ~150 ms, inventory ~150 ms, reviews/recommendations ~200 ms) and retries removed or limited to one fast retry for essentials within budget.
- [ ] Bulkheads sized per dependency (e.g. recommendations ≤ 5–10 concurrent per instance) with "full → fallback immediately".
- [ ] Circuit breakers on optional slow dependencies (at least recommendations and reviews) with concrete thresholds.
- [ ] Orchestration uses `Promise.allSettled`/races with per-call timeouts and fallbacks; essentials failing returns an error page or cached product page.
- [ ] Availability math: before ≈ 0.9995 × 0.999 × 0.995 × 0.99 × 0.98 ≈ 96.4%; after ≈ 0.9995 × 0.999 ≈ 99.85%.
- [ ] Chaos experiment injects latency into recommendations under load and checks page p99 and error rate stay within SLO, with an abort threshold.