# Exercise 2: Trace the Failures — Which Guarantee Did You Actually Get?

Each trace describes a pipeline configuration and a sequence of events including a failure. For each trace, determine **what the downstream effect is** (lost, duplicated, or exactly once) and **which line of configuration or code caused it**. Then fix it with the smallest change.

## Trace A

```
producer: acks=1, retries=0
broker:   replication.factor=3, min.insync.replicas=1
1. producer sends OrderPlaced(o-17) → leader B1 appends at offset 812, replies ACK
2. B1 crashes before followers B2/B3 fetch offset 812
3. B2 becomes leader (it was in the ISR)
4. consumer reads from B2
```

## Trace B

```
producer: acks=all, enable.idempotence=false, retries=5
1. producer sends batch [p-1, p-2]; leader writes it, replicates, sends ACK
2. the ACK is lost on the network; producer times out
3. producer retries the batch
4. consumer processes everything in the partition
```

## Trace C

```
consumer: enable.auto.commit=true, auto.commit.interval.ms=5000
handler:  await sendInvoiceEmail(event)   // takes ~8 s per message
1. poll() returns [i-1, i-2, i-3]
2. auto-commit fires at t=5s committing offset after i-3
3. consumer process is OOM-killed at t=9s while sending the email for i-2
4. replacement consumer starts from the committed offset
```

## Trace D

```
consumer (RabbitMQ): manual ack, prefetch=10
handler:
  await db.query("INSERT INTO shipments ...", ...)   // commits
  await channel.ack(msg)
1. handler inserts the shipment row for s-4
2. the consumer's TCP connection drops before the ack reaches the broker
3. broker redelivers s-4 to another consumer
```

## Trace E

```
Kafka Streams app, processing.guarantee=exactly_once_v2
reads `payments`, writes `ledger-entries` (Kafka) AND calls the bank's REST API to move money
1. app processes p-9, writes ledger entry, calls bank API (succeeds)
2. app crashes before the transaction commits
3. on restart, the transaction is aborted and p-9 is reprocessed
```

## Deliverables

For each trace (A–E):

1. The outcome: **lost**, **duplicated**, or **exactly once**.
2. The root cause, pointing at the specific setting or line.
3. The minimal fix.

Finally:

4. Rank the five fixes by how much they cost at run time (latency/throughput), and say which ones you'd apply to a pipeline handling **money**.

## Requirements Checklist

- [ ] A: **lost** — `acks=1` with `min.insync.replicas=1`; fix `acks=all` + `min.insync.replicas=2`.
- [ ] B: **duplicated** — retries without idempotence; fix `enable.idempotence=true`.
- [ ] C: **lost** (i-2, i-3 were committed but never processed; i-1 may be fine) — auto-commit commits polled, not processed, offsets; fix: manual commit after processing.
- [ ] D: **duplicated** — the DB commit happened but the ack didn't; fix: idempotent insert (unique shipment id / `ON CONFLICT DO NOTHING`) — the ack can never be made atomic with the DB write.
- [ ] E: the ledger entry is exactly once (aborted + rewritten), but the **bank transfer is duplicated** — EOS doesn't cover external calls; fix: idempotency key to the bank API (or move the call to a separate at-least-once consumer with dedup).
- [ ] The ranking recognizes `acks=all` and per-message manual commits cost latency/throughput, idempotence is nearly free, and all of A, B, D, E fixes apply to money.