# Delivery Guarantees

## 1. The Three Guarantees

Every message passes through failure-prone steps: producer → broker, broker storage, broker → consumer, consumer processing. A **delivery guarantee** says how many times a message's effect can happen when any of those steps fails.

| Guarantee | Meaning | Failure mode you accept |
|---|---|---|
| **at-most-once** | 0 or 1 times | messages can be **lost** |
| **at-least-once** | 1 or more times | messages can be **duplicated** |
| **exactly-once** | exactly 1 time | (requires cooperation from every component) |

They are not broker *features* you switch on; they're properties of the **whole pipeline**, decided by when each side acknowledges.

## 2. Producer Side

| Setting | Loss on broker failure? | Duplicates on retry? |
|---|---|---|
| fire-and-forget (`acks=0`, no confirms) | yes | no |
| wait for leader (`acks=1`) | yes, if the leader dies before followers copy it | yes, if the ack is lost and you retry |
| wait for all in-sync replicas (`acks=all` + `min.insync.replicas=2`) | no (as long as one ISR replica survives) | yes, unless idempotent |
| `acks=all` + **idempotent producer** | no | no (within the producer session) |

RabbitMQ's equivalent is **publisher confirms** + durable queues + persistent messages (+ quorum queues for replication).

## 3. Consumer Side

```
at-most-once:   receive → ACK → process        crash after ack  → message lost
at-least-once:  receive → process → ACK        crash before ack → message redelivered
```

At-least-once is the practical default: losing data is usually worse than processing it twice, and duplicates can be neutralized.

## 4. "Exactly-Once" in Practice = At-Least-Once + Idempotence

True exactly-once *delivery* across a network is impossible in general (the Two Generals problem: you can never be sure the other side received your last message). What systems actually provide is **exactly-once processing / effect**:

1. **Idempotent consumers** — processing the same message twice has the same effect as once:
   - dedup table: `INSERT INTO processed_messages(message_id) ...` in the **same DB transaction** as the side effect; a duplicate hits the unique constraint and is skipped
   - natural idempotency: "set status = SHIPPED", upserts keyed by event id
   - downstream idempotency keys (`Idempotency-Key` header to a payment API)
2. **Transactional read-process-write (Kafka EOS)** — a Kafka Streams / transactional producer writes output records **and** the consumer's offset commit in **one Kafka transaction**; consumers reading with `isolation.level=read_committed` never see aborted output. This gives exactly-once for **Kafka → Kafka** pipelines only — any external side effect (an email, an HTTP call, a non-Kafka DB) is outside the transaction.
3. **Store offsets with results** — write the processing result and the consumed offset in one database transaction; on restart, seek to the stored offset.

## 5. Picking a Guarantee

| Data | Guarantee | Why |
|---|---|---|
| metrics, telemetry, logs | at-most-once (or at-least-once) | a lost data point is fine; throughput matters |
| emails, notifications | at-least-once + dedup | a missing email is worse than a rare duplicate; dedup makes duplicates rare |
| payments, orders, ledger entries | at-least-once + **strict** idempotency | duplicates = money; loss = money |
| stream aggregation in Kafka | Kafka EOS | built-in transactional semantics |

## Under the Hood: How Kafka Transactions Commit Atomically

A transactional producer registers a `transactional.id` with a **transaction coordinator** (a broker owning a partition of the internal `__transaction_state` log). Then:

1. `beginTransaction()` — local state only.
2. Each `send()` to a new partition first registers that partition with the coordinator (`AddPartitionsToTxn`), then writes records normally — they're in the log, but marked as part of an open transaction.
3. `sendOffsetsToTransaction()` registers the consumer group's offsets as part of the same transaction.
4. `commitTransaction()` — the coordinator writes **PREPARE_COMMIT** to its log (the point of no return), then writes a **commit marker** into every involved partition (including `__consumer_offsets`), then **COMPLETE_COMMIT**.

`read_committed` consumers only read up to the **last stable offset** — the first offset of any still-open transaction — and skip records of aborted transactions. A new producer instance with the same `transactional.id` bumps an **epoch**, fencing off a zombie predecessor so it can't commit.

![At-most-once acks before processing and loses messages on a crash, at-least-once processes before acking and redelivers duplicates, and effectively-once adds an idempotent dedup check inside the same transaction as the side effect](diagrams/01-three-guarantees.svg)

## Common Pitfalls

1. **Believing a broker setting gives exactly-once end-to-end** — external side effects are never covered.
2. **`acks=1` for critical data** — leader failure loses acknowledged writes.
3. **Dedup check outside the transaction** of the side effect — a crash between them still duplicates or loses.
4. **Dedup on payload hashes** — two genuinely identical, separate events get merged. Use explicit event IDs.
5. **Unbounded dedup tables** — expire entries after the maximum redelivery window.
6. **Auto-commit + async processing** — offsets committed for messages still being processed → loss.

## Quick Reference

```
at-most-once  = ack → process
at-least-once = process → ack                         (default)
effectively-once = at-least-once + idempotent handler (dedup id in the same txn)
Kafka EOS     = transactional producer + sendOffsetsToTransaction + read_committed  (Kafka → Kafka only)
durable produce = acks=all + min.insync.replicas ≥ 2 + idempotence
```