Delivery Guarantees

1The Three Guarantees

Every message passes through failure-prone steps: producer → broker, broker storage, broker → consumer, consumer processing. A delivery guarantee says how many times a message's effect can happen when any of those steps fails.

They are not broker features you switch on; they're properties of the whole pipeline, decided by when each side acknowledges.

2Producer Side

RabbitMQ's equivalent is publisher confirms + durable queues + persistent messages (+ quorum queues for replication).

3Consumer Side

at-most-once:   receive → ACK → process        crash after ack  → message lost
at-least-once:  receive → process → ACK        crash before ack → message redelivered

At-least-once is the practical default: losing data is usually worse than processing it twice, and duplicates can be neutralized.

4"Exactly-Once" in Practice = At-Least-Once + Idempotence

True exactly-once delivery across a network is impossible in general (the Two Generals problem: you can never be sure the other side received your last message). What systems actually provide is exactly-once processing / effect:

1. Idempotent consumers — processing the same message twice has the same effect as once:
   • dedup table: INSERT INTO processed_messages(message_id) ... in the same DB transaction as the side effect; a duplicate hits the unique constraint and is skipped
   • natural idempotency: "set status = SHIPPED", upserts keyed by event id
   • downstream idempotency keys (Idempotency-Key header to a payment API)
2. Transactional read-process-write (Kafka EOS) — a Kafka Streams / transactional producer writes output records and the consumer's offset commit in one Kafka transaction; consumers reading with isolation.level=read_committed never see aborted output. This gives exactly-once for Kafka → Kafka pipelines only — any external side effect (an email, an HTTP call, a non-Kafka DB) is outside the transaction.
3. Store offsets with results — write the processing result and the consumed offset in one database transaction; on restart, seek to the stored offset.

BEGIN;
  INSERT INTO processed_messages (message_id) VALUES ('evt-81f…');
  -- unique violation here → duplicate → ROLLBACK and ack
  UPDATE accounts SET points = points + 50 WHERE id = 'c-4';
COMMIT;
-- then commit the offset / ack the message

5Picking a Guarantee

Under the HoodHow Kafka Transactions Commit Atomically▾

A transactional producer registers a transactional.id with a transaction coordinator (a broker owning a partition of the internal __transaction_state log). Then:

1. beginTransaction() — local state only.
2. Each send() to a new partition first registers that partition with the coordinator (AddPartitionsToTxn), then writes records normally — they're in the log, but marked as part of an open transaction.
3. sendOffsetsToTransaction() registers the consumer group's offsets as part of the same transaction.
4. commitTransaction() — the coordinator writes PREPARE_COMMIT to its log (the point of no return), then writes a commit marker into every involved partition (including __consumer_offsets), then COMPLETE_COMMIT.

read_committed consumers only read up to the last stable offset — the first offset of any still-open transaction — and skip records of aborted transactions. A new producer instance with the same transactional.id bumps an epoch, fencing off a zombie predecessor so it can't commit.

⚠️ Common Pitfalls

1. Believing a broker setting gives exactly-once end-to-end — external side effects are never covered.
2. acks=1 for critical data — leader failure loses acknowledged writes.
3. Dedup check outside the transaction of the side effect — a crash between them still duplicates or loses.
4. Dedup on payload hashes — two genuinely identical, separate events get merged. Use explicit event IDs.
5. Unbounded dedup tables — expire entries after the maximum redelivery window.
6. Auto-commit + async processing — offsets committed for messages still being processed → loss.

📋 Quick Reference

at-most-once  = ack → process
at-least-once = process → ack                         (default)
effectively-once = at-least-once + idempotent handler (dedup id in the same txn)
Kafka EOS     = transactional producer + sendOffsetsToTransaction + read_committed  (Kafka → Kafka only)
durable produce = acks=all + min.insync.replicas ≥ 2 + idempotence

🎤 Interview Q&A

🛠️ Practice exercises

Each exercise is a file in this topic folder. Code exercises are self-checking: run them and make the checks pass.

✅ Check yourself

Source: broker_delivery_guarantees_theory.md · generated by tools/build_lessons.py