# Dead Letter Queues

## 1. The Poison Message Problem

Some messages will **never** succeed: malformed JSON, a schema the consumer doesn't understand, a reference to a deleted customer, a bug triggered by one specific payload. Under at-least-once delivery, such a **poison message** is redelivered forever:

- in a queue with requeue-on-failure, it spins in a hot loop, burning CPU and filling logs
- in a Kafka partition, it **blocks every message behind it** (cumulative commits, `08-acknowledgements_and_message_ordering/` §6)

A **dead letter queue (DLQ)** is where messages go after they've failed enough times, so the main flow keeps moving and a human (or tool) can inspect them.

## 2. Transient vs Permanent Failures

The first decision on every failure: **retry or dead-letter?**

| Failure | Example | Action |
|---|---|---|
| transient | timeout, 503, DB failover, rate limited | retry with backoff |
| permanent | invalid payload, unknown schema version, 400/404/422 from a dependency, business rule violation | dead-letter immediately (retrying can't help) |
| unknown | unexpected exception | retry a few times, then dead-letter |

Classify errors explicitly in code (`class PermanentError extends Error`) instead of retrying everything N times.

## 3. Broker Mechanics

**RabbitMQ** — a queue declares a **dead-letter exchange** (`x-dead-letter-exchange`). A message is dead-lettered when it's rejected/nacked with `requeue=false`, its TTL expires, or the queue's length limit is hit. The broker adds an `x-death` header recording the reason, original queue and count (`02-rabbitmq/` covers the setup). Delayed retries are built from TTL'd "wait" queues that dead-letter *back* to the main exchange.

**SQS** — a **redrive policy** on the source queue: after `maxReceiveCount` receives without deletion, SQS moves the message to the DLQ automatically. SQS also offers **DLQ redrive** to move messages back to the source after a fix.

**Kafka** — no built-in DLQ; consumers implement it: on permanent failure (or after N retries), **produce** the record to `<topic>.DLT` with headers (original topic, partition, offset, exception class, message, stack, attempt count), then commit the original offset. Spring Kafka, Kafka Connect (`errors.deadletterqueue.topic.name`) and many frameworks provide this.

## 4. Retry Topology

```
main ──fail──► retry-1 (wait 10 s) ──fail──► retry-2 (wait 1 min) ──fail──► retry-3 (wait 10 min) ──fail──► DLQ
  ▲                │                               │                               │
  └────── success after wait, re-consumed ─────────┴───────────────────────────────┘
```

Separate retry stages keep the main queue/partition flowing while failed messages wait. Remember the ordering trade-off: a message that went to retry is now *behind* later messages for the same key.

## 5. What to Store With a Dead Letter

A DLQ entry is only useful if someone can act on it:

- the original payload, **unchanged**
- original topic/queue, partition, offset / message id
- error class, message and stack trace; the consumer version/commit hash
- attempt count and timestamps (first failure, last failure)
- correlation / trace id to find the related logs

## 6. Operating a DLQ

- **Alert** on DLQ depth > 0 (or a rate) — a DLQ nobody watches is a silent data-loss mechanism.
- **Triage**: is it a bug (fix and redrive), bad data (correct and redrive, or discard with a record of why), or a dependency outage (redrive later)?
- **Redrive** must be idempotent-safe: handlers must tolerate the message arriving late and possibly after newer events (`08-…/` §5 version checks).
- **Retention**: DLQ messages must outlive your triage time (days, not hours).

## Under the Hood: How RabbitMQ Dead-Lettering Rewrites a Message

When a message is dead-lettered, RabbitMQ republishes it internally to the configured DLX with:

- the **routing key** replaced by `x-dead-letter-routing-key` if set, otherwise the original routing key
- an **`x-death`** header: an array of entries `{ queue, reason (rejected|expired|maxlen|delivery_limit), count, exchange, routing-keys, time }`; the count increments if it dies again in the same queue for the same reason
- `x-first-death-queue/reason/exchange` headers that never change after the first death

**Quorum queues** add a `x-delivery-limit` (per-message redelivery counter maintained by the queue itself): exceeding it dead-letters the message with reason `delivery_limit` — a broker-enforced poison-message guard that doesn't depend on consumer code. Note that dead-lettering in classic queues is **at-most-once** by default (the source queue drops the message once it hands it to the DLX); quorum queues support **at-least-once dead-lettering** (`dead-letter-strategy=at-least-once`) which keeps the message until the DLX target confirms.

![Messages that fail transiently go through delayed retry stages and back to the main consumer, while permanent failures and exhausted retries go to a dead-letter queue with error metadata, where they are alerted on, triaged and redriven](diagrams/01-retry-and-dlq.svg)

## Common Pitfalls

1. **Infinite requeue** — `nack(requeue=true)` on a poison message is a hot loop.
2. **Retrying permanent errors** — wastes capacity and delays the alert.
3. **DLQ without alerting** — quiet data loss.
4. **Dead-lettering without context** — no error, no offset, no trace id → impossible to triage.
5. **Redrive that isn't idempotent** — reprocessing creates duplicates or overwrites newer state.
6. **One DLQ for everything** — mixing unrelated failures makes triage and redrive dangerous.

## Quick Reference

| Broker | DLQ mechanism |
|---|---|
| RabbitMQ | `x-dead-letter-exchange`, nack(requeue=false), `x-delivery-limit` (quorum) |
| SQS | redrive policy `maxReceiveCount` → DLQ; DLQ redrive back |
| Kafka | app-level: produce to `<topic>.DLT` with error headers, then commit |
| All | classify errors · bounded retries with backoff · alert on depth · idempotent redrive |