Dead Letter Queues

1The Poison Message Problem

Some messages will never succeed: malformed JSON, a schema the consumer doesn't understand, a reference to a deleted customer, a bug triggered by one specific payload. Under at-least-once delivery, such a poison message is redelivered forever:

• in a queue with requeue-on-failure, it spins in a hot loop, burning CPU and filling logs
• in a Kafka partition, it blocks every message behind it (cumulative commits, 08-acknowledgements_and_message_ordering/ §6)

A dead letter queue (DLQ) is where messages go after they've failed enough times, so the main flow keeps moving and a human (or tool) can inspect them.

2Transient vs Permanent Failures

The first decision on every failure: retry or dead-letter?

Classify errors explicitly in code (class PermanentError extends Error) instead of retrying everything N times.

3Broker Mechanics

RabbitMQ — a queue declares a dead-letter exchange (x-dead-letter-exchange). A message is dead-lettered when it's rejected/nacked with requeue=false, its TTL expires, or the queue's length limit is hit. The broker adds an x-death header recording the reason, original queue and count (02-rabbitmq/ covers the setup). Delayed retries are built from TTL'd "wait" queues that dead-letter back to the main exchange.

SQS — a redrive policy on the source queue: after maxReceiveCount receives without deletion, SQS moves the message to the DLQ automatically. SQS also offers DLQ redrive to move messages back to the source after a fix.

Kafka — no built-in DLQ; consumers implement it: on permanent failure (or after N retries), produce the record to <topic>.DLT with headers (original topic, partition, offset, exception class, message, stack, attempt count), then commit the original offset. Spring Kafka, Kafka Connect (errors.deadletterqueue.topic.name) and many frameworks provide this.

4Retry Topology

main ──fail──► retry-1 (wait 10 s) ──fail──► retry-2 (wait 1 min) ──fail──► retry-3 (wait 10 min) ──fail──► DLQ
  ▲                │                               │                               │
  └────── success after wait, re-consumed ─────────┴───────────────────────────────┘

Separate retry stages keep the main queue/partition flowing while failed messages wait. Remember the ordering trade-off: a message that went to retry is now behind later messages for the same key.

5What to Store With a Dead Letter

A DLQ entry is only useful if someone can act on it:

• the original payload, unchanged
• original topic/queue, partition, offset / message id
• error class, message and stack trace; the consumer version/commit hash
• attempt count and timestamps (first failure, last failure)
• correlation / trace id to find the related logs

{
  "payload": "{\"orderId\":\"o-17\",\"addr\":null}",
  "source": { "topic": "orders", "partition": 3, "offset": 88123 },
  "error": { "class": "ValidationError", "message": "addr is required" },
  "consumer": "shipping@3f9a2c1",
  "attempts": 4,
  "firstFailure": "2026-10-20T09:12:03Z",
  "lastFailure": "2026-10-20T09:24:41Z",
  "traceId": "4bf92f35…"
}

6Operating a DLQ

• Alert on DLQ depth > 0 (or a rate) — a DLQ nobody watches is a silent data-loss mechanism.
• Triage: is it a bug (fix and redrive), bad data (correct and redrive, or discard with a record of why), or a dependency outage (redrive later)?
• Redrive must be idempotent-safe: handlers must tolerate the message arriving late and possibly after newer events (08-…/ §5 version checks).
• Retention: DLQ messages must outlive your triage time (days, not hours).

Under the HoodHow RabbitMQ Dead-Lettering Rewrites a Message▾

When a message is dead-lettered, RabbitMQ republishes it internally to the configured DLX with:

• the routing key replaced by x-dead-letter-routing-key if set, otherwise the original routing key
• an x-death header: an array of entries { queue, reason (rejected|expired|maxlen|delivery_limit), count, exchange, routing-keys, time }; the count increments if it dies again in the same queue for the same reason
• x-first-death-queue/reason/exchange headers that never change after the first death

Quorum queues add a x-delivery-limit (per-message redelivery counter maintained by the queue itself): exceeding it dead-letters the message with reason delivery_limit — a broker-enforced poison-message guard that doesn't depend on consumer code. Note that dead-lettering in classic queues is at-most-once by default (the source queue drops the message once it hands it to the DLX); quorum queues support at-least-once dead-lettering (dead-letter-strategy=at-least-once) which keeps the message until the DLX target confirms.

⚠️ Common Pitfalls

1. Infinite requeue — nack(requeue=true) on a poison message is a hot loop.
2. Retrying permanent errors — wastes capacity and delays the alert.
3. DLQ without alerting — quiet data loss.
4. Dead-lettering without context — no error, no offset, no trace id → impossible to triage.
5. Redrive that isn't idempotent — reprocessing creates duplicates or overwrites newer state.
6. One DLQ for everything — mixing unrelated failures makes triage and redrive dangerous.

📋 Quick Reference

🎤 Interview Q&A

🛠️ Practice exercises

Each exercise is a file in this topic folder. Code exercises are self-checking: run them and make the checks pass.

✅ Check yourself

Source: broker_dead_letter_queues_theory.md · generated by tools/build_lessons.py