RabbitMQ

1What RabbitMQ Is

RabbitMQ is a message broker implementing (primarily) the AMQP 0-9-1 protocol (Advanced Message Queuing Protocol). It is the classic "smart broker, dumb consumer" design: RabbitMQ itself decides exactly which queue(s) a message ends up in based on routing rules you configure, and a consumer just connects to a queue and receives whatever arrives. This is a fundamentally different philosophy from Kafka (covered in the next topic), where the broker is closer to "dumb storage" and consumers do the work of deciding what to read.

2The Three AMQP Building Blocks

RabbitMQ's model has three core pieces, and understanding how they connect is the entire key to using RabbitMQ correctly.

Producer --publish--> [ Exchange ] --(binding rules)--> [ Queue A ] --> Consumer 1
                                    --(binding rules)--> [ Queue B ] --> Consumer 2

A message published to an exchange with no matching binding is, by default, simply dropped (unless the publisher opts into "mandatory" delivery and handles the returned message).

3Exchange Types

The exchange type determines the matching algorithm used to decide which bound queues receive a given message.

Topic exchange example — routing key "orders.eu.created" published:

  binding "orders.*.created"   -> MATCH (one wildcard word: "eu")
  binding "orders.#"           -> MATCH (# covers any remaining words)
  binding "orders.us.created"  -> NO MATCH (exact segment differs)
  binding "orders.eu.cancelled"-> NO MATCH (last segment differs)

4A Concrete Routing Walkthrough

Imagine an order-processing system publishing events to a topic exchange named orders, with routing keys shaped like orders.<region>.<event>.

Bindings:
  queue "eu-orders"        <- binding key "orders.eu.#"
  queue "all-created"      <- binding key "orders.*.created"
  queue "audit-log"        <- binding key "orders.#"

A message published with routing key orders.eu.created matches all three bindings — it lands in eu-orders, all-created, AND audit-log. A message with routing key orders.us.cancelled matches only audit-log (orders.# covers everything).

This is the mechanism that gives RabbitMQ its flexibility: one producer, one exchange, and any number of independently-evolving consumer teams each defining their own queue and binding pattern without touching the producer or each other.

5Queues: Durable, Exclusive, and Auto-Delete

Queues have properties set at creation time that control their lifecycle:

• Durable — the queue definition survives a broker restart (messages inside still need to be individually marked persistent to survive too — durability of the queue and persistence of its messages are separate settings).
• Exclusive — only the connection that created the queue can use it; deleted automatically when that connection closes. Common for temporary, per-client reply queues.
• Auto-delete — the queue is deleted once its last consumer disconnects.

A typical durable work queue for background jobs is created durable, non-exclusive, non-auto-delete, so it survives broker restarts and keeps accepting messages even with zero consumers connected.

6Basic Consumption and Acknowledgement

A consumer subscribes to a queue and receives messages one at a time (or in a prefetched batch — see Part 2). After successfully processing a message, it sends an ack back to RabbitMQ, which then permanently removes that message from the queue. If the consumer disconnects (or explicitly nacks) before acking, RabbitMQ redelivers the message — to the same consumer on reconnect, or to another consumer in the pool.

Consumer receives message  -> processes it  -> ack()  -> RabbitMQ deletes it from the queue
Consumer crashes before ack -> RabbitMQ detects the dead connection -> requeues the message
                                                                        for another consumer

This ack-then-delete model is the core reason RabbitMQ queues behave like a true work queue: once a message is acked, it is gone, full stop — there is no "replay from the beginning" the way there is with Kafka's log.

1Prefetch (QoS): Controlling How Much a Consumer Can Hold

By default, RabbitMQ pushes messages to a connected consumer as fast as it can, without waiting for previous messages to be acked. If one consumer is slow and others are idle, the slow one can end up holding a huge backlog of unacked messages while fast consumers starve.

Prefetch count (set via basic.qos) caps how many unacknowledged messages the broker will have "in flight" to a given consumer at once. prefetch = 1 means "don't send me a new message until I've acked the one I'm holding" — the simplest way to get even load distribution across a pool of consumers.

prefetch = 10:  broker can push up to 10 unacked messages to this consumer
                 before it must wait for an ack before sending an 11th

prefetch = 1:   strict "one at a time" — best load balancing, but more
                network round-trips (lower throughput per consumer)

Tuning prefetch is a direct throughput-vs-fairness trade-off: higher prefetch means fewer round trips and higher throughput per consumer, but a slow consumer can hoard a large batch while others sit idle.

prefetch = 50 → each consumer is handed 50 at once; A finishes in 100 s while B idles after 50 s
prefetch = 1  → messages go to whoever is free; both finish in ≈ 67 s

2Message and Queue Durability

Two independent settings determine whether a message survives a broker restart:

1. The queue must be declared durable — its existence/metadata is written to disk.
2. The message must be published with the persistent delivery mode — its body is written to disk, not just held in memory.

Both are required. A persistent message published to a non-durable queue is lost on restart because the queue itself disappears; a non-persistent message in a durable queue is also lost because the message body was never written to disk.

Durability has a real performance cost — writing to disk on every publish is slower than an in-memory hand-off — so RabbitMQ is usually configured to persist only what actually needs to survive a crash (e.g. order events), while allowing lower-value, high-volume traffic (e.g. live metrics) to skip persistence.

3Dead Lettering (Setup Mechanics)

A queue can be configured with a dead-letter exchange (DLX) — when a message in that queue is rejected/nacked without requeue, or expires (TTL), or the queue overflows its length limit, RabbitMQ automatically republishes it to the DLX instead of discarding it. The full mechanics and retry-then-redirect pattern are covered in the dedicated Dead Letter Queues topic; the piece worth knowing here is that this is configured as queue arguments at declaration time:

queue.declare("orders", {
  arguments: {
    "x-dead-letter-exchange": "orders.dlx",
    "x-dead-letter-routing-key": "orders.failed"
  }
})

4Clustering and Quorum Queues

A single RabbitMQ node is a single point of failure. Production deployments run a cluster of nodes. Two queue types handle replication differently:

Quorum queues are the modern recommendation for anything where message durability across a node failure matters — they trade a bit of latency (waiting for a quorum of acks) for real fault tolerance, rather than the "hope the mirror caught up" behavior of legacy mirrored classic queues.

⚠️ Common Pitfalls

1. Publishing directly assuming a queue name works like a destination. There is no way to "publish to a queue" in AMQP — you always publish to an exchange (even the unnamed default exchange, which does direct routing by queue name as a convenience, is still technically an exchange).
2. Declaring a durable queue but forgetting persistent: true on the message — the queue survives a restart, empty, because the messages inside were never written to disk.
3. Using prefetch = unlimited (the old default) in a multi-consumer pool — one consumer can end up starving the others by hoarding every in-flight message.
4. Assuming a mirrored classic queue gives the same guarantees as a quorum queue — legacy mirroring can lose unsynced messages on failover; quorum queues use actual majority consensus.
5. Forgetting that a fanout/topic exchange delivering to multiple queues means multiple independent copies — acking in one consumer's queue does not remove the message from a sibling queue that also matched.

📋 Quick Reference

Producer -> Exchange -> (binding match) -> Queue -> Consumer -> ack -> deleted

Exchange types: direct (exact key), fanout (all bound queues), topic (* / # patterns), headers
Durability: queue must be durable AND message must be persistent
Prefetch:   caps unacked messages in flight per consumer (1 = strict round robin)
Quorum queue: Raft-replicated, majority-ack durability, modern HA default
Dead letter: queue arg "x-dead-letter-exchange" reroutes rejected/expired messages

🎤 Interview Q&A

🛠️ Practice exercises

Each exercise is a file in this topic folder. Code exercises are self-checking: run them and make the checks pass.

✅ Check yourself

Source: broker_rabbitmq_theory_1.md, broker_rabbitmq_theory_2.md · generated by tools/build_lessons.py