# RabbitMQ vs Kafka

`02-rabbitmq/` and `03-kafka/` covered each broker on its own terms. This topic puts them side by side, because "which broker?" is a design decision that shapes everything downstream: how consumers scale, whether messages can be replayed, and what ordering you get.

## 1. The One-Line Difference

- **RabbitMQ is a smart broker with a queue model**: the broker routes each message (exchanges + bindings) into queues, tracks per-message delivery and acknowledgement, and **deletes a message once it's acked**. Consumers are relatively simple.
- **Kafka is a distributed append-only log**: producers append to partitions, the broker keeps everything for a retention period, and **consumers track their own position (offset)**. The broker is relatively simple; consumers do more.

```
RabbitMQ:  producer → exchange ──binding──► queue ──push──► consumer → ack → message deleted
Kafka:     producer → topic/partition (append) ◄──pull at offset N── consumer (commits offset)
                                         messages stay until retention expires
```

## 2. Side by Side

| Dimension | RabbitMQ | Kafka |
|---|---|---|
| Core model | queues; message consumed = gone | partitioned log; messages retained |
| Delivery | broker **pushes** (with prefetch limit) | consumer **pulls** batches |
| Routing | rich: direct / topic / fanout / headers exchanges | by topic + partition key only |
| Replay old messages | no (once acked, deleted) | yes — reset the offset |
| Ordering | per queue (breaks with competing consumers + requeue) | per **partition** |
| Scaling consumers | add consumers to a queue (competing consumers) | up to **#partitions** consumers per group |
| Multiple independent readers | one queue per reader (fanout copies the message) | many consumer groups read the same log |
| Per-message features | TTL, priority, per-message ack/nack, DLX | none per message; batch-oriented |
| Throughput | tens of thousands/s per queue | millions/s per cluster (sequential I/O, batching) |
| Latency | very low (sub-ms) at moderate load | low ms; tuned for throughput (batching/linger) |
| Typical use | task queues, RPC, complex routing, work distribution | event streaming, event sourcing, CDC, analytics pipelines, logs |

## 3. When RabbitMQ Fits Better

- **Work queues / background jobs**: each task done once by one worker, per-message ack, retries, DLX (`10-dead_letter_queues/`), priorities.
- **Complex routing**: route `order.eu.created` to the EU billing queue and `*.*.created` to audit, without producers knowing consumers.
- **Request/reply** (RPC over messaging) with `reply_to` and correlation IDs.
- **Low volume, many short-lived queues** (per-user, per-session).

## 4. When Kafka Fits Better

- **Event streams many teams consume independently**: analytics, search indexing, fraud detection each run their own consumer group at their own pace, on the same events.
- **Replay**: rebuild a read model, backfill a new service, or reprocess after a bug fix by rewinding offsets.
- **Very high throughput** with ordering per entity (key = `orderId` keeps one order's events in order).
- **Event sourcing / CDC**: the log *is* the source of truth or the change history (log compaction keeps the latest value per key).

## 5. Things That Look Similar but Aren't

- **"Topic"**: in RabbitMQ a topic *exchange* is a routing rule; in Kafka a topic is a durable, partitioned log.
- **"Consumer group"**: RabbitMQ has no such concept — competing consumers simply share a queue. Kafka's groups give each group its own full copy of the stream (`07-consumer_groups_partitions_offsets/`).
- **Ordering**: both offer ordering only within a unit (a queue / a partition), and both lose it if you parallelize carelessly (`08-acknowledgements_and_message_ordering/`).
- **Streams**: RabbitMQ 3.9+ has **Streams** (an append-only, replayable log type) and Kafka has **share groups** in newer versions (queue-like consumption) — the two products are converging at the edges, but their defaults and strengths still differ.

## Under the Hood: Why Their Performance Profiles Differ

RabbitMQ keeps per-message state: which consumer has it, whether it's acked, redelivery flags. Each ack is a broker-side operation, and a queue is (by default) a single Erlang process, so one queue's throughput is bounded by one core; you scale by having **more queues**. Messages that aren't consumed quickly are paged from memory to disk, which slows the broker when queues grow long — RabbitMQ is happiest when queues stay short.

Kafka keeps almost no per-message state. A partition is a set of segment files that are only ever **appended** to; consumer progress is a single integer per partition per group. Producers batch and compress many records into one request; brokers write them with sequential I/O and serve reads from the OS **page cache** using `sendfile` (zero-copy from disk cache to socket). Long backlogs are normal and cheap — a lagging consumer is just reading an older offset. The price is coarser semantics: no per-message ack, no per-message routing or priority, and parallelism capped by partition count.

![RabbitMQ routes each message through an exchange into queues and deletes it when acked, while Kafka appends to partitioned logs that retain messages and lets each consumer group track its own offset](diagrams/01-queue-vs-log.svg)

## Common Pitfalls

1. **Using Kafka as a job queue** — no per-message ack/retry/DLQ; one slow message blocks its whole partition.
2. **Using RabbitMQ for replayable event history** — acked messages are gone; you can't rebuild a read model from it.
3. **Too few Kafka partitions** — consumer parallelism is capped by partition count, and increasing it later reshuffles key → partition mapping.
4. **Long RabbitMQ queues as a buffer for days of data** — memory pressure, paging, flow control.
5. **Choosing on hype rather than on the consumption model** — ask: does each message need to be processed once by one worker (queue), or read by many independent consumers over time (log)?

## Quick Reference

| If you need… | Choose |
|---|---|
| task distribution, per-message retry/DLQ, priorities | RabbitMQ |
| flexible routing by pattern | RabbitMQ |
| replay, many independent consumers, event sourcing | Kafka |
| millions of events/s with per-key ordering | Kafka |
| both | it's common to run both, each for what it's good at |