RabbitMQ vs Kafka

02-rabbitmq/ and 03-kafka/ covered each broker on its own terms. This topic puts them side by side, because "which broker?" is a design decision that shapes everything downstream: how consumers scale, whether messages can be replayed, and what ordering you get.

1The One-Line Difference

• RabbitMQ is a smart broker with a queue model: the broker routes each message (exchanges + bindings) into queues, tracks per-message delivery and acknowledgement, and deletes a message once it's acked. Consumers are relatively simple.
• Kafka is a distributed append-only log: producers append to partitions, the broker keeps everything for a retention period, and consumers track their own position (offset). The broker is relatively simple; consumers do more.

RabbitMQ:  producer → exchange ──binding──► queue ──push──► consumer → ack → message deleted
Kafka:     producer → topic/partition (append) ◄──pull at offset N── consumer (commits offset)
                                         messages stay until retention expires

2Side by Side

3When RabbitMQ Fits Better

• Work queues / background jobs: each task done once by one worker, per-message ack, retries, DLX (10-dead_letter_queues/), priorities.
• Complex routing: route order.eu.created to the EU billing queue and *.*.created to audit, without producers knowing consumers.
• Request/reply (RPC over messaging) with reply_to and correlation IDs.
• Low volume, many short-lived queues (per-user, per-session).

4When Kafka Fits Better

• Event streams many teams consume independently: analytics, search indexing, fraud detection each run their own consumer group at their own pace, on the same events.
• Replay: rebuild a read model, backfill a new service, or reprocess after a bug fix by rewinding offsets.
• Very high throughput with ordering per entity (key = orderId keeps one order's events in order).
• Event sourcing / CDC: the log is the source of truth or the change history (log compaction keeps the latest value per key).

5Things That Look Similar but Aren't

• "Topic": in RabbitMQ a topic exchange is a routing rule; in Kafka a topic is a durable, partitioned log.
• "Consumer group": RabbitMQ has no such concept — competing consumers simply share a queue. Kafka's groups give each group its own full copy of the stream (07-consumer_groups_partitions_offsets/).
• Ordering: both offer ordering only within a unit (a queue / a partition), and both lose it if you parallelize carelessly (08-acknowledgements_and_message_ordering/).
• Streams: RabbitMQ 3.9+ has Streams (an append-only, replayable log type) and Kafka has share groups in newer versions (queue-like consumption) — the two products are converging at the edges, but their defaults and strengths still differ.

Under the HoodWhy Their Performance Profiles Differ▾

RabbitMQ keeps per-message state: which consumer has it, whether it's acked, redelivery flags. Each ack is a broker-side operation, and a queue is (by default) a single Erlang process, so one queue's throughput is bounded by one core; you scale by having more queues. Messages that aren't consumed quickly are paged from memory to disk, which slows the broker when queues grow long — RabbitMQ is happiest when queues stay short.

Kafka keeps almost no per-message state. A partition is a set of segment files that are only ever appended to; consumer progress is a single integer per partition per group. Producers batch and compress many records into one request; brokers write them with sequential I/O and serve reads from the OS page cache using sendfile (zero-copy from disk cache to socket). Long backlogs are normal and cheap — a lagging consumer is just reading an older offset. The price is coarser semantics: no per-message ack, no per-message routing or priority, and parallelism capped by partition count.

⚠️ Common Pitfalls

1. Using Kafka as a job queue — no per-message ack/retry/DLQ; one slow message blocks its whole partition.
2. Using RabbitMQ for replayable event history — acked messages are gone; you can't rebuild a read model from it.
3. Too few Kafka partitions — consumer parallelism is capped by partition count, and increasing it later reshuffles key → partition mapping.
4. Long RabbitMQ queues as a buffer for days of data — memory pressure, paging, flow control.
5. Choosing on hype rather than on the consumption model — ask: does each message need to be processed once by one worker (queue), or read by many independent consumers over time (log)?

📋 Quick Reference

🎤 Interview Q&A

🛠️ Practice exercises

Each exercise is a file in this topic folder. Code exercises are self-checking: run them and make the checks pass.

✅ Check yourself

Source: broker_rabbitmq_vs_kafka_theory.md · generated by tools/build_lessons.py