# Queues and Topics

## 1. Two Delivery Shapes

Every messaging system offers some combination of two fundamental shapes:

| | **Queue** (point-to-point) | **Topic** (publish/subscribe) |
|---|---|---|
| Each message goes to | **one** consumer | **every** subscriber |
| Adding consumers | splits the work (competing consumers) | adds another independent copy of the stream |
| Mental model | a to-do list shared by workers | a newsletter / radio broadcast |
| Typical use | jobs, commands ("resize this image") | events ("order was created") |

```
QUEUE:                          TOPIC:
             ┌─► worker 1 (m1, m3)              ┌─► email      (m1, m2, m3)
 producer ──►│                    publisher ───►├─► inventory  (m1, m2, m3)
             └─► worker 2 (m2)                  └─► analytics  (m1, m2, m3)
```

## 2. Commands vs Events

The shape usually follows the *meaning* of the message:

- A **command** ("ChargeCard", "SendWelcomeEmail") asks for something to happen **once**, by whoever handles it. → queue.
- An **event** ("OrderPlaced", "UserSignedUp") states a fact; any number of services may care, and the publisher shouldn't know which. → topic.

Naming reflects this: commands are imperative, events are past tense. Mixing them up leads to classic bugs — e.g. publishing `SendEmail` on a topic where two email services both subscribe, so customers get two emails.

## 3. Combining Both: Topic + Queue per Subscriber

Real systems usually need **both**: every interested *service* gets every event (pub/sub), and within a service, multiple *instances* share that service's copy (competing consumers).

- **RabbitMQ**: a fanout/topic exchange (pub/sub) with **one queue per subscribing service**; each service runs N consumers on its own queue.
- **Kafka**: a topic (pub/sub) and **one consumer group per service**; partitions are split among the group's instances (`07-consumer_groups_partitions_offsets/`).
- **AWS**: SNS topic → one SQS queue per subscriber ("fan-out"), see `devops-cloud/aws/07-sqs_and_sns`.

```
                         ┌─► queue "email"     ─► email-1, email-2      (compete)
OrderPlaced ─► exchange ─┼─► queue "inventory" ─► inventory-1           
                         └─► queue "analytics" ─► analytics-1..4        (compete)
```

## 4. Subscription Lifetime: Durable vs Transient

- **Durable subscription**: messages published while the subscriber is offline are kept and delivered later (a durable RabbitMQ queue bound to the exchange; a Kafka consumer group with committed offsets).
- **Transient subscription**: only receives messages while connected (an exclusive, auto-delete queue; Redis Pub/Sub). Fine for live UI updates, wrong for anything that must not be missed.

Redis Pub/Sub is the cautionary example: there's no storage at all — a subscriber that disconnects for one second misses everything published in that second. Redis **Streams** add persistence and consumer groups.

## 5. Topic Hierarchies and Wildcards

Topic names are often hierarchical so subscribers can choose granularity:

| System | Example name | Wildcards |
|---|---|---|
| RabbitMQ topic exchange | `orders.eu.created` | `*` = one word, `#` = zero or more words |
| MQTT | `factory/line1/temp` | `+` = one level, `#` = rest |
| Kafka | `orders.created` (flat) | regex subscription on topic names |
| NATS | `orders.eu.created` | `*` one token, `>` rest |

## Under the Hood: How Fan-Out Is Implemented

There are two ways a broker can give N subscribers the same message:

1. **Copy on publish** (RabbitMQ, SNS→SQS): the exchange evaluates bindings and **enqueues a reference into each matching queue**. RabbitMQ stores the message body once and keeps per-queue references, but each queue tracks delivery and acknowledgement independently, so one slow subscriber's backlog doesn't affect others. Cost grows with the number of subscribers at publish time.
2. **Shared log + per-reader cursors** (Kafka, Redis Streams, Pulsar): the message is written **once** to a log; each subscriber (consumer group) keeps an offset. Fan-out costs nothing at publish time — adding the 20th subscriber is just another cursor. The trade-off: retention is global for the topic, not per subscriber, so a subscriber that falls further behind than the retention window loses data.

Competing consumers on one queue are the mirror image: the broker hands each message to **one** of the connected consumers (round-robin, limited by each consumer's prefetch window), and only redelivers it to another if the first consumer dies or nacks it.

![A queue gives each message to exactly one of several competing workers, a topic gives every subscriber its own copy, and the combined pattern uses a topic with one queue or consumer group per subscribing service](diagrams/01-queue-topic-combined.svg)

## Common Pitfalls

1. **Commands on a topic** — duplicated side effects when two services subscribe.
2. **Events on a single shared queue** — services steal each other's messages; each event reaches only one of them.
3. **Transient subscriptions for data that matters** (Redis Pub/Sub for orders) — anything published during a disconnect is gone.
4. **One queue shared by different services "for simplicity"** — impossible to scale or deploy them independently.
5. **Assuming subscribers receive in the same order relative to each other** — each copy is consumed independently.

## Quick Reference

| Need | Shape |
|---|---|
| do this once | queue (competing consumers) |
| tell everyone this happened | topic (pub/sub) |
| every service gets it, instances share load | topic + one queue / consumer group per service |
| survive subscriber downtime | durable subscription |
| live-only updates | transient subscription is fine |