# Kafka — Part 1: Topics, Partitions, and the Log Model

## 1. What Kafka Is

Kafka is a distributed **event streaming platform**, built around a fundamentally different storage model than RabbitMQ's (`02-rabbitmq/`). Where RabbitMQ is "smart broker, dumb consumer" — the broker decides which queue a message lands in via exchanges/bindings, and deletes it once acked — Kafka is closer to "dumb broker, smart consumer": the broker just appends every message to a durable, ordered, **append-only log** and keeps it around for a configured retention period, regardless of whether anyone has read it yet. Consumers are responsible for tracking their own position in that log. A full side-by-side comparison is the dedicated `04-rabbitmq_vs_kafka/` topic — this pair of files is about how Kafka itself actually works.

## 2. Topics and Partitions

A **topic** is a named stream of events (conceptually similar to a RabbitMQ queue's role as "where messages live," but the internal shape is different). Every topic is split into one or more **partitions** — independent, ordered, append-only logs that together make up the topic.

```
Topic "orders" (3 partitions):

  Partition 0: [msg0][msg1][msg2][msg3] ...
  Partition 1: [msg0][msg1][msg2] ...
  Partition 2: [msg0][msg1][msg2][msg3][msg4] ...
```

Each message within a single partition gets a strictly increasing **offset** (0, 1, 2, ...) — its position in that partition's log. Ordering is only guaranteed **within a partition**, never across partitions of the same topic. This is the central tradeoff of Kafka's design: splitting a topic into partitions is what lets Kafka parallelize both writes and reads across many brokers/consumers, at the cost of giving up a single global ordering for the topic as a whole.

## 3. Choosing a Partition: the Message Key

When a producer publishes a message, it can optionally attach a **key**. Kafka hashes that key to deterministically pick a partition — the same key always lands in the same partition, every time. This is the mechanism for getting ordering guarantees where they matter: publishing every event for `order-id = 42` with key `"42"` guarantees all of order 42's events land in the same partition, in the order they were sent, even though the topic as a whole has no global order.

```
publish(topic="orders", key="order-42", value={...})  -> hash("order-42") % numPartitions -> partition 1
publish(topic="orders", key="order-42", value={...})  -> same key -> same partition 1, next offset
publish(topic="orders", key="order-17", value={...})  -> different key -> likely a different partition
```

A message published with **no key** is distributed round-robin (or via a sticky-partitioning strategy in modern clients) across partitions purely for load-balancing, with no ordering guarantee relative to other keyless messages.

![A producer hashes each message key to one of three partitions; each partition is an append-only log of offsets that consumers read without deleting](diagrams/01-partitioned-log.svg)

## 4. The Log Is Retained, Not Deleted on Read

This is the single biggest mental shift coming from a queue-based broker: **reading a message from Kafka does not remove it.** A consumer's read is just "fetch bytes starting at offset N" — a pure read operation with no side effect on the log itself. The message stays in the partition until Kafka's retention policy (a time window, e.g. 7 days, or a size cap, whichever is configured) expires it.

This means:
- **Multiple independent consumers (or consumer groups — `07-consumer_groups_partitions_offsets/`) can read the same topic at completely different positions**, with zero coordination or interference — one might be caught up to the latest message, another replaying from an hour ago.
- **Replay is a first-class capability.** A consumer can rewind its offset and reprocess history — rebuild a cache, backfill a new service that just came online, or recover from a bug in a downstream consumer — without the producer doing anything differently.
- Kafka is not "gone once read," so a slow or offline consumer doesn't cause messages to pile up *in the same sense* a RabbitMQ queue does — the log just keeps growing until retention trims the oldest end, independent of consumer progress.

## 5. Brokers, Producers, and Consumers — the Basic Shape

```
Producer --publish(topic, key?, value)--> [ Kafka Broker(s) ] <--fetch(topic, partition, offset)-- Consumer
```

A Kafka **cluster** is made of multiple **broker** processes; each partition of a topic is hosted on (and replicated across, see Part 2) a subset of those brokers. Producers publish to whichever broker currently leads the target partition; consumers fetch from that same leader. Unlike RabbitMQ's single logical queue abstraction, a Kafka client library has to be partition-aware — the producer knows which partition a keyed message goes to, and a consumer explicitly tracks offsets per partition it's assigned.

Part 2 continues with partition assignment across a consumer group, offset commit mechanics, and replication.