# Kafka — Part 2: Consumer Groups, Offsets, and Replication

## 1. Consumer Groups (Preview — Full Depth in `07-consumer_groups_partitions_offsets/`)

Consumers subscribe to a topic as part of a **consumer group**, identified by a group id. Kafka guarantees that within one group, **each partition is consumed by exactly one member of that group at a time** — so a 3-partition topic with a 3-consumer group gives each consumer its own partition, processed in parallel. A *different* consumer group subscribing to the same topic gets its own independent full copy of every partition, entirely unaffected by the first group's progress. This is how Kafka gets pub/sub fan-out (many independent groups) and work-queue-style parallelism (many members within one group) out of the same mechanism, simultaneously. The dedicated topic goes deep on assignment strategy and rebalancing; this file only needs the shape.

```
Topic "orders" (3 partitions), Consumer Group "billing":
  partition 0 -> consumer 1
  partition 1 -> consumer 2
  partition 2 -> consumer 3

Consumer Group "analytics" (subscribes independently):
  partition 0 -> consumer X
  partition 1 -> consumer Y
  partition 2 -> consumer Z
```

## 2. Offsets: a Consumer's Bookmark

An **offset** is just an integer position within one partition. Kafka itself doesn't track "has this message been processed" the way RabbitMQ tracks ack/unacked per message — instead, each consumer group **commits** the offset it has successfully processed up to, for each partition it owns, back to Kafka (in an internal topic, `__consumer_offsets`). On restart or rebalance, a consumer resumes from its group's last committed offset for that partition.

```
Partition 1 log:  [0][1][2][3][4][5][6][7] <- latest
                              ▲
                     committed offset = 3
                     (group will resume fetching from offset 4)
```

This offset-commit model, not per-message acking, is why Kafka's throughput profile looks so different from RabbitMQ's — committing an offset is a single, infrequent, batched operation ("I've processed through message 500"), not one round-trip per message.

## 3. Auto-Commit vs. Manual Commit

- **Auto-commit** (the default in most client libraries) periodically commits the latest fetched offset on a timer, regardless of whether processing actually finished — simple, but risks committing an offset for a message that was fetched but then failed to process (that message is silently skipped on restart), or reprocessing a message that finished just before a crash but wasn't yet auto-committed.
- **Manual commit** — the application explicitly calls commit only after it has durably finished processing a batch, giving control over the exact at-least-once boundary. This is the mechanism underlying the delivery-guarantee choices covered in `09-delivery_guarantees/`.

## 4. Replication Factor and Leader/Follower Partitions

Each partition is replicated across multiple brokers for fault tolerance, controlled by the topic's **replication factor** (commonly 3 in production). One replica is elected **leader** — all producer writes and consumer reads for that partition go through the leader — and the others are **followers**, which continuously pull new records from the leader to stay in sync.

```
Partition 1, replication factor 3:

  Broker A: [leader]    <- producers write here, consumers read here
  Broker B: [follower]  <- replicates from leader
  Broker C: [follower]  <- replicates from leader

If Broker A dies, an in-sync follower (B or C) is elected the new leader.
```

A follower that has fully caught up to the leader is called **in-sync** (part of the **ISR**, in-sync replica set). Kafka only considers a write durably safe once it's been acknowledged by enough of the ISR (configurable via `acks` — covered in `09-delivery_guarantees/`), because a write only on the leader with no replica caught up is lost if that leader crashes before any follower copies it.

## Under the Hood: Why the Log Model Scales the Way It Does

Kafka's partition is, physically, a set of append-only segment files on disk plus an index mapping offsets to byte positions — appending a new message is a sequential disk write (fast, even on spinning disks) and reading a range of offsets is a sequential read, which is why Kafka can sustain very high throughput per partition without the random-access I/O pattern that would come from, say, deleting arbitrary acked messages out of the middle of a queue the way RabbitMQ does. Partitioning is also Kafka's actual scaling knob: a single partition can only be written to (and, within a consumer group, read from) by one broker leader and one consumer at a time, so a topic's maximum parallelism is capped by its partition count — this is why partition count is chosen up front based on expected consumer parallelism, and why repartitioning an existing topic later is disruptive (it changes which partition a given key hashes to, breaking the per-key ordering guarantee for anything already in flight).

## Common Pitfalls

1. **Assuming Kafka gives topic-wide ordering.** Ordering is per-partition only; two messages with different keys (or no key) can be read out of the order they were published if they land in different partitions.
2. **Choosing too few partitions up front.** Partition count is difficult to change later without breaking key-to-partition consistency, and it hard-caps how many consumers in a group can work in parallel.
3. **Relying on default auto-commit for exactly-once-feeling behavior.** Auto-commit is a timer, not tied to actual processing completion — it can silently skip or reprocess messages around a crash.
4. **Forgetting replication factor 1 means zero fault tolerance.** A single-replica partition has no follower to fail over to — losing that one broker loses the partition's data outright.
5. **Treating "the message was read" as "the message was processed."** Reading from a partition has no effect on the log; only a committed offset reflects processing progress, and a consumer can crash between the two.

## Quick Reference

```
Topic -> split into Partitions -> each an ordered, append-only log
Offset      -> a message's position within ONE partition
Key         -> hashed to deterministically pick a partition (same key -> same partition -> ordering)
No key      -> round-robin/sticky across partitions, no ordering guarantee

Read does NOT delete -> retention (time/size) controls when data is trimmed
Consumer group -> each partition consumed by exactly ONE member of the group
Different groups -> fully independent read progress over the same topic

Offset commit -> the durability checkpoint (not per-message ack)
Replication factor -> N copies per partition; leader handles reads/writes, followers replicate
ISR (in-sync replicas) -> followers caught up enough to be eligible for leader election
```