Consumer Groups, Partitions and Offsets
Lag, duplicates, idle consumers and rebalance storms all come from three ideas: partitions, groups and offsets. Experiment with consumers joining and leaving a group, commit offsets, crash before committing, and size partition counts properly.
🎯 What you'll learn
- Partitions as the unit of parallelism, one consumer per partition per group
- Committed offsets, log end offset and consumer lag
- Commit strategies and auto.offset.reset
- Rebalancing (eager, cooperative, static) and choosing partition counts
📚 Before you start
03-kafka/ introduced these three ideas; this topic is the full mechanics, because almost every Kafka production issue — lag, duplicates, stalled consumers, "rebalance storms" — comes down to how they interact.
1Partitions Are the Unit of Parallelism
A topic is split into partitions; each partition is an ordered, append-only log. Within a consumer group, each partition is assigned to exactly one consumer at a time:
topic "orders" (6 partitions) group "billing" (3 consumers)
P0 P1 ─────────────────────────────► consumer A
P2 P3 ─────────────────────────────► consumer B
P4 P5 ─────────────────────────────► consumer CConsequences:
- Max useful consumers per group = number of partitions. A 7th consumer on a 6-partition topic sits idle.
- Ordering is per partition, and since one consumer owns a partition, that consumer sees its keys in order.
- Different groups are independent: group "billing" and group "analytics" each get every message, each with their own progress.
2Offsets: Where Each Group Is
Each record in a partition has an offset (0, 1, 2, …). A group's progress is stored as the committed offset per partition in the internal __consumer_offsets topic — and it's the offset of the next record to read, not the last one processed.
partition P0: [0][1][2][3][4][5][6][7][8]
▲ ▲
committed = 4 ┘ └ log end offset = 9
lag = 9 − 4 = 5Consumer lag (log end offset − committed offset, summed over partitions) is the single most important consumer health metric: steadily growing lag means consumers can't keep up.
3Committing
| Strategy | How | Risk |
|---|---|---|
auto-commit (enable.auto.commit) | commits the last polled position every N ms | may commit messages not yet processed (loss on crash) or reprocess after a crash (dupes) |
| commit after processing (sync) | commitSync() after a batch | at-least-once; slower |
| async commit | commitAsync() | fast; an older commit can land after a newer one — use sync on shutdown/rebalance |
| store offsets with the results | write output + offset in one DB transaction | exactly-once effect without Kafka transactions |
Where a new group starts is controlled by auto.offset.reset: earliest (replay everything retained) or latest (only new messages).
4Rebalancing
When a consumer joins, leaves, crashes, or misses its heartbeat/poll deadline, the group rebalances: partitions are reassigned among the live members.
- Eager rebalance (classic): every consumer revokes all partitions, then gets a new assignment — the whole group pauses.
- Cooperative/incremental rebalance (
CooperativeStickyAssignor): only partitions that actually move are revoked; everyone else keeps consuming. - Static membership (
group.instance.id): a consumer that restarts quickly (a rolling deploy) gets its old partitions back without triggering a rebalance.
Before giving up a partition, a consumer should commit what it has processed (the onPartitionsRevoked callback); otherwise the new owner starts from an older commit and reprocesses.
Long processing → no poll() within max.poll.interval.ms → the member is evicted → rebalance → another consumer re-reads from the last commit → duplicates → more slow processing. Process in smaller batches, raise the interval, or hand work to a pool while continuing to poll.
5Choosing the Partition Count
- Enough for peak parallelism: if one consumer handles 2,000 msg/s and peak is 50,000 msg/s, you need ≥ 25 partitions (plus headroom).
- Increasing partitions later changes key → partition mapping (
hash(key) % N), which breaks per-key ordering across the change — so over-provision moderately up front. - Too many partitions cost broker memory, file handles, and leader-election time.
Under the HoodThe Group Coordinator Protocol▾
One broker acts as the group coordinator for each group (chosen by hashing the group id onto a __consumer_offsets partition). Membership works like this:
- Each consumer sends JoinGroup. The coordinator waits for members, then picks one as the group leader.
- The leader (a client, not the broker) runs the configured assignor (range, round-robin, sticky, cooperative-sticky) over all members' subscriptions and sends the result in SyncGroup; the coordinator distributes each member's share.
- Members send heartbeats (from a background thread). If no heartbeat arrives within
session.timeout.ms, or a member doesn't callpoll()withinmax.poll.interval.ms, it's evicted and a new rebalance starts — this is how a stuck processing loop turns into a "rebalance storm". - Each assignment carries a generation id. Commits from an older generation are rejected, so a zombie consumer that was evicted can't overwrite the new owner's progress.
Offset commits are just records written to __consumer_offsets keyed by (group, topic, partition); that topic is log-compacted, so only the latest committed offset per key is retained.
⚠️ Common Pitfalls
- More consumers than partitions — idle consumers, no extra throughput.
- Long processing between polls —
max.poll.interval.msexceeded → eviction → rebalance → duplicates. - Committing before processing (or careless auto-commit) → message loss.
- Not committing on revoke → reprocessing after every rebalance.
- Adding partitions to a keyed topic without planning — per-key ordering breaks during the transition.
- Monitoring throughput but not lag — lag is what tells you you're falling behind.
📋 Quick Reference
| Concept | Rule |
|---|---|
| partition ↔ consumer | one consumer per partition per group |
| parallelism cap | #partitions |
| committed offset | next offset to read |
| lag | log end − committed |
| new group starts at | auto.offset.reset = earliest / latest |
| safer rebalances | cooperative-sticky + static membership |
| zombie protection | generation id on commits |
🧠 Key takeaways
- Within a group, each partition has exactly one consumer; parallelism is capped by the partition count.
- The committed offset is the next one to read; lag = log end − committed, the key health metric.
- Commit after processing; commit on revoke;
auto.offset.resetdecides where new groups start. - Prefer cooperative/static rebalancing; over-provision partitions moderately, since changing them later remaps keys.
🎤 Interview Q&A
Q1How do Kafka consumer groups work?core▾
Consumers sharing a group.id split the topic's partitions among themselves: each partition is assigned to exactly one consumer in the group. Different groups consume independently, each reading all partitions.
- What if there are more consumers than partitions? The extra consumers sit idle.
Q2What is a rebalance and why can it hurt?senior▾
When consumers join, leave or crash, partitions are reassigned. With the eager protocol all consumers stop during the rebalance; long processing that exceeds max.poll.interval.ms can trigger repeated rebalances. Cooperative (incremental) rebalancing and static membership reduce the disruption.
- How do you avoid accidental rebalances? Keep poll loops fast, tune
max.poll.records, and process heavy work off the poll thread carefully.
Q3When should offsets be committed?core▾
After processing a record (or batch) successfully, giving at-least-once. Committing before processing gives at-most-once. Auto-commit commits periodically regardless of processing state, which can lose or duplicate records on crashes.
- Where are offsets stored? In the internal
__consumer_offsetstopic.
Q4How do you choose the number of partitions?core▾
Based on target throughput (partitions × per-consumer throughput) and maximum consumer parallelism, plus headroom, since increasing partitions later changes key-to-partition mapping and breaks per-key ordering for existing keys.
- Downside of too many partitions? More open files, longer leader elections and rebalances.
Q5How do you reprocess data from the past?core▾
Reset the group's offsets (to earliest, a timestamp, or a specific offset) with kafka-consumer-groups --reset-offsets, or start a new consumer group. Retention must still hold that data, and consumers must handle reprocessing idempotently.
- Why is this valuable? Fixing bugs by replaying events, or building a new read model from history.
🛠️ Practice exercises
Each exercise is a file in this topic folder. Code exercises are self-checking: run them and make the checks pass.
✅ Check yourself
Answer the questions — feedback is instant.