Kafka
Kafka is a distributed append-only log: messages are retained, partitions give ordering and parallelism, and consumers track their own offsets. Produce keyed messages into a live partitioned log, add consumers to a group, and watch a leader failover.
🎯 What you'll learn
- Topics, partitions, offsets and the retained log
- Message keys, partitioning and per-key ordering
- Consumer groups and offset commits (auto vs manual)
- Replication, leaders, followers and the ISR
📚 Before you start
1What Kafka Is
Kafka is a distributed event streaming platform, built around a fundamentally different storage model than RabbitMQ's (02-rabbitmq/). Where RabbitMQ is "smart broker, dumb consumer" — the broker decides which queue a message lands in via exchanges/bindings, and deletes it once acked — Kafka is closer to "dumb broker, smart consumer": the broker just appends every message to a durable, ordered, append-only log and keeps it around for a configured retention period, regardless of whether anyone has read it yet. Consumers are responsible for tracking their own position in that log. A full side-by-side comparison is the dedicated 04-rabbitmq_vs_kafka/ topic — this pair of files is about how Kafka itself actually works.
2Topics and Partitions
A topic is a named stream of events (conceptually similar to a RabbitMQ queue's role as "where messages live," but the internal shape is different). Every topic is split into one or more partitions — independent, ordered, append-only logs that together make up the topic.
Topic "orders" (3 partitions):
Partition 0: [msg0][msg1][msg2][msg3] ...
Partition 1: [msg0][msg1][msg2] ...
Partition 2: [msg0][msg1][msg2][msg3][msg4] ...Each message within a single partition gets a strictly increasing offset (0, 1, 2, ...) — its position in that partition's log. Ordering is only guaranteed within a partition, never across partitions of the same topic. This is the central tradeoff of Kafka's design: splitting a topic into partitions is what lets Kafka parallelize both writes and reads across many brokers/consumers, at the cost of giving up a single global ordering for the topic as a whole.
3Choosing a Partition: the Message Key
When a producer publishes a message, it can optionally attach a key. Kafka hashes that key to deterministically pick a partition — the same key always lands in the same partition, every time. This is the mechanism for getting ordering guarantees where they matter: publishing every event for order-id = 42 with key "42" guarantees all of order 42's events land in the same partition, in the order they were sent, even though the topic as a whole has no global order.
publish(topic="orders", key="order-42", value={...}) -> hash("order-42") % numPartitions -> partition 1
publish(topic="orders", key="order-42", value={...}) -> same key -> same partition 1, next offset
publish(topic="orders", key="order-17", value={...}) -> different key -> likely a different partitionA message published with no key is distributed round-robin (or via a sticky-partitioning strategy in modern clients) across partitions purely for load-balancing, with no ordering guarantee relative to other keyless messages.
4The Log Is Retained, Not Deleted on Read
This is the single biggest mental shift coming from a queue-based broker: reading a message from Kafka does not remove it. A consumer's read is just "fetch bytes starting at offset N" — a pure read operation with no side effect on the log itself. The message stays in the partition until Kafka's retention policy (a time window, e.g. 7 days, or a size cap, whichever is configured) expires it.
This means:
- Multiple independent consumers (or consumer groups —
07-consumer_groups_partitions_offsets/) can read the same topic at completely different positions, with zero coordination or interference — one might be caught up to the latest message, another replaying from an hour ago. - Replay is a first-class capability. A consumer can rewind its offset and reprocess history — rebuild a cache, backfill a new service that just came online, or recover from a bug in a downstream consumer — without the producer doing anything differently.
- Kafka is not "gone once read," so a slow or offline consumer doesn't cause messages to pile up in the same sense a RabbitMQ queue does — the log just keeps growing until retention trims the oldest end, independent of consumer progress.
Because reading doesn't delete, a new service can start a consumer group at earliest and backfill months of events, or a bug-fixed consumer can rewind its offset and reprocess. That's impossible with a delete-on-ack queue.
5Brokers, Producers, and Consumers — the Basic Shape
Producer --publish(topic, key?, value)--> [ Kafka Broker(s) ] <--fetch(topic, partition, offset)-- ConsumerA Kafka cluster is made of multiple broker processes; each partition of a topic is hosted on (and replicated across, see Part 2) a subset of those brokers. Producers publish to whichever broker currently leads the target partition; consumers fetch from that same leader. Unlike RabbitMQ's single logical queue abstraction, a Kafka client library has to be partition-aware — the producer knows which partition a keyed message goes to, and a consumer explicitly tracks offsets per partition it's assigned.
Part 2 continues with partition assignment across a consumer group, offset commit mechanics, and replication.
1Consumer Groups (Preview — Full Depth in 07-consumer_groups_partitions_offsets/)
Consumers subscribe to a topic as part of a consumer group, identified by a group id. Kafka guarantees that within one group, each partition is consumed by exactly one member of that group at a time — so a 3-partition topic with a 3-consumer group gives each consumer its own partition, processed in parallel. A different consumer group subscribing to the same topic gets its own independent full copy of every partition, entirely unaffected by the first group's progress. This is how Kafka gets pub/sub fan-out (many independent groups) and work-queue-style parallelism (many members within one group) out of the same mechanism, simultaneously. The dedicated topic goes deep on assignment strategy and rebalancing; this file only needs the shape.
Topic "orders" (3 partitions), Consumer Group "billing":
partition 0 -> consumer 1
partition 1 -> consumer 2
partition 2 -> consumer 3
Consumer Group "analytics" (subscribes independently):
partition 0 -> consumer X
partition 1 -> consumer Y
partition 2 -> consumer Z2Offsets: a Consumer's Bookmark
An offset is just an integer position within one partition. Kafka itself doesn't track "has this message been processed" the way RabbitMQ tracks ack/unacked per message — instead, each consumer group commits the offset it has successfully processed up to, for each partition it owns, back to Kafka (in an internal topic, __consumer_offsets). On restart or rebalance, a consumer resumes from its group's last committed offset for that partition.
Partition 1 log: [0][1][2][3][4][5][6][7] <- latest
▲
committed offset = 3
(group will resume fetching from offset 4)This offset-commit model, not per-message acking, is why Kafka's throughput profile looks so different from RabbitMQ's — committing an offset is a single, infrequent, batched operation ("I've processed through message 500"), not one round-trip per message.
3Auto-Commit vs. Manual Commit
- Auto-commit (the default in most client libraries) periodically commits the latest fetched offset on a timer, regardless of whether processing actually finished — simple, but risks committing an offset for a message that was fetched but then failed to process (that message is silently skipped on restart), or reprocessing a message that finished just before a crash but wasn't yet auto-committed.
- Manual commit — the application explicitly calls commit only after it has durably finished processing a batch, giving control over the exact at-least-once boundary. This is the mechanism underlying the delivery-guarantee choices covered in
09-delivery_guarantees/.
4Replication Factor and Leader/Follower Partitions
Each partition is replicated across multiple brokers for fault tolerance, controlled by the topic's replication factor (commonly 3 in production). One replica is elected leader — all producer writes and consumer reads for that partition go through the leader — and the others are followers, which continuously pull new records from the leader to stay in sync.
Partition 1, replication factor 3:
Broker A: [leader] <- producers write here, consumers read here
Broker B: [follower] <- replicates from leader
Broker C: [follower] <- replicates from leader
If Broker A dies, an in-sync follower (B or C) is elected the new leader.A follower that has fully caught up to the leader is called in-sync (part of the ISR, in-sync replica set). Kafka only considers a write durably safe once it's been acknowledged by enough of the ISR (configurable via acks — covered in 09-delivery_guarantees/), because a write only on the leader with no replica caught up is lost if that leader crashes before any follower copies it.
Under the HoodWhy the Log Model Scales the Way It Does▾
Kafka's partition is, physically, a set of append-only segment files on disk plus an index mapping offsets to byte positions — appending a new message is a sequential disk write (fast, even on spinning disks) and reading a range of offsets is a sequential read, which is why Kafka can sustain very high throughput per partition without the random-access I/O pattern that would come from, say, deleting arbitrary acked messages out of the middle of a queue the way RabbitMQ does. Partitioning is also Kafka's actual scaling knob: a single partition can only be written to (and, within a consumer group, read from) by one broker leader and one consumer at a time, so a topic's maximum parallelism is capped by its partition count — this is why partition count is chosen up front based on expected consumer parallelism, and why repartitioning an existing topic later is disruptive (it changes which partition a given key hashes to, breaking the per-key ordering guarantee for anything already in flight).
⚠️ Common Pitfalls
- Assuming Kafka gives topic-wide ordering. Ordering is per-partition only; two messages with different keys (or no key) can be read out of the order they were published if they land in different partitions.
- Choosing too few partitions up front. Partition count is difficult to change later without breaking key-to-partition consistency, and it hard-caps how many consumers in a group can work in parallel.
- Relying on default auto-commit for exactly-once-feeling behavior. Auto-commit is a timer, not tied to actual processing completion — it can silently skip or reprocess messages around a crash.
- Forgetting replication factor 1 means zero fault tolerance. A single-replica partition has no follower to fail over to — losing that one broker loses the partition's data outright.
- Treating "the message was read" as "the message was processed." Reading from a partition has no effect on the log; only a committed offset reflects processing progress, and a consumer can crash between the two.
📋 Quick Reference
Topic -> split into Partitions -> each an ordered, append-only log
Offset -> a message's position within ONE partition
Key -> hashed to deterministically pick a partition (same key -> same partition -> ordering)
No key -> round-robin/sticky across partitions, no ordering guarantee
Read does NOT delete -> retention (time/size) controls when data is trimmed
Consumer group -> each partition consumed by exactly ONE member of the group
Different groups -> fully independent read progress over the same topic
Offset commit -> the durability checkpoint (not per-message ack)
Replication factor -> N copies per partition; leader handles reads/writes, followers replicate
ISR (in-sync replicas) -> followers caught up enough to be eligible for leader election🧠 Key takeaways
- A topic is split into partitions: ordered, append-only logs addressed by offset.
- The key picks the partition (
hash(key) % N): same key → same partition → per-key order. - Reading never deletes: retention trims data, consumers commit offsets, and replay is first-class.
- Partitions are replicated (RF 3); writes go to the leader, and an in-sync follower takes over on failure.
🎤 Interview Q&A
Q1What is Kafka's core abstraction?core▾
A distributed, partitioned, replicated append-only log. Topics are split into partitions; each record gets an offset; consumers track their own offsets and can re-read. Records are retained by time or size, not deleted on consumption.
- Why is that useful? Replay, multiple independent consumers, and event sourcing / stream processing.
Q2How does Kafka achieve high throughput?core▾
Sequential disk writes and reads, the OS page cache, zero-copy transfer (sendfile), batching and compression of records, and parallelism through partitions spread across brokers.
- How do you increase consumer throughput? More partitions and more consumers in the group, up to one consumer per partition.
Q3How does replication work in Kafka?core▾
Each partition has a leader and followers; producers and consumers talk to the leader. Followers that are caught up form the ISR (in-sync replicas). With acks=all and min.insync.replicas=2, a write is acknowledged only when enough replicas have it, so a leader failure doesn't lose data.
- What's unclean leader election? Electing an out-of-sync replica as leader: available but loses data; usually disabled.
Q4How do you get exactly-once semantics in Kafka?senior▾
Idempotent producers (dedupe by producer ID + sequence) avoid duplicates from retries; transactions let a consume-process-produce cycle atomically write output and commit offsets. Consumers read with isolation.level=read_committed. This is exactly-once within Kafka; external side effects still need idempotency.
- Does it cover writing to a database? No; use idempotent writes or store offsets in the same DB transaction.
Q5What is log compaction?core▾
Instead of deleting by age, Kafka keeps at least the latest record per key, removing older values. It turns a topic into a changelog/snapshot of current state, used for things like configuration or KTable state.
- How do you delete a key? Write a tombstone (null value) for it.
🛠️ Practice exercises
Each exercise is a file in this topic folder. Code exercises are self-checking: run them and make the checks pass.
✅ Check yourself
Answer the questions — feedback is instant.