Kafka Architecture Deep Dive
Offsets — Kafka's Position Tracking System
How consumer offsets track reading position across partitions.
Introduction
An Offset is a unique, monotonically increasing 64-bit integer assigned sequentially to each message as it is appended to a partition log. It serves as the primary coordinate for locating and reading records in Kafka.
Why It Matters
In traditional brokers, the server tracks what has been read. In Kafka, consumers track their own offsets. This stateless broker design is why Kafka achieves extreme scale. However, developers must manage how and when offsets are committed; improper commits lead to duplicate processing or silent message loss.
Real-World Analogy
Think of reading a long textbook. - The Log-End Offset (LEO) is the page number of the last page in the printed book. - Your Current Position is the page number your eyes are currently focused on. - Your Committed Offset is the sticky note you place in the book before closing it. If you forget to place the sticky note (crash without commit), you must start reading again from your last bookmark.
How It Works
Kafka tracks three offsets for partition tracking:
- Current Offset: The next message the consumer polls.
- Committed Offset: The last acknowledged offset saved back to the broker's offset store.
- Log-End Offset (LEO): The next offset to be written by a producer to the partition leader log.
Internal Architecture
Offsets are stored in a special, highly compact internal Kafka topic named __consumer_offsets. When a consumer group commits an offset, it appends a message containing the consumer group ID, topic name, partition, and offset. Kafka compacts this topic, retaining only the latest committed offset per partition.
Visual Explanation
Below is an ASCII visualization of partition log offsets, detailing LEO and Committed offsets:
Partition Log File:
| Msg 0 | Msg 1 | Msg 2 | Msg 3 | Msg 4 | Msg 5 | (Next Slot)
^ ^
| |
[Committed Offset: 3] [Log-End Offset (LEO): 6]
Practical Example
Below is a Java consumer snippet demonstrating manual synchronous offset commits rather than relying on auto-commits:
Common Mistakes
- Using Auto-Commit with Asynchronous Jobs: If
enable.auto.commit = true, Kafka commits offsets automatically every 5 seconds. If your consumer delegates work to a background thread and immediately polls again, it may commit offsets before the worker finishes processing, risking silent data loss on crashes. - Committing Offsets in a Loop: Committing offsets after processing every single record (rather than at the end of a batch) causes excessive network calls to
__consumer_offsets, degrading consumer throughput.
Quick Quiz
Q1: If a consumer crashes before committing its offset for a processed message, what delivery guarantee does the system fall back to?
At-Least-Once Delivery. The restarted consumer will re-read and process the uncommitted message again, potentially generating duplicates.
Q2: What is Consumer Lag?
The difference between the Log-End Offset (LEO) and the consumer group's Committed Offset. A high lag indicates the consumer group is falling behind.
Scenario-Based Challenge
The Challenge: Designing an Idempotent Billing Audit
You are building a billing system. You consume message offsets sequentially. If your database write fails or your consumer crashes midway, committing the offset will lose the invoice. If you don't commit, you risk duplicate bills. How do you solve this?
Solution Tip: Implement idempotent processing. Store the committed offset inside the database transaction that writes the invoice. When processing a message, check if the record's offset is already in the database; if it is, skip it.
Debugging Exercise
A developer reports that every time their microservice container restarts, it reprocesses the last 5 minutes of orders, producing duplicate notifications.
The Fix: The microservice has enable.auto.commit = true but has long batch execution steps. The container crashed before the 5-second auto-commit interval fired, so the broker had no record of the read offsets. To resolve this, disable auto-commits and invoke commitSync() manually once processing is complete.
Interview Questions
1. What is the impact of commitAsync() on consumer execution?
It performs a non-blocking offset commit, improving consumer throughput. However, since it doesn't block, retrying failed commits can cause offset order issues.
2. What happens to a consumer's offset when a rebalance occurs?
The consumer commits its current offset. The group coordinator assigns the partition to another consumer, which reads starting from the last committed offset.
Production Considerations
Monitor consumer lag using metrics like records-lag-max. Set up alerts on Prometheus/Grafana to flag when lag grows beyond safe thresholds.