Consumers & Consumer Groups
Offset Management — Auto-commit vs Manual commit
Guarantees of at-least-once, at-most-once, and manual commit patterns.
Introduction
Offset management defines how consumers record their reading progress back to the cluster. Consumers must decide whether to rely on Auto-Commit (which updates offsets periodically in the background) or Manual Commit (which explicitly saves offsets via code), determining the delivery guarantees of the system.
Why It Matters
Offset management is where system reliability is won or lost. Relying on auto-commit is simple but easily leads to duplicate processing or silent message loss when containers crash. Manual commits allow developers to implement strict At-Least-Once or At-Most-Once delivery pipelines.
Real-World Analogy
Think of a bookmark in a workbook. If you read a page and automatically write down your progress every 5 minutes (auto-commit), and you fall asleep midway, you'll wake up and assume you read pages you actually missed. If you bookmark only after completing each page's exercises (manual commit), you guarantees no missed steps, though you may re-read a page if you drop the book midway.
How It Works
Kafka supports three offset committing workflows:
- Auto-Commit: The consumer background thread automatically commits the last returned offset every 5 seconds (defined by
auto.commit.interval.ms). - Manual Sync Commit: The application thread blocks, sending an RPC request to the broker to save offsets:
consumer.commitSync(). - Manual Async Commit: The application thread registers a callback and triggers a non-blocking offset write:
consumer.commitAsync().
Internal Architecture
Committed offsets are stored in the internal partition topic __consumer_offsets. When a commit occurs, the coordinator appends the offset metadata. The consumer reads this offset during startup to resume consumption.
Visual Explanation
Below is a comparison of offset updates relative to processing stages:
[Auto-commit] Poll Records ──> [5-sec Timer: Auto-Commit Offset] ──> Process Records (Crash here loses data)
[Manual Commit] Poll Records ──> Process Records ──> [commitSync() Client Call] (Safe At-Least-Once Delivery)
Practical Example
Below is a Java snippet showing safe manual commit handling with error catching and async/sync fallback:
Common Mistakes
- Using Auto-Commit with Asynchronous Workers: If you poll messages and delegate them to background thread pools while immediately calling
poll()again, auto-commit will save offsets before workers finish, risking silent loss on crashes. - Not handling OffsetCommitCallback errors: Async commits do not retry automatically. If a commit fails due to a temporary network blip, subsequent writes could cause out-of-order offset saves if not monitored.
Quick Quiz
Q1: What delivery guarantee is achieved when you process a record first and then manually commit its offset?
At-Least-Once Delivery.
Q2: Why is commitSync() generally avoided inside the main message processing loop?
It blocks the consumer thread, waiting for network confirmation from the broker, which degrades throughput.
Scenario-Based Challenge
The Challenge: Implementing At-Most-Once delivery
You are building a non-critical telemetry logs dashboard. If a worker crashes, you prefer to drop missing records rather than reprocess duplicate logs. How do you configure commits?
Solution Tip: Set enable.auto.commit = false and commit offsets immediately after calling poll() before processing the records.
Debugging Exercise
After service redeployments, users complain they receive duplicate shipping notifications for items purchased up to 10 seconds before the update.
The Fix: The system is using auto-commit with default auto.commit.interval.ms=5000. Senders shutdown containers without calling commitSync(), forcing restarted nodes to reprocess the last 5 seconds of logs. Fix this by invoking manual commits during JVM shutdown hooks.
Interview Questions
1. What is the impact of commitAsync() on offset commit ordering?
If commit 1 for offset 100 fails, and commit 2 for offset 110 succeeds, a retry of commit 1 could overwrite offset 110 with 100. Thus, async commits should not retry.
2. Where does Kafka store committed offsets?
In the internal topic __consumer_offsets on the broker.
Production Considerations
Always disable auto-commit for transactional pipelines. Monitor lag using __consumer_offsets logs.