ReviseAlgo Logo

Consumers & Consumer Groups

How Kafka Consumers Work

The poll loop, partition subscription, and deserialization.

Introduction

An Apache Kafka consumer is a client application that pulls record batches from a Kafka cluster. Kafka utilizes a pull-based model, where consumers actively request data from specific partition leaders, allowing clients to consume at their own pace without broker-side backpressure bottlenecks.

Why It Matters

Writing a high-throughput, resilient consumer requires mastering the consumer poll loop, partition subscriptions, and network connection properties. Misunderstanding how the poll loop triggers heartbeats can lead to continuous consumer timeouts and rebalance loops in production.

Real-World Analogy

Think of a Kafka consumer like a library book reader. The library (broker) keeps books on shelves. The reader (consumer) walks to the library, selects books they want to read (pulls data), and takes them to their desk. The library does not throw books at the reader; the reader controls their reading speed, pausing to take notes (process records) as needed.

How It Works

The lifecycle of consumer coordination flows as follows:

  1. Group Registration: The consumer contacts the broker designated as Group Coordinator to join a consumer group.
  2. Partition Assignment: The Group Coordinator assigns specific topic-partitions to the consumer.
  3. The Poll Loop: The consumer calls poll() in a loop, retrieving record batches from assigned partitions.
  4. Deserialization: Converts raw byte arrays from the broker into Java/Python/Go objects.
  5. Heartbeat Thread: In the background, the consumer sends periodic heartbeats to the Group Coordinator to declare its node liveness.

Internal Architecture

The consumer client utilizes a single-threaded execution model for its public API. The application thread invokes poll() to execute network polls and trigger offset commits. In the background, a network client coordinates socket writes, and a coordinator thread manages heartbeats to prevent timeouts. Senders must ensure the execution time between poll() calls is lower than max.poll.interval.ms.

Visual Explanation

Below is a flow diagram of the consumer poll loop pipeline: 

[Application Thread Loop]
consumer.poll() ──> Fetcher (Pulls socket bytes) ──> Deserializer ──> App Process
       │
       ▼
[Background Thread]
Heartbeat Thread ──(Liveness ping every heartbeat.interval.ms)──> Group Coordinator

Practical Example

Below is a Java snippet showing how to configure and run a standard consumer poll loop:

Common Mistakes

  • Sharing a Consumer instance across threads: KafkaConsumer is NOT thread-safe. Attempting to use a single instance across multiple threads will cause ConcurrentModificationException. Run one consumer instance per thread instead.
  • Blocking the poll loop thread: Running long-running operations (e.g., database writes or API calls taking minutes) inside the poll loop blocks heartbeats, causing the broker to mark the consumer dead and trigger a rebalance.

Quick Quiz

Q1: What exception is thrown if you try to share a single KafkaConsumer instance across multiple application threads?

ConcurrentModificationException.

Q2: Why does Kafka use a pull-based model instead of a push-based model?

To prevent the broker from overwhelming slow consumers. Consumers pull data only when they are ready to process it.

Scenario-Based Challenge

The Challenge: Handling slow processing batches

Your consumer processes batch records. Each record requires calling a slow external shipping API. The API is rate-limited and averages 500ms per call. A poll returns 500 records. How do you configure the consumer to prevent rebalances?

Solution Tip: Either decrease the batch size using max.poll.records (e.g. set it to 50), or increase max.poll.interval.ms to give the consumer thread enough time to complete the batch, or delegate work to worker thread pools.

Debugging Exercise

A consumer application starts logging warnings: Commit failed: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member.

The Fix: The consumer took longer than max.poll.interval.ms to process a batch of records. The coordinator assumed the consumer crashed, evicted it from the group, and reassigned its partitions. To fix this, decrease max.poll.records or optimize processing code.

Interview Questions

1. Explain the difference between subscribe() and assign().

subscribe() joins a consumer group and dynamically balances partitions across group members. assign() explicitly binds a consumer to specific partition numbers, disabling group coordination and rebalances.

2. What is the role of heartbeat.interval.ms?

It defines the frequency of background heartbeats sent to the Group Coordinator to maintain group membership. It is normally set to 1/3 of session.timeout.ms.

Production Considerations

Set session.timeout.ms=45000 (45 seconds) and heartbeat.interval.ms=15000 (15 seconds) to avoid false failures due to minor network blips.