ReviseAlgo Logo

Kafka Integration

@KafkaListener — Consuming Messages in Spring Boot

Configuring consumer group parameters, message handling methods, and concurrency configs.

Last Updated: June 15, 2026 • 15 min read

Introduction

Consuming messages asynchronously from a Kafka topic in Spring Boot is achieved using the @KafkaListener annotation. When you mark a method with @KafkaListener, Spring registers a message listener container behind the scenes to run the consumption poll loop, handle multi-threaded message routing, and commit partition offsets.

Why It Matters

Understanding consumer groups, thread limits, and offset commit parameters is critical to scaling applications. If your processing speed lags, you must spin up more consumer threads in the group. Properly configuring container properties like concurrency and AckMode guarantees your app handles rebalances safely without processing duplicate records or skipping uncommitted events.

Real-World Analogy

Think of a **busy delivery center desk** with multiple shelves (partitions) holding incoming mail:

A single consumer thread (clerk) has to walk back and forth, checking every shelf one by one. If you define a consumer group with concurrency = 3, you hire three clerks working in a shared workgroup. Each clerk takes exclusive responsibility for exactly one shelf. When mail arrives on their assigned shelf, they process it immediately. If one clerk goes on break (rebalance), the others temporarily cover their shelf.

How It Works

When a class bean has a method marked with @KafkaListener, Spring's post-processors wrap it inside a ConcurrentMessageListenerContainer:

  1. Thread Spin-Up: The container spins up separate worker threads matching the configured concurrency value.
  2. Consumer Poll Loop: Each thread instantiates a native KafkaConsumer client and starts a continuous poll loop (while (true) { consumer.poll() }).
  3. Offset Commits: Spring manages commit behaviors based on ContainerProperties.AckMode. The default mode is BATCH (commits offsets after the records retrieved from a single poll are processed). You can switch to MANUAL to explicitly trigger commits only after business transactions succeed.

Practical Example

Here is a listener configured with explicit concurrency, consumer groups, and manual offset acknowledgment:

Quick Quiz

Q1: What happens if concurrency is configured as 4 on a topic that only has 2 partitions?

A) Messages are duplicated to ensure all threads work

B) The application crashes at startup

C) Two consumer threads will allocate partitions, while the other two threads will remain idle

D) Kafka automatically creates 2 more partitions

Answer: C — In Kafka, only one consumer within a group can read from a partition. Extra threads remain idle as backup replicas.

Q2: Which container AckMode is required to use the Acknowledgment.acknowledge() method in listener signatures?

A) RECORD

B) BATCH

C) MANUAL or MANUAL_IMMEDIATE

D) TIME

Answer: C — If AckMode is set to default (BATCH), invoking acknowledge() has no effect and throws errors. Manual modes hand offset committing authority to the developer.

Scenario-Based Challenge

Production Scenario:

Your listener method takes 60 seconds to process a batch of records because it calls a legacy external service. The broker keeps triggering partition rebalances, assuming your consumer is dead. What properties must you tune to resolve this without reducing batch sizes?

View Solution

Increase the consumer property max.poll.interval.ms. This property defines the maximum time allowed between consecutive poll calls before the coordinator assumes the consumer thread has hung and triggers a rebalance.
Set max.poll.interval.ms: 120000 (2 minutes) to give your thread ample time to process the batch, or reduce max.poll.records so each poll retrieves fewer items.

Interview Questions

1. What causes a partition rebalance in a consumer group?

A rebalance occurs when: a new consumer instance joins the group, an active consumer leaves (shut down), or a consumer fails to send heartbeats within the session.timeout.ms window, or the consumer poll loop hangs longer than max.poll.interval.ms.

2. Can you receive metadata headers directly inside the @KafkaListener method?

Yes, using the @Header annotation. Common headers include KafkaHeaders.RECEIVED_KEY, KafkaHeaders.RECEIVED_PARTITION, KafkaHeaders.OFFSET, and KafkaHeaders.RECORD_METADATA.

Production Considerations

Keep the code inside your @KafkaListener methods fast and stateless. If an operation takes significant time, delegate it to an asynchronous worker thread pool (e.g. using Spring's @Async) or save the state and commit immediately, preventing consumer lag and coordinator heartbeat timeouts.