ReviseAlgo Logo

Kafka Integration

KafkaTemplate — Producing Messages with Headers and Keys

Publishing records asynchronously and handling success/failure callbacks.

Last Updated: June 15, 2026 • 12 min read

Introduction

Publishing events to Apache Kafka from a Spring Boot application is handled by KafkaTemplate. The template wraps the native KafkaProducer client, managing the thread-safe lifecycle of publishing records. It provides overloaded send methods to write simple payloads, send keys for partition routing, and add metadata headers.

Why It Matters

Publishing messages is inherently asynchronous. When you call KafkaTemplate.send(), the method returns a CompletableFuture immediately, pushing the record to a local memory accumulator. If you block on this future (using `.get()`), you degrade throughput significantly. Writing callback handlers ensures your application processes receipt confirmations or connection failures without blocking active application threads.

Real-World Analogy

Think of sending packages through a **courier warehouse**:

Calling send() is like tossing a package onto a loading conveyor belt. If you write a routing key (like "Aisle 4") on it, the sorting machinery guarantees that every package marked "Aisle 4" will land in the exact same delivery van (partition), ensuring they arrive in order. When the truck delivers the package, the courier returns a delivery receipt callback containing timestamps and logs.

How It Works

The lifecycle of a publish request through KafkaTemplate follows these stages:

  1. Record Formatting: Spring maps payload objects to a ProducerRecord, resolving serializers, headers, and routing keys.
  2. Partition Routing: If a key is present, the partitioner hashes it (murmur2) to assign it to a consistent partition. If the key is null, it uses a round-robin / sticky partition strategy.
  3. Record Accumulator: The record is pushed to a thread-safe local memory buffer categorized by topic-partition.
  4. Sender Loop: A background sender thread flushes the buffer to the brokers based on configurations like linger.ms and batch.size.

Practical Example

Here is how to publish a message containing routing keys and metadata headers, registering asynchronous callback handlers:

Quick Quiz

Q1: How can you guarantee that all messages relating to a specific customer are processed in the exact order they were sent?

A) Route them to different partitions using random keys

B) Set the consumer concurrency to 1

C) Use the customer ID as the routing key when calling KafkaTemplate.send()

D) Put all messages in a database table first

Answer: C — Specifying the same routing key guarantees that all messages for that key land in the same partition, preserving sending order.

Q2: What is the risk of calling future.get() directly after calling KafkaTemplate.send()?

A) The message is deleted on the broker

B) It forces the producer to block, converting the publication flow into a synchronous call and reducing throughput

C) The serializer crashes

D) The partition is locked permanently

Answer: B — CompletableFuture.get() is a blocking call. Blocking on every send removes the benefits of Kafka's asynchronous memory buffering.

Scenario-Based Challenge

Production Scenario:

Your billing service needs to process payments and publish a PaymentCompleted event. If the database update succeeds but the Kafka publish fails, your system enters an inconsistent state. How do you guarantee atomicity across the database and Kafka?

View Solution

Configure Transactional Producers in Spring Kafka.

  1. Define a transactional-id prefix in your producer properties (this enables idempotent and transactional behavior).
  2. Ensure your service method is annotated with standard Spring @Transactional.
When transactional IDs are present, Spring automatically registers a KafkaTransactionManager. If the database save fails or the Kafka send fails, both the database write and the Kafka message are rolled back atomically.

Interview Questions

1. What are the key producer configs required to ensure no message loss (At-Least-Once delivery)?

You must set acks=all (or acks=-1, requiring all in-sync replicas to confirm receipt), configure retries to a high number, and ensure enable.idempotence=true (to prevent duplicate writes on retry).

2. How does linger.ms and batch.size improve throughput in KafkaTemplate?

batch.size sets the memory size threshold for grouping records before sending. linger.ms sets a delay (e.g. 5ms) to wait for more records to join the batch before sending. This aggregates small requests into a single network packet, boosting network throughput.

Production Considerations

In production, enable idempotent producing: spring.kafka.producer.properties.enable.idempotence=true (default in modern clients). This protects your cluster from duplicate event writes caused by producer-broker connection retries when acknowledgments are lost over unstable networks.