ReviseAlgo Logo

Kafka Integration

Spring Kafka Overview — Producer and Consumer Setup

Defining properties configs for bootstrap servers, serializers, and partition topics.

Last Updated: June 15, 2026 • 12 min read

Introduction

The Spring for Apache Kafka (Spring Kafka) framework wraps the native Apache Kafka Java client libraries with clean, declarative configuration models and familiar Spring patterns (like template classes, admin clients, and listener containers). Instead of writing low-level poll loops and connection builders manually, Spring developers can build message-driven architectures using standard dependency injection and stereotyping annotations.

Why It Matters

Establishing stable connections to a Kafka broker cluster is critical. Misconfiguring serializers (e.g., mismatching String vs JSON formats) or buffer sizes is a primary cause of startup failures, payload processing bottlenecks, and data loss. Setting up robust producer and consumer connection factories guarantees high-performance serialization, automatic thread safety, and transparent connection recovery.

Real-World Analogy

Think of setting up Spring Kafka like designing a global cargo shipping port. Before you can transport any shipping containers (messages), you must construct the entry lanes (producers), exit lanes (consumers), establish standardized container labeling rules (serializers and deserializers), and map out the shipping docks (topics). Once these connections and protocols are laid down, cargo flows smoothly without trucks needing to coordinate manually.

How It Works

When a Spring Boot application boots up, the auto-configuration engine (via KafkaAutoConfiguration) registers core beans:

  1. Properties Binding: Spring reads settings under the spring.kafka.* prefix in application.yml and maps them to a KafkaProperties object.
  2. Factories Initialization: Using those properties, Spring registers a DefaultKafkaProducerFactory and a DefaultKafkaConsumerFactory. These factories act as connection pools, generating thread-safe client producer and consumer instances dynamically.
  3. Topic Management: A KafkaAdmin bean scans the application context. If it detects any NewTopic beans, it uses the AdminClient API to automatically provision those topics on the Kafka broker at startup.

Practical Example

Here is how to configure a custom ProducerFactory, ConsumerFactory, and a Topic provisioning bean using Java Configuration:

Alternatively, you can configure these parameters cleanly inside your application.yml:

Quick Quiz

Q1: Which factory is responsible for pooling and instantiating Kafka Consumer instances in Spring Boot?

A) DefaultKafkaProducerFactory

B) DefaultKafkaConsumerFactory

C) KafkaAdminContainerFactory

D) ConcurrentMessageListenerContainer

Answer: B — DefaultKafkaConsumerFactory is the default pooled implementation creating native KafkaConsumer instances based on configured properties.

Q2: How does KafkaAdmin detect and create new topics on the brokers at startup?

A) It scans for database table mappings

B) It runs SQL setup scripts

C) It automatically detects any registered NewTopic beans in the Spring application context and provisions them

D) You must manually write REST controllers to create topics

Answer: C — Defining a NewTopic bean registered in the Spring context is sufficient; KafkaAdmin detects it and provisions it asynchronously.

Scenario-Based Challenge

Production Scenario:

Your application connects to an enterprise Kafka cluster requiring SASL_SSL authentication. When running integration tests locally, you want to use a local plain-text broker instead of checking credentials. How do you design your properties setup?

View Solution

Use Spring Profiles. Define a default application.yml with the base bootstrap servers. Create a profile-specific config application-prod.yml containing the security properties:


When running locally, do not activate the prod profile, ensuring it uses the default plain-text configuration.

Interview Questions

1. Why must you configure the spring.json.trusted.packages property when using JsonSerializer/JsonDeserializer?

By default, Jackson deserializers prevent random object mappings to prevent remote code execution (RCE) vulnerabilities. Configuring spring.json.trusted.packages explicitly whitelist package prefixes (or "*" for all) that the deserializer is allowed to bind incoming JSON payloads to.

2. What happens if auto.create.topics.enable is true on the broker, but you do not define a NewTopic bean in Spring?

When a producer attempts to publish to or a consumer attempts to read from a missing topic, the broker will create it automatically with default partitions (usually 1) and replication factors (usually 1). This is dangerous in production, as single-partition topics restrict client scaling. Always provision topics explicitly.

Production Considerations

In production environments, always set auto.create.topics.enable=false on your Kafka broker configuration. Topics should be explicitly provisioned via infrastructure-as-code (Terraform) or explicitly declared via NewTopic beans with replica counts matching broker counts (minimum of 3 replicas for high availability).