ReviseAlgo Logo

Kafka Integration

Error Handling and Dead Letter Topics in Spring Kafka

Setting up SeekToCurrentErrorHandler, retry policies, and dead letter publication.

Last Updated: June 15, 2026 • 15 min read

Introduction

Fault tolerance is mandatory in message-driven systems. If a consumer throws an exception, Spring's default behavior is to retry. If the exception is permanent (e.g., malformed JSON), retrying blocks the partition (poison pill). Spring Kafka resolves this via custom Error Handlers and Dead Letter Queues (DLQs).

Why It Matters

Uncontrolled errors block offset commits, causing message lag. Dead Letter Topics (DLT) allow routing unprocessable records to a separate stream, enabling manual inspect or reprocessing without stopping consumer pipelines.

Real-World Analogy

Think of a **manufacturing inspection line**. If a bottle has a loose cap (transient error), the mechanical arm attempts to tighten it again (retry). If a bottle is cracked and empty (poison pill), trying to fill it again is useless and spills liquid (lag). Instead, the machine pushes it off the belt into a "waste box" (DLT) so the conveyor belt can keep moving bottles.

How It Works

In modern Spring Kafka, error management is handled by the container factory:

  1. DefaultErrorHandler: Handles consumer exceptions, executing configured retries (e.g. 3 attempts) and backoffs (e.g. exponential backoff).
  2. Recoverer Delegation: If retries are exhausted, the handler delegates recovery to a DeadLetterPublishingRecoverer.
  3. DLQ Publication: The recoverer publishes the failed record to a designated DLT topic (defaults to <original-topic-name>.DLT). It adds debugging headers (like stack traces and exception messages) directly to the Kafka record.

Practical Example

Here is how to configure a custom DefaultErrorHandler with a DeadLetterPublishingRecoverer, including exponential retry backoff:

Quick Quiz

Q1: What is a "poison pill" message in a Kafka topic?

A) A message containing delete script payloads

B) A malformed record that fails processing or deserialization consistently, blocking consumer progress on the partition

C) An encrypted payload

D) A message that triggers consumer group coordinator rebalances

Answer: B — Poison pills are messages that cannot be processed successfully. Without exclusion or DLQ routing, they lock partitions since offset commits are blocked.

Q2: Where does DeadLetterPublishingRecoverer store information about why the message failed?

A) In a local database file

B) It appends the logs to the end of the message payload

C) In the Kafka record headers (e.g. x-exception-stacktrace, x-exception-message)

D) It displays them on the system console only

Answer: C — Exception properties and trace data are written cleanly into the headers of the published DLT record.

Scenario-Based Challenge

Production Scenario:

Your consumer processes order messages. You encounter a NullPointerException (permanent business logic issue) and a ResourceAccessException (transient database timeout). How should your error handling handle these differently to minimize load?

View Solution

Configure classification in your DefaultErrorHandler. Use addNotRetryableExceptions(NullPointerException.class) to route null pointer errors directly to the DLT without retrying, preventing useless CPU cycles. Allow default or exponential backoff retries for ResourceAccessException, giving the database time to recover.

Interview Questions

1. What is the difference between blocking retries and non-blocking retries in Kafka?

Blocking retries pause the consumer thread on the current partition, retrying the record before polling new ones (DefaultErrorHandler default). Non-blocking retries forward the failed message to a retry topic (e.g. topic-retry-5s) and commit the original offset, allowing subsequent messages in the partition to be processed without waiting.

2. What are the key headers added to a DLT record by Spring's recovery handler?

Spring appends headers like x-original-topic, x-original-partition, x-original-offset, x-exception-fqn (fully qualified class name), x-exception-message, and x-exception-stacktrace.

Production Considerations

Always ensure that your Dead Letter Topics are created with the exact same partition counts as your original topics, or configure your DeadLetterPublishingRecoverer with a custom destination resolver that maps all failures to partition 0 of the DLT. This prevents broker routing errors when publishing recovery payloads.