ReviseAlgo Logo

Consumers & Consumer Groups

Dead Letter Queues and Error Handling in Consumers

Retries, backoffs, and routing poison pills.

Introduction

When a consumer encounters a malformed or corrupt message (a "poison pill"), processing fails. A Dead Letter Queue (DLQ) is a dedicated topic where consumers route failed messages, allowing the pipeline to continue reading subsequent records without locking.

Why It Matters

Without error routing, a single malformed message will stall a consumer partition indefinitely. If the consumer fails to parse a message, retrying in a loop blocks the thread, building up lag across the entire system.

Real-World Analogy

Think of a customs inspection line. If an inspector finds a suspicious package with incorrect labeling, they don't halt the entire shipping lane and block all trucks behind it. Instead, they move the suspicious package to a side holding room (Dead Letter Queue) for investigation, letting the main line resume.

How It Works

The DLQ routing flow operates as follows:

  1. Consume: The consumer retrieves a batch of records.
  2. Error Detection: A record fails to deserialize or parse (e.g. invalid JSON schema).
  3. Publish to DLQ: The consumer catches the exception and publishes the malformed message payload along with header errors to a dedicated DLQ topic.
  4. Commit and Resume: The consumer commits the offset of the failed record, resuming the poll loop for the next message.

Internal Architecture

The DLQ is implemented as a standard Kafka topic. Error handlers write to this topic asynchronously. Senders include header metadata (e.g., x-exception-message, x-original-topic, x-original-partition) to aid debugging.

Visual Explanation

Below is a visual layout of error routing in a consumer parse loop: 

Main Topic ──> [Consumer Parse Loop] ──(Success)──> Process Node
                       │
                    (Failure)
                       │
                       ▼
           [Publish to DLQ Topic] ──> Admin/Alerting Analysis

Practical Example

Below is a Java snippet demonstrating how to catch parse errors and route failed records to a DLQ topic:

Common Mistakes

  • Routing to DLQ without committing the original offset: If you send the bad message to the DLQ but fail to commit the offset, the consumer will pull the exact same message in the next poll, getting stuck in an infinite retry loop.
  • Using a blocking write for DLQ publishing without timeouts: Blocking on the DLQ producer write without safety bounds can cause the consumer to hang if the DLQ topic partition broker goes offline.

Quick Quiz

Q1: What is a "poison pill" message?

A malformed or corrupt record that continuously fails processing, preventing the consumer thread from progressing.

Q2: Why should you include error messages in record headers when routing to a DLQ?

To let SRE and developers inspect why the message failed parsing without needing to dig through application logs.

Scenario-Based Challenge

The Challenge: Handling transient database failures

Senders write logs. The consumer writes them to a SQL database. The database goes offline for 1 minute. Should you route these records to a DLQ?

Solution Tip: No. DLQs are for permanent code/data errors. For transient database connectivity failures, implement retries with backoffs, or pause consumer consumption using consumer.pause() to wait.

Debugging Exercise

Malformed records are piling up in your DLQ topic, but SRE reports they cannot find which consumer node routed them or what exception occurred.

The Fix: The DLQ producer code is routing raw keys and values without attaching metadata headers. Add headers detailing the source hostname, original topic-partition, and stack trace bytes.

Interview Questions

1. What is the difference between retry queues and dead letter queues?

Retry queues are for transient network/resource failures (retried later). DLQs are for permanent parsing or logic failures that require manual code reviews.

2. How do you avoid processing ordering issues when using a DLQ?

DLQ routing breaks strict partition order because subsequent events are processed while the failed event is held. If strict order is required, you must block the partition and alert, rather than routing to a DLQ.

Production Considerations

Define alerting thresholds on the DLQ topic log sizes. Set up automated scripts to re-inject messages after code deployments.