ReviseAlgo Logo

Real-World System Design with Kafka

Kafka in a Microservices Architecture — End-to-End Design

Structuring domains, topics, message contracts, and handling sync/async boundaries.

Introduction

Integrating microservices using synchronous REST APIs creates a fragile system: if one service is slow, the entire dependency chain stalls. Kafka in a Microservices Architecture replaces blocking calls with asynchronous event streams. This decouples services, manages failures with Dead Letter Queues (DLQs), and guarantees transaction boundaries.

Why It Matters

When a consumer encounters a corrupted or unparseable record (known as a "poison pill"), it fails to commit its offset. Without a dedicated recovery strategy, the consumer gets stuck in a loop retrying the same record, blocking all subsequent partition events. Dead Letter Queues bypass these poison pills, ensuring continuous pipeline flow.

Real-World Analogy

Think of microservice event routing like a factory assembly line. Workers assemble boxes coming down the conveyor belt (event partitions). If a worker picks up a box with missing parts (poison pill), they cannot stop the entire factory line while they debug it. Instead, they place the problematic box in a red bin on the side (Dead Letter Queue) and immediately grab the next box, leaving the red bin for an inspector to examine later.

How It Works

A resilient microservice event pipeline processes errors using a layered topology:

  1. Primary Processing: The consumer pulls records from the main topic and attempts to execute the business logic.
  2. Local Retries: If transient exceptions occur (e.g. database timeout), it retries the process locally using backoff delays.
  3. Redirect to DLQ: If retries exhaust or if a non-transient error occurs (e.g., validation failure), the event is published to a dead letter topic (e.g., topic-orders-dlq) along with error context headers.
  4. Commit & Continue: The consumer commits the original offset and proceeds to the next record immediately.

Internal Architecture

Dead Letter Queues are normal Kafka topics. When a consumer redirects an event, it injects custom diagnostic headers into the record: x-original-topic, x-exception-message, and x-exception-stacktrace. Operations teams monitor DLQ topic traffic to spot software bugs, while alert rules notify engineers when DLQ volumes rise, preventing silent failures.

Visual Explanation

Below is an ASCII representation showing the retry and dead letter routing topology for handling message processing failures: 

 [Main Ingestion Stream] ──> [ Consumer App ] ──(Success)──> [DB Commit]
                                   │
                                   ▼ (Exception)
                        [ Retry 3x with Backoff ]
                                   │
                                   ▼ (Fails / Poison Pill)
                        [ Publish to orders-DLQ ] ──> Commit Main Offset

Practical Example

Here is a Spring Boot configuration defining a retry template that routes persistent errors to a DLQ topic:

Common Mistakes

  • Routing all exceptions immediately to the DLQ: Bypassing retries for temporary database socket timeouts. This inflates DLQ volume unnecessarily. Always retry transient failures locally first.
  • Executing infinite retries on poison pills: Leaving no max limit on retries. The partition processing freezes permanently, halting pipeline throughput.

Quick Quiz

Q1: How does a dead letter queue prevent partition blockages when a poison pill record is read?

By copying the invalid record to a secondary DLQ topic, committing the original partition offset, and moving to the next message.

Q2: Why must exceptions like JSON parsing errors skip retry loops and route to DLQs immediately?

Because syntax errors are non-transient; retrying them will yield the identical failure, wasting CPU cycles and blocking the thread.

Scenario-Based Challenge

The Challenge: Re-processing repaired events from the DLQ

A software bug caused your payment gateway to reject messages with a missing field, routing 5,000 orders to the DLQ. After deploying a fix, how do you re-ingest these 5,000 orders back into the primary pipeline without restarting or duplicating data?

Solution Tip: Write a utility consumer that subscribes exclusively to the DLQ topic, fixes the missing attributes, and re-publishes the repaired payloads back to the primary ingestion topic. Since the bug is fixed, the main consumer will process them cleanly.

Debugging Exercise

Your consumer group stops processing, but no errors are printed in application logs and partition lag metrics are mounting.

The Fix: Check the thread logs. A thread is blocked inside a synchronous HTTP client call (e.g. REST request to billing). Always configure connection and read timeouts on client instances, and wrap calls in try-catch blocks to ensure exceptions fire, triggering error handlers and DLQ routing rather than freezing.

Interview Questions

1. Compare transactional dead letter queues with database-backed retry tables.

DLQ topics scale linearly using Kafka's log partitions without database lock contentions. Database-backed retry tables are easier to inspect and edit but introduce heavy write overheads under massive scales.

2. What diagnostic headers should be included when routing a message to the DLQ?

Inject the original topic name, original partition index, offset, failure timestamp, exception message, and the stack trace details.

Production Considerations

Always monitor DLQ topic lags and configure automated alerts when DLQ writes spike. Restrict read access to DLQ topics as they contain raw payloads which may carry sensitive business data.