ReviseAlgo Logo

Kafka Internals — Storage & Replication

Log Compaction — Event Sourcing Use Cases

Cleaning logs via key deduplication for stateful caches.

Introduction

Log Compaction is a storage optimization policy where Kafka retains only the latest record value for each message key, discarding older updates. This allows partitions to serve as compact, stateful caches.

Why It Matters

Log compaction is fundamental for event-driven architectures. Senders must understand how compaction threads merge records and how Tombstone messages are used to delete keys in database synchronization pipelines.

Real-World Analogy

Think of log compaction like a household address book. Instead of writing down every house move a friend has ever made in history (log deletion), you only care about their current address. When they move, you overwrite their old address. If you want to delete them entirely, you scratch them out with a line (tombstone).

How It Works

The log cleaner thread operates as follows:

  1. Dirty segment: The log cleaner thread evaluates the uncleaned portion of closed segments.
  2. Key Deduplication: It builds a hash table mapping keys to their latest physical offset coordinates.
  3. Segment Merging: Reads dirty segments, copying only the latest offset values to new, compacted segments.
  4. Tombstone Deletion: A message with a null value represents a deletion (Tombstone). The log cleaner retains it for a set period (delete.retention.ms) before purging it from disk.

Internal Architecture

Log compaction utilizes the Log Cleaner background thread. It splits the partition segment logs into a "Clean" segment (deduplicated) and a "Dirty" segment (new append-only writes). The cleaner merges dirty records, preserving key ordering.

Visual Explanation

The diagram on the page shows the timeline of event-sourcing and how the Log Cleaner consolidates updates by retaining only the latest value per key:

Practical Example

Below is a configuration script demonstrating how to create and customize a log-compacted topic: 

# Create a log-compacted topic for database state storage
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create \
  --topic customer-profiles --partitions 3 --replication-factor 3 \
  --config cleanup.policy=compact \
  --config delete.retention.ms=86400000 # Retain tombstones for 24 hours

Common Mistakes

  • Publishing null-value records without understanding tombstones: Sending a null value to a compacted topic deletes the key. Senders who use null values to represent default states will see their data disappear during compaction.
  • Compacting topics with null keys: Log compaction requires keys. Topics containing records with null keys will trigger errors in the cleaner thread, stalling log cleanup.

Quick Quiz

Q1: What cleanup policy must be configured to enable log compaction on a topic?

cleanup.policy=compact.

Q2: What is a tombstone message in Kafka?

A record with a key and a null value, signaling the deletion of that key during log compaction.

Scenario-Based Challenge

The Challenge: Designing a user profile cache

Senders publish profile updates. The consumer needs to rebuild a local key-value state store on boot. Scanning the entire history of edits takes hours. How do you optimize the topic?

Solution Tip: Configure cleanup.policy=compact on the profile topic. Compaction will discard historical edits, leaving only the latest profile state, which speeds up consumer boot times.

Debugging Exercise

You observe that a compacted topic's size continues to grow, and old key updates are not being purged, even though the compaction policy is enabled.

The Fix: Compaction only cleans closed segments. The active log segment is never compacted. If segment.bytes is set to 1GB and write traffic is slow, the active segment has not rolled, preventing cleanup. Reduce segment.bytes or set segment.ms.

Interview Questions

1. How does Kafka handle deletions in compacted topics?

By writing a tombstone message (a record with a key and null value). The cleaner retains it for delete.retention.ms before purging it.

2. Can you combine log deletion and log compaction?

Yes. By setting cleanup.policy=compact,delete, Kafka compacts keys but also deletes compacted segments once they exceed retention thresholds.

Production Considerations

Allocate dedicated resources to log cleaner threads via log.cleaner.threads. Monitor JMX cleaner metrics closely.