ReviseAlgo Logo

Kafka Architecture Deep Dive

Log Segments and Retention Policies

How files are segmented on disk and cleaned up via time/size thresholds.

Introduction

In Kafka, a partition is physically represented as a directory on the broker's local filesystem. Senders write events to this directory. However, Kafka does not write to a single file. Instead, the partition is split into Log Segments.

Why It Matters

Writing to a single file degrades performance. Searching for messages by offset requires scanning the entire file, and deleting old data blocks is extremely slow. Segmenting the log files into smaller blocks allows Kafka to quickly roll old segments, clean up data, and keep disk seeks low.

Real-World Analogy

Think of partition logs like a daily newspaper archivist. Instead of pasting every day's news onto one giant scroll of paper, the archivist binds them into daily folders (segments). When a folder is 7 days old, they can shred it (retention) easily without touching the other papers.

How It Works

The lifecycle of segment management flows as follows:

  1. Active Segment: The current segment where writes occur. Only one active segment exists per partition.
  2. Closed Segments: Read-only segments. Once a segment reaches size (default 1GB) or time limits (default 7 days), it rolls and closes.
  3. Index Files: Every segment has .log (raw events), .index (offset to physical byte map), and .timeindex files.

Internal Architecture

Kafka supports two data retention policies:

  • Log Deletion: Discards inactive segments once they exceed age thresholds (log.retention.hours) or size limits (log.retention.bytes).
  • Log Compaction: Retains only the latest value for each message key, discarding historical updates. This is useful for state cache storage.

Visual Explanation

Below is an ASCII representation of the partition segment directories on disk: 

/var/lib/kafka/data/orders-topic-0/
  ├── 00000000000000000000.log         (Closed segment, offsets 0-999)
  ├── 00000000000000000000.index
  ├── 00000000000000001000.log         (Active segment, offsets 1000+)
  └── 00000000000000001000.index

Practical Example

Below are CLI configurations demonstrating how to set segment size and time limits on a topic dynamically: 

# Set log retention to 72 hours (3 days)
kafka-configs.sh --bootstrap-server localhost:9092 --alter   --entity-type topics --entity-name orders-topic   --add-config retention.ms=259200000

# Set log segment roll size to 256MB
kafka-configs.sh --bootstrap-server localhost:9092 --alter   --entity-type topics --entity-name orders-topic   --add-config segment.bytes=268435456

Common Mistakes

  • Setting Retention Bytes lower than maximum batch sizes: Causes segments to delete as soon as they roll, resulting in message loss.
  • Enabling compaction on null-key topics: Log compaction requires key-value pairings. Null keys cause the compaction thread to throw errors and stall.

Quick Quiz

Q1: Does Kafka clean up or compress the active log segment?

No. Only closed, inactive segments are eligible for deletion or compaction. The active segment is left open for write appending.

Q2: How does Kafka find a message at offset 1005?

By performing a binary search on the segment index file (e.g. 1000.index) to locate the closest byte position, and scanning sequentially from there.

Scenario-Based Challenge

The Challenge: Sizing Retention for Disaster Recovery

You run a microservices log stream. Downstream analytics engines take 2 days to recover from database corruption and replay logs. The log volume is 2TB per day. How do you configure the topic's retention parameters?

Solution Tip: Set retention.ms to at least 3 days (259200000ms) to ensure the 2-day recovery window is fully covered. Additionally, verify that retention.bytes is set to at least 6TB (or disabled) to prevent disk space limits from triggering earlier deletions.

Debugging Exercise

A Kafka broker disk partition is 100% full, but the log cleanups are configured with a 3-day retention time limit. Senders cannot write.

The Fix: Check the size of the active segment. If segment.bytes=1073741824 (1GB) and traffic is very slow, the active segment has not reached 1GB and will not roll, preventing cleanups. To fix this:

  • Reduce segment.bytes.
  • Or configure a time-based roll setting: segment.ms=86400000 (24 hours).

Interview Questions

1. Explain what happens during log compaction.

A cleaner thread reads the dirty ratio of closed segments, merges duplicate keys, and writes compact segments, keeping only the latest value for each key.

2. What are .timeindex files used for?

To search for messages by timestamp. It maps a Unix timestamp to an offset number, letting consumers seek to a specific date/time.

Production Considerations

Configure XFS filesystem mount options (e.g. noatime) to prevent the OS from writing updates to files during read polls, improving IO performance.