ReviseAlgo Logo

Kafka Internals — Storage & Replication

How Kafka Stores Data on Disk

Disk formats, indexes (.index, .timeindex), and sparse indexing.

Introduction

Under the hood, Apache Kafka stores partition events as raw binary data files on the broker's local filesystem. Senders write records to an active segment file, which is indexed using separate, highly compact index files that support fast, binary seeks by offset or timestamp.

Why It Matters

Kafka's disk storage engine is designed to treat disk access sequentially. Senders must understand how index files (.index and .timeindex) map to partition log segments to debug issues like disk fragmentation, indexing timeouts, and slow seek times during consumer initialization.

Real-World Analogy

Think of Kafka's storage like a large warehouse catalog. The physical goods (raw events) are stored in sequential aisles (segment .log files) in the order they arrived. Instead of checking every box to find a product, you look at a brief summary sheet (sparse .index file) listing coordinates for every 4000th item, helping you skip directly to the target shelf.

How It Works

The segment storage pipeline coordinates files as follows:

  1. Append-Only Write: Senders append binary records sequentially to the active segment file (e.g. 0000.log).
  2. Index Interleaving: For every N bytes written (configured by index.interval.bytes, default 4KB), Kafka appends an index entry containing the relative offset and physical file byte position.
  3. Time indexing: Appends a timestamp-to-offset mapping to 0000.timeindex to support date-time seeking.
  4. Roll and Lock: Once the active segment exceeds segment.bytes (default 1GB), it is closed as read-only and a new active segment is initialized.

Internal Architecture

The indices are stored as sparse indexes. Instead of mapping every single offset, Kafka maps physical byte positions at set intervals (default 4KB). This allows the entire index block to fit into the Linux OS kernel page cache, avoiding expensive disk reads during offset seek operations.

Visual Explanation

Below is an ASCII representation of the partition log directory and sparse indexing mechanism: 

[Segment 000000.log (Aisles)] ────> Offset 0 (0B) ──> Offset 10 (4KB) ──> Offset 20 (8KB)
                                          ▲                  ▲                  ▲
[Segment 000000.index (Sparse)] ───> Offset 0 (0B) ──────────┼──────────────────┘
                                                             (Skip scanning intermediate offsets)

Practical Example

Below is an CLI execution demonstrating how to dump and analyze the binary segment log and index metadata files directly using Kafka's tools: 

# Run from the Kafka installations directory to dump log file records
bin/kafka-run-class.sh kafka.tools.DumpLogSegments \
  --files /var/lib/kafka/data/orders-0/00000000000000000000.log \
  --print-data-log

# Dump index offsets to inspect sparse offset mapping
bin/kafka-run-class.sh kafka.tools.DumpLogSegments \
  --files /var/lib/kafka/data/orders-0/00000000000000000000.index

Common Mistakes

  • Manually editing or deleting log files: Deleting .log files without cleaning up matching .index files corrupts partition offsets, causing brokers to crash on startup with index validation check failures.
  • Setting index.interval.bytes too low: Decreasing the sparse indexing interval increases index size in memory, causing page cache pressure and degrading search performance.

Quick Quiz

Q1: What type of index does Kafka use to map message offsets to physical file byte coordinates?

A sparse index.

Q2: What is the default size limit for a single partition log segment file before it rolls?

1GB (configured by log.segment.bytes).

Scenario-Based Challenge

The Challenge: Optimizing a topic with slow time-based seeking

You seek messages by timestamp (e.g. "replay payments from yesterday"), but the consumer initial seek blocks for over 10 seconds. You verify that page cache swappiness is disabled. How do you resolve this?

Solution Tip: Check if the topic has been configured with very small segment sizes (e.g., 1MB instead of 1GB). Having tens of thousands of segment files force the time index seek to perform binary searches across thousands of index files, degrading latency. Merge segments by increasing segment.bytes.

Debugging Exercise

A broker crashes and restarts with the error: CorruptIndexException: Found index file with size 10485760 but expected....

The Fix: The index file was not closed cleanly due to an abrupt power failure. To resolve this, delete the corrupted .index or .timeindex file. On startup, Kafka will automatically scan the raw .log file and rebuild the index files from scratch.

Interview Questions

1. Why does Kafka use sparse indexes instead of indexing every message offset?

To keep index file sizes extremely small, allowing the OS to load them entirely into the kernel page cache (RAM), minimizing disk lookups.

2. What are the three primary file types that represent a partition log segment on disk?

.log (raw event data payload), .index (offset-to-byte mapping), and .timeindex (timestamp-to-offset mapping).

Production Considerations

Run on XFS or ext4 filesystems with noatime flags. Ensure sufficient disk capacity to handle peak streaming volumes.