ReviseAlgo Logo

Kafka Architecture Deep Dive

Kafka's Commit Log — Why It's Different

How append-only logs on disk achieve sequential I/O and zero-copy performance.

Introduction

Relational databases use complex B-tree indexes, locking mechanisms, and transaction buffers to support arbitrary queries. Apache Kafka is built as an immutable, append-only commit log. This design difference is why Kafka can stream gigabytes of data at disk wire speed.

Why It Matters

Understanding why Kafka writes to disk sequentially explains why it is incredibly fast. Disk random I/O is slow, but sequential disk access is extremely fast. Leveraging page caches and zero-copy transfers allows Kafka to bypass JVM heap memory limits, maximizing network utilization.

Real-World Analogy

Think of an immutable assembly line conveyor belt. Items are added to the end of the belt and move forward sequentially. No worker searches the middle of the belt or moves items out of order. They simply grab items sequentially as they pass, matching the speed of the line.

How It Works

Kafka's sequential disk append and read delivery workflow uses these techniques:

  • Sequential Disk Access: Writes always append to the active log segment, keeping disk head seeks minimal.
  • OS Page Cache: Kafka delegates message buffering to the OS Page Cache. All reads and writes hit system RAM first.
  • Zero-Copy Data Transfer: Leverages the sendfile system call to transfer byte blocks directly from the OS page cache to the network socket, bypassing the JVM space.

Internal Architecture

Traditional databases route messages from the disk controller, to the OS page cache, copy to the user space JVM memory, copy back to the kernel socket buffer, and push to the network interface card.

Kafka's Zero-Copy pipeline routes data directly: 

Page Cache (Kernel space) ──[sendfile() system call]──> Socket Buffer ──> NIC

This zero-copy route removes CPU context switches and context duplication overhead.

Visual Explanation

The visual page explains this kernel-space bypass, illustrating data routing from kernel cache directly to network sockets without crossing JVM borders.

Practical Example

Below is a Java snippet showing how to use FileChannel.transferTo() to perform zero-copy file transfer, matching Kafka's inner transport mechanism:

Common Mistakes

  • Over-allocating JVM Heap Memory: Setting the JVM heap size to 32GB on a 64GB machine. Kafka brokers need very small heaps (e.g. 4GB-6GB). The remaining system RAM must be left free for the OS Page Cache.
  • Using slow network filesystems (NFS) for log storage: NFS creates high latency network roundtrips, defeating the speed of local sequential disk appends.

Quick Quiz

Q1: Why is sequential disk write speed comparable to sequential RAM writes?

Because sequential writes write to contiguous disk blocks, avoiding mechanical disk head seek latency.

Q2: How does zero-copy transfer reduce broker resource utilization?

It removes the CPU overhead of moving data back and forth between kernel space and user space, reducing CPU cycles.

Scenario-Based Challenge

The Challenge: Sizing JVM heap for a 128GB RAM Node

You deploy a Kafka broker on a high-spec server with 128GB RAM. Senders write 200MB/s of events. SRE wants to know how to configure the JVM garbage collector and heap limits.

Solution Tip: Set the JVM heap size to only 6GB-8GB. Allocate the remaining 120GB to the Linux OS Page Cache. This ensures that most of the log files reside in RAM, enabling consumers to read using zero-copy transfers from memory.

Debugging Exercise

A broker suffers from heavy GC (Garbage Collection) pauses, slowing down throughput. Investigation shows the JVM heap was set to 64GB on a 96GB broker node.

The Fix: The large JVM heap forces the garbage collector to scan a massive object space, causing pauses. Reduce the JVM heap to 6GB and apply the G1GC garbage collector options: 

export KAFKA_HEAP_OPTS="-Xms6g -Xmx6g"
export KAFKA_JVM_PERFORMANCE_OPTS="-XX:+UseG1GC -XX:MaxGCPauseMillis=20"

Interview Questions

1. How does the immutable nature of commit logs simplify concurrency?

Because data is append-only, there are no random update locks or B-tree rebalances. Senders and receivers can access files concurrently without locking.

2. What is the role of page cache in Kafka reads?

When a consumer requests a message, the broker checks the page cache. If the message was recently written, it is read from RAM cache, avoiding disk read I/O.

Production Considerations

Configure the Linux OS vm.swappiness=1 parameter to prevent the kernel from swapping page cache bytes to disk, which would degrade throughput.