ReviseAlgo Logo

Kafka Architecture Deep Dive

The Kafka Cluster — Brokers, ZooKeeper, and KRaft

How clusters coordinate metadata and coordinate partition leadership.

Introduction

A Kafka cluster is a collection of Brokers cooperating to handle partition writes. Historically, Kafka relied on Apache ZooKeeper to manage cluster metadata and state consensus. Modern Kafka implements KRaft (Kafka Raft Metadata Mode), moving consensus directly inside Kafka itself to remove external dependencies.

Why It Matters

In ZooKeeper mode, when a cluster grows to millions of partitions, ZooKeeper becomes a metadata bottleneck. If the active Controller broker crashes, electing a new controller and synchronizing partition metadata takes several minutes, blocking reads and writes. KRaft processes metadata changes as a streaming event log, scaling failovers down to milliseconds.

Real-World Analogy

- ZooKeeper Mode is like a company board of directors relying on a separate external auditing firm (ZooKeeper) to hold all their contracts and coordinate board elections. - KRaft Mode is the board managing files internally using their own secure cabinet. They elect their own officers directly using internal rules (Raft consensus) without having to hire outside consultants.

How It Works

Let's compare the metadata management patterns:

  1. ZooKeeper: ZooKeeper acts as the single source of truth. The elected Controller broker watches ZooKeeper paths, propagating state changes to brokers.
  2. KRaft: A selected group of controllers form a Raft quorum. Metadata changes are written to an internal topic (@metadata). All brokers read this topic sequentially.

Internal Architecture

The KRaft metadata architecture splits controller roles directly on the brokers:

  • Active Controller: The Raft leader node. It processes metadata writes (e.g. creating topics, updating partition maps) and appends them to the metadata log.
  • Standby Controllers: Raft followers. They replicate the metadata log from the leader and are ready to take over instantly if the active controller goes offline.

Visual Explanation

The diagram below shows active controller elections, metadata synchronization logs, and partition replication across broker nodes.

Practical Example

Below is a comparison of metadata properties files, showing how node roles are configured in KRaft versus ZooKeeper:

1. ZooKeeper configuration (server.properties):

2. KRaft configuration (kraft/server.properties):

Common Mistakes

  • Co-locating ZooKeeper and Kafka on the same disks: ZooKeeper is latency-sensitive. Sharing disks with high-throughput Kafka sequential writes causes ZooKeeper timeouts, triggering false broker offline events.
  • Configuring an even number of voting controllers: Raft requires a majority quorum. Running 4 controllers requires a majority of 3 to vote. If a network split divides the controllers 2 vs 2, the quorum is lost. Always run an odd number of controllers (3 or 5).

Quick Quiz

Q1: How does KRaft improve broker recovery times?

By keeping the metadata log active in memory on all brokers. Brokers do not need to pull partition records from ZooKeeper during boot.

Q2: Can a single node act as both broker and controller in KRaft?

Yes, using the configuration process.roles=broker,controller. This is common in smaller test environments, though separate nodes are preferred in production.

Scenario-Based Challenge

The Challenge: Surviving a Controller Network Partition

You run a 5-node KRaft controller cluster. A core switch failure isolates 2 controllers from the other 3. Senders are actively trying to write to partitions. Explain if the cluster survives, and what happens to metadata writes.

Solution Tip: The majority of 3 controllers remains connected. Since 3 constitutes a quorum of 5, the active controller can continue to commit metadata writes. Senders will continue normal writes. The isolated 2 nodes will sit idle until they rejoin the cluster.

Debugging Exercise

A developer attempts to format a new KRaft storage directory, but receives: MetaException: Cluster ID XXXXXX does not match the metadata log ID YYYYYY.

The Fix: Senders formatted the broker's storage folder with a different cluster ID than the one configured on the active controller. Delete the data log directories on the broker and run the storage formatting command using the correct cluster ID generated by the controller: 

# Generate cluster ID on controller
kafka-storage.sh random-bootstrap-id
# Format broker log directory
kafka-storage.sh format -t <cluster-id> -c config/kraft/server.properties

Interview Questions

1. What is ZooKeeper split-brain?

When a network partition creates two isolated ZooKeeper segments, each electing its own controller node. This results in conflicting partition state changes and data corruption.

2. What is the Raft log consensus mechanism?

A consensus protocol where a leader node appends logs, replicates them to follower nodes, and commits them only after a majority quorum acknowledges the write.

Production Considerations

In production, run three or five dedicated controller nodes separate from the broker nodes. This isolates the latency-sensitive consensus CPU/disk metrics from broker network workloads.