ReviseAlgo Logo

Kafka Internals — Storage & Replication

Leader Election — What Happens When a Broker Dies

Controller broker elections and partition leader elections.

Introduction

When a partition leader broker goes offline, Kafka must elect a new leader from the In-Sync Replicas (ISR) list to minimize downtime. The cluster Controller broker manages this process, updating metadata configurations and notifying active clients.

Why It Matters

Efficient leader election prevents cluster write bottlenecks. Developers must understand how elections are triggered and how metadata is propagated under KRaft consensus to avoid partition timeouts during broker node crashes.

Real-World Analogy

Think of partition leader election like electing a backup quarterback in sports. The team captain (Controller) monitors node liveness. If the active quarterback (Leader) gets injured, the captain selects a replacement from the list of players who have memorized the playbook (ISR list). The captain calls a huddle to notify all players (Metadata update) and resume play.

How It Works

The leader election process operates in these steps:

  1. Failure Detection: The Controller detects broker disconnection (ZooKeeper session timeout or KRaft heartbeat failure).
  2. ISR Evaluation: The Controller queries metadata for partitions where the dead broker was leader. It selects the first node in the partition's ISR list as the new leader.
  3. Metadata propagation: The Controller writes the new partition states to the metadata log.
  4. Client Notification: Senders and consumers receive metadata refresh updates, directing subsequent requests to the new leader broker node.

Internal Architecture

In ZooKeeper mode, electing a leader requires the controller to write metadata updates back to ZooKeeper paths, which then propagates to other brokers via LeaderAndIsrRequest RPCs. Under KRaft mode, leader elections are recorded directly as event records in the metadata log (the @metadata topic). Brokers read this stream, making metadata changes immediate and eliminating ZooKeeper write bottlenecks.

Visual Explanation

Below is an ASCII diagram illustrating the sequence of events during partition leader failover: 

Broker 1 (Leader) ──(Crashes)──X

[Controller Node] ──(Detects failure) ──> Selects Broker 2 (from ISR)
       │
       ├──(KRaft Quorum)──> Appends State Change to @metadata log
       │
       └──(Metadata Sync)──> notify Senders/Consumers to read/write to Broker 2

Practical Example

Below is a Java AdminClient snippet demonstrating how to programmatically describe topics and monitor partition leaders during testing:

Common Mistakes

  • Enabling Unclean Leader Election in critical systems: Setting unclean.leader.election.enable = true allows out-of-sync followers to become leaders on failure, causing silent data loss.
  • Failing to configure multiple controller nodes in production: Running only 1 KRaft controller node creates a single point of failure. If it crashes, partition leader elections are blocked.

Quick Quiz

Q1: What broker node is responsible for electing new partition leaders when a broker dies?

The active Controller broker.

Q2: What is unclean leader election?

The election of an out-of-sync replica (not in the ISR list) as leader when no ISR replicas are available, risking data loss.

Scenario-Based Challenge

The Challenge: Hard failover recovery time

You run a large cluster with 10,000 partitions. A broker crashes. Under ZooKeeper, leader elections take 2 minutes, during which clients block. How do you reduce recovery time?

Solution Tip: Migrate the cluster to KRaft mode. KRaft processes metadata updates as event logs, reducing failover recovery time to milliseconds.

Debugging Exercise

A broker crashes, and a partition becomes completely unavailable, throwing LeaderNotAvailableException continuously, even though multiple follower nodes are running.

The Fix: Unclean leader election is disabled (unclean.leader.election.enable=false), and all active followers fell out of the ISR list before the leader crashed. To recover, bring the original leader broker online, or voluntarily enable unclean leader election to allow a follower to take over.

Interview Questions

1. Explain what happens during unclean leader election.

When the leader crashes and no ISR replicas are active, a non-ISR replica (which is lagging behind) is elected leader. This enables partition writes to resume but drops any un-replicated records, corrupting consistency.

2. How does KRaft improve leader election performance over ZooKeeper?

ZooKeeper requires writing partition metadata changes to ZooKeeper paths, creating disk/network bottlenecks. KRaft processes state changes directly as streaming logs, propagating updates instantly.

Production Considerations

Disable unclean leader election for transactional applications. Monitor the active controller role closely.