ReviseAlgo Logo

Kafka Internals — Storage & Replication

Replication — Leaders, Followers, and ISR

In-Sync Replicas, replica lag, and high watermark sync.

Introduction

Kafka achieves fault tolerance by replicating partition logs across multiple broker nodes. Each partition has one Leader broker (handling client reads and writes) and zero or more Follower brokers that replicate data from the leader.

Why It Matters

Replication configurations define system durability. Senders must understand how the In-Sync Replicas (ISR) list is managed and how the High Watermark sync protects consumers from reading uncommitted data that could be rolled back on leader failures.

Real-World Analogy

Think of partition replication like a lead accountant (Leader) and backup clerks (Followers) maintaining corporate logbooks. The lead accountant logs new bills (writes) and reads updates. The backup clerks sit behind the lead accountant, copying lines as they are completed. If the backups fall behind (lag), they are marked out-of-sync. Clients only see entries that *all* active clerks have successfully copied.

How It Works

The replica replication cycle functions as follows:

  • Write Appending: Senders write records to the partition leader.
  • Fetch Requests: Follower replica brokers send periodic Fetch RPC requests to the leader to fetch new record pages.
  • High Watermark Advance: The leader tracks the offset reached by all active in-sync followers (the High Watermark). Consumers can only read up to this watermark.
  • ISR Membership: Followers that fail to send fetch requests within replica.lag.time.max.ms (default 30 seconds) are evicted from the ISR list.

Internal Architecture

The replication pipeline utilizes the Log End Offset (LEO) and High Watermark (HW) offsets. LEO is the next write offset on a broker, while HW is the largest offset copied by all active ISR members. HW ensures that even on leader failure, no read data is rolled back.

Visual Explanation

Below is a visual diagram of partition replication sync state: 

[Leader Broker 1]   ──> [Msg 10] ──> [Msg 11] ──> [Msg 12 (LEO)]
                                        ▲
                                        │ (High Watermark: Msg 11)
[Follower Broker 2] ──> [Msg 10] ──> [Msg 11 (LEO)] (In Sync)
[Follower Broker 3] ──> [Msg 10 (LEO)]               (Lagging, Out of ISR)

Practical Example

Below is a command line utility demonstrating how to describe a topic to view its partition replication state, leaders, and current ISR lists: 

# Describe topic partitions to check leaders and ISR nodes
bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic order-events

# Example output showing degraded partitions:
# Topic: order-events | Partition: 0 | Leader: 1 | Replicas: 1,2,3 | Isr: 1,2

Common Mistakes

  • Configuring replica.lag.time.max.ms too low: Setting it to 5 seconds in network-heavy clusters causes followers to flap in and out of the ISR list constantly, triggering metadata updates and degrading cluster throughput.
  • Letting ISR drop to 1 broker: If follower replicas crash, leaving only the leader in the ISR, data durability is lost, and a single disk failure will corrupt the partition.

Quick Quiz

Q1: What offset defines the boundary for records that are considered committed and safe to read by consumers?

The High Watermark.

Q2: After what timeout period is a lagging follower evicted from the ISR list?

30 seconds by default (configured by replica.lag.time.max.ms).

Scenario-Based Challenge

The Challenge: Sizing replication for geographical redundancy

You deploy a Kafka cluster across 3 AWS Availability Zones. You want to ensure that a complete zone outage does not lose data or block writes. How do you configure the topic?

Solution Tip: Set the replication factor to 3, distribute brokers across the three zones using rack-aware configurations (broker.rack), configure min.insync.replicas=2, and set acks=all on producers.

Debugging Exercise

You observe that a consumer lag monitor shows 0 lag, but downstream services report missing messages. Senders confirm writes succeeded, and logs show no client errors.

The Fix: Senders are using acks=all but min.insync.replicas is configured to 1 (the default). Followers fell out of the ISR, and the leader acknowledged writes before syncing to disk. The leader crashed, and a stale follower was elected leader, rolling back the writes. Fix this by setting min.insync.replicas=2.

Interview Questions

1. What is the difference between log end offset (LEO) and high watermark (HW)?

LEO is the offset of the next record to be written to a partition replica's local log. HW is the largest offset that has been successfully replicated across all active in-sync replicas (ISR) for the partition.

2. What happens if a follower is evicted from the ISR?

It is no longer eligible to be elected partition leader on failure. It can rejoin the ISR only after fetching all missing records from the leader.

Production Considerations

Always configure min.insync.replicas=2 on topics with a replication factor of 3 to guarantee durability during rolling updates.