ReviseAlgo Logo

Kafka Internals — Storage & Replication

Under-Replicated Partitions — Detection and Recovery

Monitoring JMX metrics and rebalancing partition allocations.

Introduction

A partition is Under-Replicated when one or more of its follower replicas fall behind the leader or go offline, reducing the partition's active replica count below its configured replication factor.

Why It Matters

Under-replicated partitions are a primary indicator of cluster degradation. Senders must monitor JMX replication metrics to detect resource exhaustion, disk failure, or network bottlenecks before they trigger system unavailability.

Real-World Analogy

Think of an under-replicated partition like a delivery van missing its spare tire. The van can still drive and deliver packages (writes continue), but if the active tire punctures (broker crashes), the van is stuck, and deliveries stop. Under-replication means you are running without safety backups.

How It Works

The partition degradation cycle follows these stages:

  1. Monitoring: The broker tracks the lag time of followers.
  2. Out of Sync: If a follower replica fails to query the leader within replica.lag.time.max.ms, it is removed from the ISR list.
  3. Alerting: The metric UnderReplicatedPartitions is incremented on the broker node.
  4. Recovery: Once the follower rejoins the network, it pulls logs from the leader's LEO. Once caught up, it is re-added to the ISR list.

Internal Architecture

Under-replication is tracked broker-side via JMX metrics under the name kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions. A value greater than 0 indicates replica degradation.

Visual Explanation

Below is a diagram of replication status transitions: 

[State: Healthy]           ISR = [1, 2, 3]  Replication Factor = 3 (Safe)
                                 │
                         (Node 3 crashes)
                                 ▼
[State: Under-Replicated]  ISR = [1, 2]     Replication Factor = 3 (Degraded)
                                 │
                         (Node 3 recovers & syncs)
                                 ▼
[State: Healthy]           ISR = [1, 2, 3]  Replication Factor = 3 (Safe)

Practical Example

Below are CLI scripts demonstrating how to query under-replicated partitions across a cluster: 

# Query the cluster to find all under-replicated partitions
bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --under-replicated-partitions

# Example output showing degraded partitions:
# Topic: telemetry | Partition: 2 | Leader: 1 | Replicas: 1,2,3 | Isr: 1,2

Common Mistakes

  • Ignoring Under-Replicated Partition Alerts: Treating under-replication as a minor warning. If a second node crashes while partitions are under-replicated, partitions will go offline.
  • Forcing broker restarts during under-replication recovery: Interrupting a syncing broker forces it to re-initialize log recovery validation, prolonging recovery time.

Quick Quiz

Q1: What JMX metric should you monitor to detect under-replicated partitions?

UnderReplicatedPartitions under ReplicaManager.

Q2: Does an under-replicated partition block client writes by default?

No. Writes continue normally as long as the active ISR count is greater than or equal to min.insync.replicas.

Scenario-Based Challenge

The Challenge: Recovering from disk failure

Senders alert on under-replicated partitions. Describe shows broker 3 disk crashed. A new disk is mounted. How do you trigger partition sync without restarting the broker?

Solution Tip: Use the kafka-reassign-partitions.sh tool to migrate or reassign partitions off the degraded node, or configure the node to rebuild replicas on the new disk.

Debugging Exercise

You observe that partitions become under-replicated every day at 2:00 AM, but recover automatically by 2:30 AM.

The Fix: Check cron workloads. A backup script or batch import job is saturating network I/O or CPU, causing replication fetch requests to timeout. Optimize batch workloads or rate-limit ingestion rates.

Interview Questions

1. What causes a partition to become under-replicated?

Follower broker crashes, network partitions preventing replicas from contacting the leader, or disk I/O bottlenecks causing the replication thread to lag.

2. What is the recovery process for an under-replicated partition?

Once the lagging node recovers and resumes its fetch requests, it replicates missing segments sequentially until its LEO matches the leader's watermark.

Production Considerations

Set up high-severity alerts on Prometheus/Grafana when UnderReplicatedPartitions is greater than 0 for more than 5 minutes.