ReviseAlgo Logo

Performance Tuning & Monitoring

Key Kafka Metrics Every Engineer Must Know

Lag, Under-Replicated Partitions, active controller count, and request handler idle time.

Introduction

Operating a distributed log like Apache Kafka at scale requires deep runtime visibility. Key Kafka Metrics represent the primary diagnostic indicators exposed via Java Management Extensions (JMX) that help developers and operations engineers monitor cluster health, detect resource constraints, and prevent data loss.

Why It Matters

Unlike traditional monolithic databases, Kafka failures manifest as consumer processing delays or partition imbalances rather than outright socket crashes. Without tracking metrics like Under-Replicated Partitions or Consumer Lag, clusters can silently degrade, leading to partition replication loss or unrecoverable client-side timeouts.

Real-World Analogy

Think of monitoring partition replicas like running a tandem bicycle fleet. The captain rider at the front (Partition Leader) steers and does most of the pedal work. The rear riders (Followers) must pedal in perfect lockstep to keep the bike moving smoothly. If one follower stops pedaling or falls behind (Under-Replicated Partition), the bike slows down, and if the captain falls off, the team lacks a synchronized rider to take over the steer handles, crashing the journey.

How It Works

Kafka exposes hundreds of metrics via Java Management Extensions (JMX). Production engineering teams filter these down to four primary categories:

  1. Partition Health Metrics: UnderReplicatedPartitions and OfflinePartitionsCount indicate replica synchronization problems and broker offline status.
  2. Consumer Progress Metrics: Consumer Lag measures the gap between the latest offset written to the broker and the last offset read by the consumer group.
  3. Broker Resource Metrics: RequestHandlerIdleRatio measures the fraction of time I/O threads spend idle (should be high, e.g. > 20%).
  4. Network and Socket Metrics: NetworkProcessorAvgIdlePercent monitors network thread capacity for processing client TCP socket requests.

Internal Architecture

A partition is considered under-replicated when a follower replica fails to catch up to the leader's log end offset within the time window specified by replica.lag.time.max.ms (default 30 seconds). If the follower fails to send a fetch request within this window, the leader removes it from the In-Sync Replicas (ISR) list and increments the broker's UnderReplicatedPartitions metric.

Visual Explanation

Below is an ASCII representation showing consumer lag as the difference between the Log End Offset and the Consumer Committed Offset: 

Topic Partition Log Segment:
[ Record 0 ] [ Record 1 ] [ Record 2 ] [ Record 3 ] [ Record 4 ] [ Record 5 ]
                                       ▲                         ▲
                                       │                         │
                            Committed Offset: 3          Log End Offset: 6
                                 ◄──────── Consumer Lag: 3 Records ──────►

Practical Example

Here is a Prometheus configuration sample illustrating standard YAML alerting rules for monitoring Under-Replicated Partitions and High Consumer Lag:

Common Mistakes

  • Only monitoring host OS metrics: Tracking only standard CPU, RAM, and disk capacity while completely ignoring JVM and JMX metrics.
  • Setting alerts on total bytes alone: Measuring throughput without correlating with network interface capacity.

Quick Quiz

Q1: What does a non-zero value for Under-Replicated Partitions indicate?

It indicates that one or more follower brokers are failing to fetch records from the partition leader within the configured time limit.

Q2: Why is monitoring Request Handler Idle Ratio crucial for cluster capacity planning?

Because it measures the processing head-room. When idle ratios approach 0%, request processing queues saturate, causing client side socket timeouts.

Scenario-Based Challenge

The Challenge: Pinpointing the root cause of a sudden Under-Replicated Partition alert

Your monitoring system alerts that Under-Replicated Partitions have spiked to 12. At the same time, network alerts show a small bandwidth drop. How do you identify which specific broker is unhealthy?

Solution Tip: Check the ActiveControllerCount metric on each broker. The broker hosting the controller should read 1, and all others should read 0.

Debugging Exercise

Your Prometheus graphs show that the broker's Request Handler Idle Ratio has dropped to 5%, causing clients to report frequent TimeoutException errors during writes.

The Fix: The brokers are struggling to keep up with the load. Increase num.network.threads and num.io.threads in server.properties to utilize more CPU cores.

Interview Questions

1. How do you distinguish between consumer group lag and partition replication lag?

Consumer group lag is client-side, showing the gap between consumer group progress and broker data logs. Partition replication lag is broker-side, showing the gap between follower replicas and partition leaders.

2. What metrics would you watch to ensure a Kafka producer is not losing records due to buffer limitations?

Monitor the producer's bufferpool-wait-time-ns and record-queue-time-max.

Production Considerations

Configure metrics collection with a scrape interval of 15 or 30 seconds.