ReviseAlgo Logo

Performance Tuning & Monitoring

Monitoring Kafka with Prometheus and Grafana

Setting up exporters and building dashboards for production visibility.

Introduction

While key metrics exist inside brokers and clients as raw statistics, they must be formatted, aggregated, and visualized for production use. Prometheus and Grafana represent the industry standard stack for pulling JMX metrics, configuring alerts, and rendering real-time dashboards.

Why It Matters

Without active metric visualization, system administrators cannot spot trends (such as gradual memory leaks or slowly mounting consumer lag) before they trigger cascading failures. Setting up Grafana charts provides real-time visibility, and Prometheus Alertmanager integration guarantees immediate notification of critical events.

Real-World Analogy

Think of monitoring like an airport control tower. The planes (brokers) publish telemetry data. Air traffic controllers do not read raw telemetry logs in text format. They use a radar screen (Grafana dashboard) that plots flights visually, and automated sound warnings (Prometheus Alertmanager) that sound an alarm immediately if two paths cross.

How It Works

Metrics collection and visualization involves the following stages:

  1. JMX Mapping Exporter: A Java agent (jmx_prometheus_javaagent.jar) is injected into the Kafka process to scrape MBeans and export them over an HTTP page.
  2. Prometheus Scrape: The Prometheus server periodically calls the endpoint (e.g. http://broker-ip:7071/metrics) to ingest metrics.
  3. Grafana Visualization: Grafana connects to Prometheus as a data source, rendering charts and trends based on PromQL queries.

Internal Architecture

JMX metrics are hierarchically nested and difficult to read out of the box. The exporter uses a configuration file containing regular expression rules. These rules intercept raw JMX MBean structures (e.g. kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec) and rename them into flat, clean metrics formats (e.g. kafka_server_brokertopicmetrics_bytesinpersec).

Visual Explanation

Below is an ASCII representation showing the pipeline from broker JMX MBeans to Grafana graphs: 

[Kafka JMX MBeans] ──> (Prometheus Java Agent) ──> http://broker:7071/metrics
                                                           │
                                                           ▼ (Scraped)
  [Grafana Chart]  <── (PromQL Queries)       <── [Prometheus DB]

Practical Example

Here is a complete prometheus-jmx-config.yaml file configuring the exporter rules to map Kafka broker MBeans correctly:

Common Mistakes

  • Using a wild-card exporter rule: Exposing all JMX MBeans without filtering. This creates millions of unnecessary metrics (e.g. JVM thread details), bloating the Prometheus database and causing dashboard queries to time out.
  • Exposing metrics on public web networks: Leaving the Prometheus JMX port (7071) exposed without firewall rules, allowing external users to inspect cluster metadata.

Quick Quiz

Q1: What is the role of the JMX Prometheus Java Agent in a Kafka deployment?

It runs inside the broker JVM process, reads JMX metrics locally, and translates them into an HTTP metrics page readable by Prometheus.

Q2: How does PromQL calculate the rate of incoming messages on a topic?

Using the rate() function (e.g. rate(kafka_server_brokertopicmetrics_messagesinpersec_total[5m])).

Scenario-Based Challenge

The Challenge: Writing a PromQL expression to alert when consumer lag is rising

You need to create a Prometheus alert that triggers only when consumer lag increases continuously over a 10-minute window, preventing false positive alerts on temporary spikes. How do you write this PromQL expression?

Solution Tip: Use the deriv() function to calculate the rate of change of the consumer lag over time: deriv(kafka_consumergroup_lag[10m]) > 0. This triggers only when the derivative is positive (meaning lag is growing).

Debugging Exercise

Your Grafana charts display No Data for Kafka metrics, and checking the Prometheus dashboard shows that target broker endpoints are marked as DOWN.

The Fix: Check the Java agent startup parameters. Ensure the broker env variables include -javaagent:/path/to/jmx_prometheus_javaagent.jar=7071:/path/to/config.yaml. Verify that port 7071 is open on the broker host, and the Prometheus prometheus.yml scrape config targets the correct IP.

Interview Questions

1. How do you monitor consumer lag without running JMX queries on the clients?

By running tools like Burrow or using the Confluent Control Center, which read offset commits directly from the internal consumer offset topic __consumer_offsets.

2. What is the difference between push-based and pull-based metrics monitoring?

Prometheus uses a pull-based model (scraping HTTP metrics from target agents), whereas platforms like Datadog or StatsD use a push-based model (sending events to a central collector).

Production Considerations

Implement Thanos or Cortex in front of Prometheus to achieve long-term storage retention and high availability for metrics databases.