Monitoring, Logging & Observability
Monitoring Kubernetes with Prometheus and Grafana
Deploying kube-prometheus-stack and visualizing control plane metrics.
Introduction
Observability is critical to running production workloads in Kubernetes. While traditional servers can be monitored by logging onto a single VM and reading system resource usage, Kubernetes runs dynamically rescheduled containers across a distributed pool of worker nodes.
Prometheus is an open-source, metrics-based monitoring system designed specifically for cloud-native applications. It uses a pull-based architecture to scrape HTTP endpoints exposing numerical metrics. Grafana is the visualization dashboard layer that connects to Prometheus, allowing developers to build interactive panels and charts representing real-time system health.
Why It Matters
Without active metric collection, resource saturation or container restart loops go unnoticed until they lead to cascading service failures. Having Prometheus allows teams to track the Four Golden Signals (Latency, Traffic, Errors, and Saturation) across every microservice and node within the cluster boundary.
Real-World Analogy
Think of monitoring a cluster like **an intensive care unit (ICU) telemetry system**. Instead of doctors manually taking a patient's temperature every few hours (manual log checks), patients are connected to constant vital monitors (Prometheus scrapers). These monitors pull heart rate, blood pressure, and oxygen data every few seconds (scraping metrics endpoints). If a vital drops below a threshold, a red light flashes and a chime rings in the staff room (Grafana alert dashboard).
The Kube-Prometheus-Stack
In Kubernetes, deploying raw Prometheus and Grafana instances is complex. Production setups rely on the kube-prometheus-stack, a Helm chart providing an operator-driven ecosystem:
- Prometheus Operator: Manages the lifecycle of Prometheus instances using Custom Resource Definitions (CRDs) like
ServiceMonitorandPodMonitor. - Kube-State-Metrics: A service that listens to the Kubernetes API server and generates metrics about the state of resources (e.g. how many pods are running, pending, or in crash loops).
- Node Exporter: Runs as a DaemonSet on every worker node to collect hardware-level metrics (CPU, disk space, network throughput).
- Grafana: Connects to Prometheus as a data source, pre-configured with dashboards visualizing cluster health.
Practical Example
To monitor a custom microservice, we define a ServiceMonitor. This CRD instructs Prometheus to discover and scrape the microservice's metric endpoint:
Key PromQL Queries for Troubleshooting
Prometheus uses PromQL (Prometheus Query Language) to query metrics. Here are essential queries for production operators:
- Find Pod CPU Usage Rate over 5 minutes:
sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (pod) - Find Memory Usage Percentage per Node:
(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100 - Identify Pods in a Crash Loop:
increase(kube_pod_container_status_restarts_total[30m]) > 5
Quick Quiz
Q1: Why does Prometheus use a pull-based metrics model instead of having applications push metrics to it?
A) Pushing metrics is blocked by Kubernetes NetworkPolicies by default.
B) A pull-based model allows Prometheus to control the scraper load, detect targets going offline, and simplifies client library setups.
C) Pulling metrics automatically increases disk speeds.
D) The Kubernetes API server only allows inbound HTTP pull traffic.
Answer: B — By pulling metrics at configured intervals, Prometheus controls its resource ingestion rate, avoids getting overwhelmed by spikes, and knows immediately if a scraping target is down (checking endpoint state).
Q2: What is the main purpose of a ServiceMonitor in an operator-managed Prometheus deployment?
A) It displays visual panels directly on the user's dashboard interface.
B) It restarts target pods if they fail their readiness check.
C) It declaratively tells the Prometheus controller how to locate, authenticate with, and scrape metrics from a Kubernetes Service.
D) It encrypts persistent volumes storing metric data.
Answer: C — ServiceMonitors are CRDs that the Prometheus Operator processes. It dynamically updates Prometheus scrape configurations based on changes to matching Services, eliminating manual configuration editing.
Scenario-Based Challenge
Production Scenario:
You deploy a new ServiceMonitor to scrape a billing API, but the service metrics do not appear in Prometheus or Grafana. Running kubectl get servicemonitors shows your monitor exists. How do you troubleshoot this target discovery failure?
Follow these debugging steps:
1. **Verify Labels**: The ServiceMonitor must have a label that matches the Prometheus instance's ServiceMonitorSelector (e.g. release: prometheus-stack). If missing, Prometheus ignores it.
2. **Verify Service Selectors**: Ensure your Kubernetes Service matches the pod labels correctly, and that the ServiceMonitor's endpoints.port references the exact port name (not port number) defined in the Service manifest.
3. **Check Prometheus Targets UI**: Access the Prometheus expression browser, go to **Status -> Targets**, and check if your service is listed. If listed but red, verify network connectivity or endpoint paths (e.g. security tokens blocking scraping requests).
Interview Questions
1. Conceptual: How does Prometheus handle metrics storage, and what is cardinality?
Prometheus stores metrics in a time-series database (TSDB) using an append-only format on local disk. Cardinality refers to the total number of unique time-series data combinations, calculated by multiplying labels and their unique value footprints. High-cardinality (e.g., adding user IDs or transaction hashes as labels) balloons memory usage and can crash the Prometheus server.
2. Technical: What is the difference between kube-state-metrics and node-exporter?
kube-state-metrics queries the Kubernetes API server to generate resource-state metrics (deployment replicas, pod phases, node statuses). node-exporter runs as a DaemonSet to query node kernel metrics (OS metrics, disk space, IO, socket states, CPU/Memory hardware usage).
Production Considerations
Do not use local temporary storage for production Prometheus instances. Configure persistent volume claims (PVC) with high IOPS, or integrate Prometheus with long-term metrics storage layers (Thanos, Cortex, or Amazon Managed Prometheus) for high availability and historical analysis.