Observability & Performance Tuning
Micrometer + Prometheus + Grafana — Full Observability Setup
Injecting metrics counters and timers, scraping endpoints, and charting dashboards.
Interview: Full-stack observability architecture. Expect questions on Micrometer facade features, difference between metric types, scraping configurations, and metric tags cardinality limits.
Introduction
Spring Boot Actuator collects raw system statistics (like JVM thread state, memory allocation, HTTP request counts). However, Actuator cannot plot trends, track metrics over time, or trigger alert notifications during anomalies.
Modern application monitoring uses a three-tier observability stack: **Micrometer** gathers metrics programmatically inside the application code, **Prometheus** scrapes and stores these metrics in a time-series database, and **Grafana** connects to Prometheus to plot performance trends.
Why It Matters
Observability setups verify API performance (p99 latency) under peak load, track business metrics (such as completed orders), and identify memory leaks or CPU throttling trends.
Understanding Micrometer Metric Types
Micrometer provides three primary metric types for application tracking:
- Counter: A monotonically increasing value that tracks occurrences (e.g. total HTTP requests, completed orders). It never decreases.
- Gauge: A snapshot value representing current state (e.g. active database connection counts, thread counts). It can increase or decrease.
- Timer: Measures short-duration latencies (e.g. payment processing times, external API query speeds). It provides sample counts, cumulative times, and maximum latencies.
Practical Example
Let's write a Spring controller class that injects a custom Counter and Timer using a MeterRegistry:
Quick Quiz
Q1: Which metric type should you use to monitor the current thread utilization of your application's execution pool?
A) Counter
B) Gauge
C) Timer
D) Histogram
Answer: B — Gauges monitor current state values that can fluctuate up or down, making them suitable for thread counts, memory usage, or queue sizes.
Scenario-Based Challenge
Production Scenario:
You add a label email_address to your order counter to track user-specific purchases. After a deployment with heavy traffic, your Prometheus server crashes due to Out-Of-Memory errors. What caused this crash, and how do you resolve it?
To prevent metric storage crashes:
1. **Identify the Cause:** Adding high-cardinality tags (like email addresses, user IDs, or transaction hashes) generates thousands of unique label combinations. Prometheus stores each unique combination as a separate time-series key. This causes **Metrics Cardinality Explosion**, saturating Prometheus memory.
2. **Remove High Cardinality Labels:** Remove the email tag from the metric counter. Metrics should only use low-cardinality tags (e.g. status code, region, payment type).
3. **Use Logs for High Cardinality:** Use structured logging (JSON) and log aggregation tools (like Elasticsearch or Loki) to trace user-specific order statistics.
Interview Questions
1. What is the difference between Micrometer and the Prometheus registry dependency?
Micrometer acts as an abstract metrics gathering interface, similar to SLF4J for logging. It decouples metrics tracking from backend storage engines. The Prometheus registry is a plugin dependency that formats Micrometer metrics data into Prometheus scraping formats, allowing Prometheus databases to collect metrics via HTTP.
Production Considerations
Configure Micrometer to output percentiles (like p95 and p99 latencies) using SLAs. Calculate these server-side or enable them in client configurations using management.metrics.distribution.percentiles-roles.all=true.