Monitoring, Logging & Observability
Distributed Tracing with Jaeger and OpenTelemetry
Tracing inter-service request flows through pod boundaries.
Introduction
In monolithic architectures, tracking performance issues is straightforward as all function calls happen in a single thread or memory space. In distributed Kubernetes systems, a single client request can trigger a cascading web of HTTP/gRPC requests, database queries, and message queue events across multiple independent microservice pods.
Distributed Tracing maps the flow of requests as they traverse physical and network boundaries. By using OpenTelemetry standards to instrument applications and Jaeger to query collected traces, developers can visualize request lifecycles.
Why It Matters
If a client checkout request fails or takes 3 seconds, logs and metrics might show database latency or 500 errors across several services. However, they cannot correlate these events to a single request. Tracing resolves this by linking spans together into a single trace graph.
Real-World Analogy
Think of distributed tracing like **a baggage tracking system at an international airport**. When you check in, your bag is tagged with a unique barcode (Trace ID). As your bag moves along conveyor belts, through security scanners, and onto transfer flights, handlers scan the barcode, recording checkpoint entry and exit timestamps (Span IDs). If a bag is delayed, the airline looks up the barcode to see exactly where the bag spent the most time.
Core Tracing Terminology
Distributed tracing relies on specific data concepts governed by the CNCF OpenTelemetry standard:
- Span: The basic building block. A span represents a single unit of work (e.g. an HTTP request, a database query, or a function execution) with start and end timestamps.
- Trace: A collection of spans that form a directed acyclic graph (DAG), representing the complete lifecycle of a request from start to finish.
- Trace Context: The metadata passed across network calls (typically via HTTP headers) containing the Trace ID and Parent Span ID, linking downstream calls to the parent trace.
- OpenTelemetry Collector: A high-performance proxy that receives, processes, and exports telemetry data (metrics, logs, traces) to backends like Jaeger or Zipkin.
Practical Example
To trace requests across services, we must propagate HTTP headers using the W3C Trace Context standard. Here is how header context propagation looks conceptually in custom headers:
Deploying an OpenTelemetry Collector in Kubernetes
Here is an OpenTelemetry Collector configuration manifest to receive traces via OTLP (OpenTelemetry Protocol) and export them to Jaeger:
Quick Quiz
Q1: What happens if a microservice in a request path receives an HTTP call but fails to extract and pass the traceparent header to the next downstream service?
A) The downstream service will throw a compilation error.
B) The trace path breaks, and the downstream service will generate a new Trace ID, splitting a single request path into two disconnected trace records.
C) The Kubernetes service mesh will automatically shut down the container.
D) The request will time out instantly.
Answer: B — If context propagation is broken, trace continuity is lost. Downstream spans cannot link back to the parent trace, rendering distributed analysis ineffective.
Q2: Why do production environments apply tail-based sampling instead of head-based sampling for distributed tracing?
A) Head-based sampling is slower because it queries database indexes.
B) Tail-based sampling allows the tracing framework to evaluate the entire request path (e.g. check if it errored or was slow) before deciding to save or discard the trace, saving storage.
C) Tail-based sampling does not require OpenTelemetry Collectors.
D) Kubernetes only supports tail-based network parsing.
Answer: B — Head-based sampling decides to sample a request at the very beginning (randomly). Tail-based sampling stores spans in memory until the request completes, ensuring that 100% of errors or high-latency events are kept, while discarding redundant successful traces.
Scenario-Based Challenge
Production Scenario:
Your company runs a Java frontend service that communicates with a Node.js backend using a Kafka message queue. After deploying Jaeger, you notice that messages routed through Kafka show as two separate traces rather than a single continuous path. How do you fix this?
View Solution
Since Kafka is an asynchronous message broker rather than a direct synchronous HTTP call, HTTP headers cannot be used.
1. **Inject Context into Kafka Headers**: Before publishing the message to the Kafka topic, configure the Java producer to inject the active Trace Context into the Kafka record's metadata headers (e.g. traceparent key).
2. **Extract Context in Consumer**: Configure the Node.js consumer to extract the traceparent metadata header from the incoming message payload, and pass it to the OpenTelemetry SDK to start the consuming span as a child of the injected context.
Interview Questions
1. Conceptual: What is the role of OpenTelemetry in the modern observability landscape?
OpenTelemetry (OTel) is a standardized vendor-neutral framework that provides a unified set of APIs, SDKs, and tooling to generate and export traces, metrics, and logs. It eliminates vendor lock-in, allowing developers to instrument applications once and switch telemetry backends (e.g. Jaeger, Datadog, Honeycomb) by changing configurations.
2. Technical: How does auto-instrumentation differ from manual instrumentation?
Auto-instrumentation injects tracing hooks dynamically at runtime (using Java agents, monkey-patching in Node, or middleware wrappers), capturing framework-level network calls (Express, Spring, HTTP requests) without code changes. Manual instrumentation requires writing explicit code to start/end spans and record custom attributes, providing deeper business-logic visibility.
Production Considerations
Tracing generates vast quantities of data. In production clusters, configure sampling rates (e.g. 5-10% of HTTP requests) in the OpenTelemetry Collector to prevent tracing pipelines from overwhelming network bandwidth and inflating storage costs.