Monitoring, Logging & Observability
Log Aggregation — Fluentd, Loki, and the ELK Stack
Collecting and querying stdout/stderr logs from all nodes in real-time.
Introduction
Metrics tell us that a system is broken, but logs explain why it is broken. When a container writes standard output (stdout) or standard error (stderr) lines, the container runtime redirects these lines into JSON log files stored on the host worker node's local disk.
Because containers are ephemeral and can be deleted during scaling or reschedule operations, we must aggregate logs off node boundaries. Centralized log aggregation architectures capture, enrich, and store container logs in searchable centralized databases.
Why It Matters
When a production pod crashes with a fatal exception (like an OutOfMemory or database timeout), Kubernetes restarts it. If the pod fails multiple times, it enters a CrashLoopBackOff. If logs are not streamed to a centralized indexer, debugging requires executing manual shell commands on nodes—which is blocked in secure environments.
Real-World Analogy
Think of log aggregation like **a retail bank's branch ledger system**. If customers interact at individual ATMs, their receipts are stored locally on paper rolls (node log files). If an ATM loses power or is vandalized (pod crash), those paper receipts could be lost forever. Instead, every action is immediately sent over network lines to the bank's central mainframe database (log aggregator). If an audit is required, bank security queries the central database for the customer's ID (querying Loki or Elasticsearch).
Logging Architecture Strategies
There are two primary paradigms for log aggregation in Kubernetes clusters:
- The ELK/EFK Stack (Elasticsearch, Fluentd, Kibana): Fluentd runs as a DaemonSet, reads container log files, parses logs into JSON, and indexes them in Elasticsearch. This is highly flexible, supporting full-text search index generation, but requires significant storage and CPU resources.
- The PLG Stack (Promtail, Loki, Grafana): Promtail acts as the local log shipper. Unlike Elasticsearch, Loki does not index the message body text. Instead, Loki only indexes the stream labels (e.g. namespace, app, container). This results in smaller index structures and lower infrastructure costs, though text querying is slower.
Practical Example
Loki uses LogQL (Log Query Language). Below are query configurations to filter and query logs in Grafana:
1. Searching Logs with LogQL
2. Promtail Scraper Configuration (DaemonSet ConfigMap snippet)
Here is how Promtail discovers and routes local container logs to the Loki endpoint:
Quick Quiz
Q1: Why is Loki called "like Prometheus, but for logs"?
A) It uses a pull-based model to query log files on client laptops.
B) It uses the same label-based metadata schema, matches PromQL syntax with LogQL, and does not create full-text search indexes, making it lightweight.
C) It only works on Prometheus nodes.
D) It converts all container log lines into numbers before storing them.
Answer: B — By indexing only label metadata instead of the message text content, Loki leverages the same labels as Prometheus, allowing seamless transition from a metric alert directly to matching pod logs in Grafana.
Q2: What is the primary benefit of deploying log shippers (like Fluent Bit or Promtail) as a DaemonSet?
A) It runs as a deployment in a single namespace to save memory resources.
B) It scales down to zero when the API traffic decreases.
C) It guarantees exactly one shipper pod runs per Kubernetes node, allowing it to mount host VM directories and parse all local container logs.
D) It bypasses kernel logging restrictions.
Answer: C — Logs are stored on the worker node filesystem. Running shippers as a DaemonSet ensures every worker node has an active shipper to read local logs and forward them to the log collector.
Scenario-Based Challenge
Production Scenario:
Your application writes verbose debug logs during a sales event. This triggers storage pressure on the worker nodes, causing them to enter DiskPressure state and evict pods. How do you resolve this at both the container and cluster architectural layers?
To prevent local disk exhaustion:
1. **Configure Log Rotation**: Configure the container runtime (e.g. containerd) on worker nodes to rotate logs, enforcing max size limits per file (e.g. 10MB) and retaining a maximum of 3 files per container.
2. **Adjust Log Levels**: Configure your application dynamically (via ConfigMaps or API endpoints) to restrict logs to INFO or WARN in production, avoiding DEBUG log storms.
3. **Non-blocking Buffer**: Ensure log shippers drop messages or throttle if the remote log database (e.g. Elasticsearch) is down, preventing log shipping queues from consuming host disk memory.
Interview Questions
1. Conceptual: What are the trade-offs between full-text search engines (Elasticsearch) and label-indexed logs (Loki)?
Elasticsearch indexes the entire log body, supporting sub-millisecond keyword searches across petabytes of logs. However, this indexing consumes massive RAM and storage. Loki indexes only labels, keeping storage costs minimal and ingestion fast. However, querying Loki for a specific unindexed string requires scanning raw log streams, which is CPU-intensive and slower.
2. Technical: How does a sidecar logging container differ from a DaemonSet shipper?
A DaemonSet shipper runs one pod per node, reading logs from all containers on that node (highly cost-effective). A sidecar logging container runs inside the application pod namespace (shared volumes), capturing custom file logs that aren't printed to stdout/stderr. Sidecars consume more resources because they duplicate agent processes per application pod.
Production Considerations
Ensure that your container applications write logs to standard output (stdout/stderr) rather than local files inside the container filesystem. Writing logs inside the container layer increases write amplification, risks disk leakage, and prevents shippers from reading them.