ReviseAlgo Logo

Kubernetes Architecture

How Kubernetes Reconciliation Works — The Control Loop

Continuous convergence toward desired cluster states.

Interview: Reconciliation is the fundamental design pattern of Kubernetes. Be prepared to explain the Observe-Compare-Act loop, how self-healing processes (like rescheduling on node failure or CrashLoopBackOff exponential backoffs) resolve discrepancies, and why this model makes Kubernetes exceptionally resilient.

Last Updated: June 15, 2026 • 15 min read

Introduction

Reconciliation is the fundamental architectural pattern that makes Kubernetes so resilient. Instead of running a sequence of setup commands once, Kubernetes runs continuous control loops to guarantee the cluster matches your desired configuration.

Why It Matters

In classic environments, systems suffer from configuration drift—files get edited manually, servers go down, and systems break silently. K8s reconciliation loops identify and self-heal these drifts within seconds.

Real-World Analogy

Reconciliation is exactly like a thermostat in your home. You set the thermostat to 72°F (desired state). The thermostat continuously reads the room temperature (observe). If the room drops to 68°F (compare), the thermostat turns on the furnace (act) to heat the house back up to 72°F.

How It Works

Every controller in the control plane executes a loop containing three distinct phases:

  1. Observe: Query the API Server to inspect the current active state of the cluster resource objects (the status field).
  2. Compare: Evaluate the observed state against the target configuration stored in the object specification (the spec field).
  3. Act: If a discrepancy is found, execute API commands to resolve the difference and bring the current state closer to the desired state.

Internal Architecture

Reconciliation loops are managed by individual controllers inside the kube-controller-manager. Each controller watches a specific kind of resource (like Deployments, ReplicaSets, or Services) via API Server watch streams, avoiding expensive polling.

Visual Explanation

Loading diagram…

Practical Example

Let's trace how the Deployment controller behaves when we scale a deployment using kubectl:

Common Mistakes

1. Loop Cascading Collisions: Writing two different controllers that watch and write to the same field, causing them to fight each other in an infinite cycle.
2. Ignoring CrashLoopBackOff: Believing a failing container restarts infinitely at high speed. The kubelet exponential backoff delay scales up to 5 minutes to prevent CPU starvation.

Quick Quiz

Q1: What are the three phases of the reconciliation control loop?

A) Build, Push, Run

B) Observe, Compare, Act

C) Read, Write, Delete

Answer: B — The loop continuously observes current status, compares it to spec, and acts on differences.

Scenario-Based Challenge

Scenario:

A rogue developer SSHs directly into a worker node and runs raw Docker CLI commands to kill an application container. Describe how the reconciliation loop reacts to this event.

View Solution

1. The local node agent (kubelet) observes that the container process has terminated.
2. kubelet compares this with the declared PodSpec (which requires container to be running).
3. kubelet immediately invokes the container runtime to pull the image (if missing) and restart the container process, returning the pod status back to running, defeating the manual deletion.

Debugging Exercise

Failure Case:

An application Pod status shows CrashLoopBackOff.

How do you debug this using kubectl?

View Solution
CrashLoopBackOff means the container starts, but exits immediately, triggering the backoff delay.
1. Fetch the logs of the container to read errors (like missing DB configs):
2. If logs are empty, print the container exit code and details:
Look for "Last State: Terminated" and "Reason: ContainerCannotRun" or exit code (e.g. 137 represents OOM Out Of Memory).

Interview Questions

1. What is CrashLoopBackOff and why does it occur?

It is a safety state where kubelet delays restarting a failing container. It occurs when a container crashes repeatedly on boot, and K8s throttles restart speed exponentially to protect system resources.

2. Describe the reconciliation loop behavior when a worker node goes offline.

The Node Controller notices missing heartbeats and marks the node NotReady. The Deployment controller compares active replicas with target specs, realizes a shortfall, and requests Pod launches on healthy nodes.

Production Considerations

Configure correct liveness and readiness probes in your specs. Probes prevent reconciliation loops from routing client traffic to containers that are still loading resources or initialization scripts.