Cluster Autoscaler — Adding and Removing Nodes

Introduction

Pod-level scaling (HPA and VPA) is bounded by the total resource capacity of your physical or virtual nodes. If your cluster runs out of compute nodes, new pods cannot be scheduled. The Cluster Autoscaler resolves this by automatically adding or removing nodes in the cluster's node groups based on workload demand.

It communicates with cloud providers (such as AWS, GCP, or Azure) to scale VM Auto Scaling Groups up when there are resource shortages, and scale them down when nodes are under-utilized.

Why It Matters

Without a Cluster Autoscaler, pod scaling is hard-capped. A sudden scale-out request will leave pods stuck in a Pending state, causing application performance degradation or outages. Alternatively, during low traffic periods, you pay for idle virtual machines, inflating infrastructure costs.

Real-World Analogy

Think of a hotel during a holiday rush. If all rooms are fully booked and there is a queue of guests waiting in the lobby (Pending pods), the hotel manager opens up a secondary wing of rooms (adding nodes). Once the holiday ends and rooms sit empty, the manager closes down the wing to save on heating, electricity, and housekeeping costs (removing nodes).

How It Works

The Cluster Autoscaler runs a continuous monitoring loop, evaluating two primary conditions:

• Scale-Up Condition: If any pods are in a Pending state because the scheduler could not find a node with enough CPU, memory, ports, or affinity rules (marked with a FailedScheduling event), Cluster Autoscaler triggers the cloud provider's API to launch new nodes in the appropriate node group.
• Scale-Down Condition: If a node's resource utilization (calculated as the sum of its running pods' Resource Requests divided by node capacity) is below a target threshold (default 50%) for a sustained period (typically 10 minutes), CA simulates whether the pods on that node can be rescheduled onto other active nodes. If yes, it drains the node and deletes the VM.

Internal Architecture

Unlike HPA/VPA, Cluster Autoscaler does not monitor CPU/Memory metrics. Instead, it simulates the Kube-Scheduler's routing algorithm. When pods are pending, CA determines which Node Group (or Auto Scaling Group) can satisfy the pods' resource, nodeSelector, toleration, and affinity requirements. It then makes an API call to the host cloud provider (e.g. AWS ASG, Azure VMSS, or GCP MIG) to increment the group size.

Visual Explanation

Practical Example

Here is a configuration snippet showing typical startup arguments for the Cluster Autoscaler deployment running in a cloud environment:

Common Mistakes

• Cloud IAM Permission Failures: Failing to attach correct permissions (e.g. AWS IAM permissions for AutoScaling and EC2) to the service account running the CA. This blocks CA from editing the VM scaling groups.
• Missing Pod Disruption Budgets (PDBs): If PDBs are not defined, scale-down evictions can terminate critical replicas, violating SLA requirements during scaling events.
• Blocks on Scale-Down: Pods with local ephemeral storage (like emptyDir) or pods without controllers (naked pods) will prevent Cluster Autoscaler from terminating a node to avoid data loss, unless explicit command-line flag overrides are defined.

Quick Quiz

Scenario-Based Challenge

When CA attempts to drain a node, the eviction API will respect this PDB. If evicting a pod would bring the active replicas below 2, the eviction is blocked, forcing the autoscaler to wait or reschedule other pods first.

Debugging Exercise

View Debugging Steps & Fix

Reason: The Cluster Autoscaler pod does not have permission to modify the cloud provider's Auto Scaling Group. In this case, the IAM role mapped to the Kubernetes Service Account lacks the autoscaling:SetDesiredCapacity permission.

The Fix:
1. Locate the IAM Policy attached to the cluster-autoscaler role.
2. Add the required permissions (e.g. AWS Auto Scaling permissions):

3. Verify the service account matches the role annotations and restart the autoscaler pod.

Interview Questions

Production Considerations

Use **PriorityClass ballooning** to pre-warm nodes. This involves running low-priority pause pods that consume resources but are evicted instantly when normal-priority application pods need to start, enabling near-instantaneous scaling. Always set limits on node group size (min and max) to protect against unexpected cloud resource costs.