Autoscaling & Resource Management
Resource Requests and Limits — CPU and Memory
Avoiding OOM kills and configuring CPU shares correctly.
Introduction
Kubernetes manages workloads across a cluster of nodes. To schedule and run these containers efficiently, we must specify their resource footprint. This is defined using Resource Requests and Resource Limits for CPU and memory.
Requests represent the minimum guaranteed amount of a resource that a container needs to run, which the scheduler uses to place the pod. Limits represent the hard ceiling that a container is not allowed to exceed during execution.
Why It Matters
Failing to define requests and limits creates a "noisy neighbor" environment. A single misbehaved pod with a memory leak or an infinite CPU loop can consume all available resources on a worker node, starving other critical workloads, leading to node instability, and triggering cluster-wide crashes.
Real-World Analogy
Imagine renting a shared co-working office. Your Request is the size of the desk you are guaranteed to have when you show up (e.g., a 1.5-meter desk to place your laptop and monitor). Your Limit is the maximum desk space you can spread your folders and coffee mugs across before security stops you from blocking the shared corridor and infringing on other workers' spaces.
How It Works
When a Pod is submitted, the Kubernetes Scheduler filters nodes based on whether their Allocatable capacity is greater than or equal to the sum of the Resource Requests of all pods already running on that node plus the new pod.
Based on the specified requests and limits, Kubernetes assigns one of three Quality of Service (QoS) classes to the pod:
- Guaranteed: CPU and memory requests are set and equal to their respective limits for every container in the pod.
- Burstable: The pod does not meet the Guaranteed criteria, but at least one container has a CPU or memory request set.
- BestEffort: No containers in the pod have any CPU or memory requests or limits configured.
Internal Architecture
Kubernetes relies on Linux control groups (cgroups) to enforce these boundaries at the operating system level:
- CPU Requests (cgroups:
cpu.shares): Translates to relative shares of CPU time. If nodes are fully loaded, CPU is distributed proportionally based on these shares. It is a compressible resource; containers are throttled, not terminated. - CPU Limits (cgroups:
cpu.cfs_quota_us&cpu.cfs_period_us): Enforces a hard ceiling using Completely Fair Scheduler (CFS) bandwidth control. If a container uses more than its quota in a given period, the kernel throttles it. - Memory Requests & Limits (cgroups:
memory.limit_in_bytes/memory.max): Memory is an incompressible resource. If a container attempts to allocate memory beyond its limit, the host kernel triggers the Out-Of-Memory (OOM) killer, sending a SIGKILL (Exit Code 137) to terminate the container process.
Visual Explanation
Practical Example
Here is a standard Deployment manifest configured with resource requests and limits for a backend microservice:
Common Mistakes
- No Resource Configuration: Leaving requests and limits empty. This defaults the pod to
BestEffort, meaning it is the first to be evicted during resource contention. - Incorrect JVM Heap Config: Setting JVM maximum heap size (
-Xmx) equal to or larger than the container's memory limit. The JVM consumes overhead outside the heap, causing the kernel to OOM kill the pod before a garbage collection is triggered. Keep-Xmxat 70-80% of the container limit. - Over-throttling CPU: Setting CPU limits too low (e.g.,
50m) for multi-threaded applications, causing micro-throttling that results in massive latency spikes despite low average CPU usage.
Quick Quiz
Q1: Which resource is considered "compressible" and what happens to a container when it exceeds its limit for that resource?
A) Memory; the container is immediately terminated with Exit Code 137.
B) CPU; the container processes are throttled (slowed down) but not killed.
C) CPU; the container is evicted and rescheduled on a larger node.
D) Ephemeral Storage; the container is paused indefinitely.
Answer: B — CPU is a compressible resource. When limits are exceeded, cgroups enforce limits by throttling CPU cycles using CFS quota windows, slowing down the workload without terminating it.
Q2: How does the Kubernetes Scheduler determine if a node has enough capacity to run a new Pod?
A) By evaluating the actual real-time CPU and Memory usage of the node.
B) By evaluating the Resource Limits of all pods currently running on that node.
C) By evaluating the Resource Requests of all pods currently scheduled on that node.
D) By checking the network bandwidth latency of the target node.
Answer: C — The scheduler makes scheduling decisions based on the sum of Resource Requests of all pods scheduled to a node, not their limits or real-time usage.
Scenario-Based Challenge
Production Scenario:
Your Java backend Pod runs fine under normal load. However, during high-volume batch processing, it restarts repeatedly. Running kubectl describe pod reveals Last State: Terminated with Reason: OOMKilled and Exit Code: 137. The pod's memory limit is 1Gi, and the JVM options include -Xmx1024m. How do you resolve this?
The OOMKilled state indicates that the container memory consumption exceeded the cgroup memory limit of 1Gi (1024Mi).
The root cause is setting the JVM heap limit (-Xmx1024m) equal to the container limit (1Gi). The JVM requires extra memory for metaspace, thread stacks, garbage collection tables, and native code libraries, pushing total container memory usage past 1.2Gi.
Solution:
1. Increase the container memory limit to 1.5Gi while keeping requests at 1Gi, or
2. Decrease the JVM heap size using -Xmx750m (roughly 70-75% of the container's 1Gi limit) to leave enough overhead space for the JVM runtime.
Debugging Exercise
Broken Configuration:
The following deployment configuration is rejected by the Kubernetes API server when attempting to apply:
Reason: In Kubernetes, a resource request cannot be greater than its limit. This makes logical sense as a container cannot guarantee a reservation that is higher than its maximum allocation ceiling. Here, requests (512Mi, 500m) exceed limits (256Mi, 250m).
The Fix: Correct the requests to be less than or equal to limits:
Interview Questions
1. Conceptual: What are the differences between compressible and incompressible resources in Kubernetes?
Compressible resources (like CPU) can be throttled. If a container uses more than its requested share or limits, Kubernetes slows down its execution by limiting access to CPU cycles, but does not terminate the pod. Incompressible resources (like Memory or Storage) cannot be throttled. If a pod runs out of memory, the kernel has no choice but to terminate the container using the OOM killer, as the system cannot reclaim memory dynamically without terminating processes.
2. Technical: How does the OOM killer determine which pod to terminate when a node runs out of memory? Explain the role of QoS classes and OOM score adjustments.
The Linux kernel uses an oom_score (ranging from 0 to 1000) to decide which process to kill. Kubernetes adjusts this score (oom_score_adj) based on the pod's QoS class:
- BestEffort: Receives an oom_score_adj of 1000, making them the first targets for eviction.
- Burstable: Receives a score adjusted based on the percentage of requested memory the pod is using, ranging between 2 and 999.
- Guaranteed: Receives an adjustment of -997 (resulting in an effective score of 3 or less), ensuring they are only terminated as a absolute last resort if system processes or BestEffort/Burstable pods are already cleared.
Production Considerations
Use LimitRanges at the namespace level to automatically assign default requests/limits to pods that lack them. Establish ResourceQuotas to prevent individual teams or applications from monopolizing cluster compute. Keep monitoring metrics such as container_cpu_cfs_throttled_seconds_total to identify hidden performance degradations from CPU starvation.