ReviseAlgo Logo

CI/CD, GitOps & Production Clusters

Production Cluster Checklist — What You Must Have Before Going Live

Verifying backups, network rules, scaling thresholds, and security hardening.

Last Updated: June 15, 2026 • 15 min read

Introduction

Moving workloads from development environments to live production clusters requires auditing configuration safety. A default, raw Kubernetes deployment is insecure, prone to resource starvation, and lacks self-healing checks.

A **Production Cluster Checklist** lists the essential configurations (probes, limits, budgets, security contexts, and backups) required to keep workloads stable under traffic loads.

Why It Matters

Without status probes, Kubernetes may route client HTTP traffic to a starting pod before the database connection is ready, causing requests to fail. Similarly, without CPU limits, a container memory leak can cause node-wide crashes, terminating adjacent workloads.

The Core Checklist Pillars

  1. Resource Control: Every container must specify explicit CPU/Memory request and limit allocations to prevent resource starvation.
  2. Self-Healing Probes:
    • Liveness Probe: Monitors if the app crashed internally, triggering a container restart.
    • Readiness Probe: Monitors if the app is ready to handle traffic, routing traffic only after success.
  3. High Availability Budgets: Use PodDisruptionBudgets (PDB) to ensure a minimum number of pods remain running during cluster maintenance (e.g. node upgrades).
  4. Security Hardening: Configure securityContext to run containers as non-root users, drop Linux privileges, and mount filesystems as read-only.

Practical Example

Here is a production-ready Kubernetes Pod deployment manifest incorporating resource allocations, status probes, and security controls:

Quick Quiz

Q1: What is the primary difference between a Liveness Probe and a Readiness Probe?

A) Liveness verifies permissions; Readiness checks database schemas.

B) Liveness restarts a container if it fails; Readiness stops traffic routing to a container until it passes, preventing client errors during boot.

C) Liveness checks memory volume; Readiness checks CPU limits.

D) There is no difference; they execute identical tasks.

Answer: B — If a liveness probe fails, Kubernetes kills and restarts the container. If a readiness probe fails, it removes the pod from the service endpoint list to stop routing traffic, keeping existing connections stable.

Q2: What happens if a container exceeds its memory limit configured in resources.limits.memory?

A) Kubernetes throttles the container's CPU usage.

B) The container is terminated by the host kernel with an OOMKilled (Out of Memory) status.

C) The cluster automatically scales the node size.

D) The pod is deleted and moved to default namespace.

Answer: B — Unlike CPU pressure which only causes throttling, exceeding memory limits triggers immediate kernel termination (OOMKill) to protect node stability.

Scenario-Based Challenge

Production Scenario:

Your cluster uses a cloud database. During a routine database restart, the database is unavailable for 120 seconds. All 10 web pods in the cluster fail their database connection checks. Because you configured a database connection check in the Liveness Probe, the cluster starts restarting all 10 containers simultaneously, creating a cascade loop that keeps the site down even after the database is back online. How do you fix this?

View Solution

Never put external dependency checks (like database connectivity, Redis checks, or external APIs) inside a Liveness Probe. The Liveness Probe should only monitor internal process state. If an external service is down, restarting the container will not resolve the issue and can create a startup loop. Instead, place dependency checks in the Readiness Probe. When the database is down, the readiness probe fails, and Kubernetes stops routing traffic to the pods. Once the database is back, the readiness probe passes, and traffic routing resumes without restarting the containers.

Interview Questions

1. What is a Pod Disruption Budget (PDB), and why do you need one?

A PDB defines the minimum number or percentage of replica pods that must remain active during voluntary disruptions (e.g. cluster node drains, scaling down, or updates). It prevents cluster administrators or automation scripts from terminating too many pods at once, maintaining application availability.

2. What does readOnlyRootFilesystem do inside securityContext?

It locks the container filesystem, making it write-protected. If an attacker breaches the application container runtime, they cannot download binaries, write scripts, or overwrite system directories. If the application requires temporary folder writing, you must mount an emptyDir volume specifically for that path.

Production Considerations

Schedule automated backup scripts for your etcd database registry daily. Keep cluster node configurations locked, and configure CPU requests equal to CPU limits (Guaranteed Quality of Service class) to ensure predictable performance for critical workloads.