CI/CD, GitOps & Production Clusters
Multi-Environment Clusters — Dev, Staging, Production
Designing isolation strategies using namespaces and distinct physical clusters.
Introduction
When delivering enterprise software, applications run across distinct deployment zones: development clusters for unit validation, staging environments for load testing, and production environments for client traffic. Designing a multi-environment architecture requires choosing between logical isolation and physical isolation.
Using Kubernetes namespaces, we partition resources logically within a shared cluster. For stricter security boundaries, we deploy workloads across separate physical clusters.
Why It Matters
If dev staging workloads share namespaces with production without quotas, a memory leak in a test service can exhaust node resources, causing the host operating system to terminate critical production pods.
Real-World Analogy
Think of multi-environment setups like **office space partition structures**:
- Logical Isolation (Namespaces): A shared coworking floor with glass dividers. Teams sit at separate tables (namespaces) but share the same air, water pipes, and bathrooms (nodes and CPU/memory resources). A loud conversation (resource hog) will distract neighbors.
- Physical Isolation (Clusters): Completely separate office buildings in different districts. If one building loses power (cluster failure), the other continues operating normally, providing strong isolation.
Isolation Strategies compared
| Strategy | Pros | Cons |
|---|---|---|
| Namespaces (Logical) | Low cost, fast setup, high resource utilization. | Shared control plane (etcd), potential noisy neighbors, weak security isolation. |
| Separate Clusters (Physical) | Strong security boundaries, independent control planes, isolated performance. | High operational costs, management complexity, potential resource fragmentation. |
Practical Example
Here is how to structure a Kustomize configuration directory to manage base manifests and apply dev/prod environment overlays:
Deploying staging or production configurations is done by referencing the target overlay directory:
kubectl apply -k overlays/dev or kubectl apply -k overlays/prod
Quick Quiz
Q1: Which resource is used to restrict the total amount of CPU and Memory that can be consumed by all pods within a specific namespace?
A) LimitRange
B) ResourceQuota
C) HorizontalPodAutoscaler
D) NetworkPolicy
Answer: B — ResourceQuotas set hard limits on total resource consumption (CPU, Memory, Pod counts) per namespace, preventing resource exhaustion.
Q2: How does Kustomize differ from Helm in managing multi-environment configurations?
A) Kustomize uses Go template placeholders; Helm uses patches.
B) Kustomize is a template-free tool that overlays changes onto base manifests using a declarative folder structure; Helm compiles templates with values files.
C) Kustomize requires Tiller installation.
D) Helm is only used in development environments.
Answer: B — Kustomize uses a "template-free" approach, merging overlays into base YAML structures, whereas Helm relies on Go template variable interpolation.
Scenario-Based Challenge
Production Scenario:
You run Dev, Staging, and Production namespaces in a single shared cluster. A developer deploys a buggy microservice in the Dev namespace that runs an infinite loop, causing node CPU usage to spike to 100%. The production application sharing those nodes experiences severe request latency. How do you prevent this logical bleed?
View Solution
To isolate logical namespaces:
1. Define a ResourceQuota for the Dev namespace to cap its total CPU allocations (e.g. limit to 4 Cores).
2. Create a LimitRange in the Dev namespace to enforce default CPU request/limit ratios on all pods.
3. If CPU pressure persists, partition the nodes using **Taints and Tolerations** or **NodeAffinity**, pinning production pods to dedicated, high-performance node pools while dev workloads run on cheaper nodes.
Interview Questions
1. What is the difference between ResourceQuota and LimitRange?
A ResourceQuota sets total resource limits (CPU, memory, storage) for an entire namespace. A LimitRange defines minimum/maximum limits and default values for individual pods or containers within that namespace.
2. Can pods in separate namespaces communicate by default?
Yes. Kubernetes networks are flat by default, meaning any pod can ping or call any other pod in another namespace using its fully qualified domain name (FQDN): [service-name].[namespace].svc.cluster.local. To block this, you must apply explicit NetworkPolicies.
Production Considerations
Use separate physical clusters for development and production workloads. Shared clusters present risk: if the cluster API server experiences configuration failure or a security breach occurs at the control plane layer, all namespaces fail together.