CI/CD, GitOps & Production Clusters
GitOps with ArgoCD — Declarative Deployments
Syncing Kubernetes resources continuously from Git source code repositories.
Introduction
Traditional CI/CD pipelines push changes imperatively using cluster access tokens. **GitOps** reverses this flow by using Git as the single source of truth for declarative infrastructure and application states.
ArgoCD is a declarative, GitOps continuous delivery tool for Kubernetes. Running as an agent inside the cluster, it monitors a target Git repository, compares the declared manifest state to the active live cluster state, and automatically resolves any differences.
Why It Matters
Without GitOps, developers often make manual, temporary changes directly to cluster resources using kubectl edit or kubectl apply. These changes are undocumented, making disaster recovery difficult. ArgoCD prevents this configuration drift by continuously reconciling the cluster back to the state declared in Git.
Real-World Analogy
Think of ArgoCD like a **hotel security manager and a master key registry**. Instead of giving every delivery agent a master key to enter rooms and place furniture (push-based keys), you write a detailed layout plan in the lobby registry (Git). The security manager (ArgoCD) reviews the lobby registry hourly, enters the rooms, and aligns the furniture to match the plan. If a guest moves a chair (manual configuration drift), the security manager returns to the room and restores it to its planned position.
How It Works
ArgoCD acts as a reconciliation controller, operating through a loop:
- Declaration: Developers declare manifests (YAML, Helm, Kustomize) in a Git repo.
- Scan: ArgoCD scans the repository (either on a schedule or triggered by a Webhook).
- Compare (Diff): It compares the resource definition in Git to the live resources in the cluster:
Synced: The live state matches the repository declaration.OutOfSync: The live state differs from the repository (e.g. wrong replica count or modified settings).
- Sync (Reconcile): If configured for automated sync, ArgoCD applies the Git declaration, correcting configuration drift and deleting resources that are no longer declared in the repository.
Practical Example
Here is a declarative manifest for an ArgoCD Application Custom Resource Definition (CRD) that syncs a web service namespace:
Quick Quiz
Q1: What is the effect of setting syncPolicy.automated.prune=true in an ArgoCD Application definition?
A) It periodically restarts the pods to release memory leakages.
B) It automatically deletes active cluster resources if their corresponding manifests are removed from the Git repository.
C) It moves outdated commits to a backup branch.
D) It limits container logs storage volume sizes.
Answer: B — Pruning ensures that resources deleted in Git are automatically cleaned up in the cluster, maintaining resource alignment.
Q2: Why does GitOps improve the security posture of a Kubernetes cluster?
A) It automatically encrypts YAML manifests stored in public Git repositories.
B) It eliminates the need to expose cluster administrator access tokens (kubeconfig) to external pipeline engines like GitHub Actions.
C) It blocks all incoming HTTP traffic using NetworkPolicies.
D) It forces the cluster to run without etcd databases.
Answer: B — By using a pull-based model, the deployment agent is local to the cluster, meaning no external access configurations or passwords need to be shared outside the cluster boundary.
Scenario-Based Challenge
Production Scenario:
A critical production outage occurs. A developer imperatively changes a Deployment container replica count from 3 to 10 using kubectl scale to handle incoming load. Ten minutes later, the replicas drop back to 3, worsening the outage. You discover that ArgoCD is configured with self-healing. How do you resolve this conflict during emergencies?
Since self-healing is active, ArgoCD will immediately revert any cluster-level changes that do not match the Git configuration. During an emergency, you have two options:
1. **The GitOps Way (Recommended):** Submit a quick commit to the Git repository updating the replica count to 10. ArgoCD will instantly detect the commit and scale the deployment. This preserves audit histories.
2. **Temporary Override:** Disable automated sync or self-healing via the ArgoCD UI/CLI, perform the manual scale, and re-enable it after the incident is resolved.
Interview Questions
1. What is configuration drift and how does ArgoCD detect it?
Configuration drift occurs when cluster resources are modified manually, causing their configuration to deviate from the state defined in Git. ArgoCD continuously runs a diff between the live state fetched from the Kubernetes API server and the target state parsed from the Git repository, displaying any differences in the dashboard.
2. What happens if Git is down? Can ArgoCD still reconcile the cluster?
If the Git host is down, ArgoCD cannot fetch new changes, meaning you cannot deploy updates. However, the existing active cluster state is not affected, and ArgoCD can still detect and revert manual cluster changes using its local cache.
Production Considerations
Avoid checking base64-encoded Secrets manifests into public git repositories. Use GitOps tools like SealedSecrets or external secrets operators (like HashiCorp Vault integrations) to manage sensitive production credentials.