Deployment Strategies

Shipping code to production is the most dangerous moment in any software lifecycle. A faulty release can crash services, corrupt databases, or expose security vulnerabilities to millions of users within seconds. In legacy systems, deployments were rare weekend events requiring maintenance windows and team-wide war rooms. Modern cloud-native platforms deploy dozens of times per day. This speed is only possible because of carefully designed Deployment Strategies that control how traffic transitions from the old version to the new one, minimizing blast radius and enabling instant rollbacks.

1. Learning Objectives

• Compare five core deployment strategies: Recreate, Rolling, Blue-Green, Canary, and Shadow.
• Analyze the trade-offs between deployment speed, rollback time, infrastructure cost, and risk exposure.
• Explain how load balancers and service meshes orchestrate traffic shifting during deployments.
• Identify database migration compatibility requirements for zero-downtime deployments.
• Implement a Rolling Deployment Controller simulator with health checks in Java, Python, and C++.

2. Prerequisites

Before studying deployment strategies, ensure you understand:

• Load Balancers: How traffic is distributed across application instances.
• Health Checks: HTTP probes that verify instance readiness.
• Containers & Orchestration: Basic understanding of Docker images and Kubernetes pods.

3. Why This Topic Matters

Choosing the wrong deployment strategy can cascade into catastrophic production incidents.

For example, if an e-commerce platform uses a Recreate deployment (shutting down all old instances before launching new ones), a 30-second gap appears where zero instances are serving traffic. During Black Friday, this outage window loses millions of dollars in revenue. A Rolling deployment eliminates this window by replacing instances one-by-one while the remaining fleet continues handling requests.

4. Real-world Analogy

Think of upgrading the engines on a commercial airliner fleet:

• Recreate: Ground the entire fleet at once, replace all engines simultaneously, then resume all flights. Fast completion, but zero flights operate during the upgrade window.
• Rolling: Take one plane out of service each night, replace its engines, verify it flies, then move to the next plane. The fleet operates at reduced capacity but never fully stops.
• Blue-Green: Build a completely separate fleet of new planes. Redirect all passengers to the new fleet in one cut. If engines fail, instantly redirect passengers back to the old fleet sitting on standby.
• Canary: Replace the engines on a single plane first. Fly it on a low-traffic regional route. Monitor engine performance for a week. If metrics are healthy, upgrade the rest of the fleet.

5. Core Concepts

• Recreate Deployment: Terminate all v1 instances, then start all v2 instances. Simple but causes full downtime.
• Rolling Deployment: Replace v1 instances with v2 instances incrementally (e.g., 1 at a time or 25% batches). No full downtime, but both versions coexist temporarily.
• Blue-Green Deployment: Maintain two identical production environments (Blue = current, Green = new). Switch all traffic from Blue to Green atomically via load balancer or DNS.
• Canary Deployment: Route a small percentage of traffic (e.g., 1-5%) to the new version. Monitor error rates and latency. Gradually increase traffic if metrics are healthy.
• Shadow (Dark Launch) Deployment: Mirror a copy of live production traffic to the new version. The new version processes requests but its responses are discarded. Only the old version serves actual user responses.

6. Visualization

Deployment Strategy Comparison Timeline

7. How It Works: Rolling Deployment Lifecycle

1. Deployment Trigger: The CI/CD pipeline pushes a new container image tagged v2.1.0 to the container registry.
2. Orchestrator Selection: The deployment controller (e.g., Kubernetes) selects the first batch of pods running v1 for termination.
3. Draining: The load balancer marks the selected pod as "draining", completing in-flight requests but refusing new ones.
4. Replacement: Once drained, the pod is terminated. A new pod running the v2 image is launched in its place.
5. Readiness Probe: The orchestrator sends HTTP health check probes (e.g., GET /healthz). Only when the new pod responds with 200 OK is it registered in the load balancer to receive traffic.
6. Repeat: The controller moves to the next batch and repeats steps 2-5 until all pods run v2.

8. Internal Architecture

9. Request Lifecycle

• 1. Deploy Trigger: Engineer merges a pull request to the main branch. CI pipeline builds image app:v2.3.0 and pushes it to ECR.
• 2. Controller Reconciliation: Kubernetes detects the new desired state (image: app:v2.3.0) and begins a rolling update with maxSurge=1 and maxUnavailable=0.
• 3. Pod Spin-up: A new pod with v2.3.0 starts. The readiness probe (/healthz) checks internal connectivity to Redis and PostgreSQL.
• 4. Traffic Shift: Once the probe passes, the Kubernetes Service endpoint list is updated. The load balancer begins routing requests to the new pod.
• 5. Old Pod Drain: The old v2.2.0 pod receives a SIGTERM. It enters a graceful shutdown, finishing in-flight HTTP requests within a 30-second termination grace period before exiting.

10. Deep Dive: Database Migrations & Backward Compatibility

The most dangerous hidden failure in zero-downtime deployments is incompatible database schema changes.

• The Problem:
  During a rolling deployment, both v1 and v2 of the application coexist. If v2 introduces a migration that renames column email to user_email, v1 pods immediately crash because they query SELECT email FROM users and the column no longer exists.
• The Expand-Contract Pattern:
  - Phase 1 (Expand): Deploy v2 code that can read from both email and user_email. Run a migration that adds user_email as a new column and backfills data.
  - Phase 2 (Migrate): Complete the rolling deployment. All pods now run v2 code.
  - Phase 3 (Contract): Deploy v3 that only reads user_email. Drop the old email column.
  This three-phase process keeps schemas backward-compatible at every stage.

11. Production Example

• Kubernetes: Natively supports Rolling deployments via Deployment resources. Parameters like maxSurge, maxUnavailable, and minReadySeconds control rollout velocity and safety.
• AWS CodeDeploy: Supports Blue-Green deployments on EC2 and ECS, automatically swapping Auto Scaling Groups and re-pointing the Application Load Balancer target group.
• Netflix Zuul: Uses weighted routing to implement Canary deployments, shifting 1% of user traffic to new builds before full rollout.

12. Advantages

• Continuous Delivery: Enables frequent, safe releases without manual maintenance windows.
• Blast Radius Control: Canary and rolling strategies limit the percentage of users affected by a faulty release.
• Instant Rollback Capability: Blue-Green and Canary strategies allow reverting to the previous version in seconds.

13. Limitations

• Version Coexistence Complexity: During rolling and canary deployments, two versions run simultaneously. APIs must be backward-compatible, and database schemas must support both.
• Infra Cost for Blue-Green: Maintaining a full duplicate environment doubles compute and memory costs during the transition window.

14. Trade-offs

Select deployment strategies based on organizational priorities:

• Speed vs. Safety: Recreate is fastest to execute but riskiest. Canary is slowest but safest, detecting regressions before they reach all users.
• Cost vs. Rollback Speed: Blue-Green doubles infrastructure costs but enables instant rollback. Rolling updates are cost-efficient but require a slower reverse-rollback process.

15. Performance Considerations

• Cold Start Overhead: New instances require time to warm up JVM caches, database connection pools, and in-memory stores. Use minReadySeconds to ensure pods are stable before receiving traffic.
• Connection Draining Timeout: If draining time is too short, in-flight requests are killed mid-response. If too long, deployments are slow. Typical safe values range from 15-60 seconds.

16. Failure Scenarios: The Thundering Herd Rollout

The Thundering Herd Rollout Failure:
A team configures a rolling deployment with maxSurge=100%, meaning all new pods are launched simultaneously while old pods are drained.
- All new v2 pods start at the same instant. Each pod initializes a database connection pool of 50 connections.
- With 20 pods, 1,000 simultaneous database connections are opened within 2 seconds.
- The PostgreSQL database has a max connection limit of 200. It rejects 800 connection attempts, causing all new pods to fail their readiness probes.
- The orchestrator marks all new pods as unhealthy and terminates them, while old pods have already been drained.
- Result: Total service outage.
Mitigation: Limit maxSurge to 25% and stagger pod startup intervals. Use connection poolers like PgBouncer between application pods and the database.

17. Best Practices

• Always Use Health Check Probes: Never mark an instance as healthy immediately after boot. Wait for the readiness probe to verify database connectivity and cache warmup.
• Automate Rollbacks: Configure deployment controllers to automatically roll back if the error rate exceeds a threshold within the first 5 minutes of a rollout.

18. Common Mistakes

• Ignoring Schema Compatibility: Deploying breaking database migrations during a rolling update, crashing old pods still querying the previous schema.
• Skipping Smoke Tests: Deploying directly to production without running a quick automated health verification suite on the new version first.

19. Implementation: Rolling Deployment Controller Simulator

The following code simulates a rolling deployment controller that replaces application instances incrementally, verifying health checks before proceeding to the next batch:

20. Interview Questions

21. Practice Exercises

22. Challenge Problem

23. Summary

Deployment strategies control how traffic transitions between application versions. Recreate is simple but causes downtime. Rolling updates eliminate downtime by replacing instances incrementally. Blue-Green provides instant rollback via environment switching. Canary minimizes risk by exposing only a fraction of users. Shadow deployments validate new versions under real traffic without any user impact. The optimal strategy depends on the organization's tolerance for risk, cost, and deployment speed.

24. Cheat Sheet

25. Quiz

26. Further Reading

• Kubernetes Deployment Strategy Documentation.
• Continuous Delivery by Jez Humble and David Farley (Chapters on Deployment Pipelines).
• AWS CodeDeploy User Guide.

27. Next Lesson Preview

Rolling deployments replace instances incrementally, but some applications need an even safer approach: maintaining two complete environments. In the next lesson, we deep-dive into Blue-Green Deployment, exploring how to maintain parallel production stacks, switch traffic atomically, handle database state synchronization, and execute instant rollbacks.