Disaster Recovery

Distributed systems operate on physical hardware that can fail. A power grid blackout, a fiber-optic cable cut, or a natural disaster can take an entire data center region offline. Disaster Recovery (DR) defines the strategies, architectures, and metrics used to restore data and infrastructure availability after a catastrophic outage.

1. Learning Objectives

• Differentiate between RTO (downtime limit) and RPO (data loss limit).
• Analyze four disaster recovery strategies: Backup & Restore, Pilot Light, Warm Standby, and Multi-Site Active-Active.
• Understand the synchronization challenges of active-active database replication across regions.
• Handle failover traffic redirection using DNS routing policies.
• Design failback workflows to restore traffic to a recovered region safely.
• Implement a replication lag and failover simulator that measures RTO and RPO in Java, Python, and C++.

2. Prerequisites

Before learning about disaster recovery, ensure you understand:

• Database Replication: Master-replica architectures and sync vs. async writes.
• DNS Routing: Geo-proximity and failover routing rules.
• SLA, SLO & SLI: Uptime definitions and objectives.

3. Why This Topic Matters

Disaster recovery is the ultimate insurance policy for distributed systems.

If a primary data center region goes offline and there is no disaster recovery strategy in place, a business can remain down for days. In addition to losing sales revenue, the company can suffer permanent data loss, violating legal regulations and losing customer trust.

A well-designed DR strategy achieves two main goals:

• Data Preservation: Guarantees transaction records are not lost during outages.
• High Availability: Restores application traffic quickly to minimize business disruptions.

4. Real-world Analogy

Think of a Hospital Emergency Backup Power System:

The hospital is normally powered by the city's electrical grid (the primary region).

The Disaster: A severe storm knocks down the main power lines.

The Recovery System (Warm Standby / Pilot Light): The hospital has a diesel generator in the basement. It runs on a low idle setting (pilot light). When the main power fails, the generator automatically starts up, reaches full capacity, and restores power to critical life-support equipment within 10 seconds.

The 10 seconds of darkness represents the Recovery Time Objective (RTO). Since no patient data was lost, the Recovery Point Objective (RPO) was zero.

5. Core Concepts

• Recovery Time Objective (RTO): The maximum acceptable downtime before the service must be restored. It answers: *How quickly must we recover?*
• Recovery Point Objective (RPO): The maximum acceptable data loss window, measured in time. It answers: *How much data can we afford to lose?* (e.g. an RPO of 4 hours means we can lose up to 4 hours of recent transactions).
• Backup & Restore: The slowest and cheapest strategy. Data is backed up to cheap storage (e.g. AWS S3). If a disaster strikes, a new environment is built and data is restored from the backup.
  Metrics: RTO = hours/days, RPO = 24 hours.
• Pilot Light: Core database servers are kept running and synchronized in a secondary region. Application servers are kept turned off, configured as templates, and turned on only during disasters.
  Metrics: RTO = minutes/hours, RPO = minutes.
• Warm Standby: A scaled-down version of the entire environment runs continuously in the secondary region. During a disaster, the secondary region is scaled up to handle full traffic.
  Metrics: RTO = minutes, RPO = seconds.
• Multi-Site / Hot Standby (Active-Active): Full, active duplicate environments run in multiple regions, split-routing client traffic. If one region fails, the remaining regions take over the load instantly.
  Metrics: RTO = near-zero, RPO = near-zero.

6. Visualizations

RTO and RPO Timeline

Disaster Failover Sequence

7. How It Works Step-by-Step

Disaster Recovery Failover Sequence

1. Outage Detection: Heartbeat monitors detect that the primary region is completely unresponsive.
2. Failover Initiation: The SRE team (or an automated orchestrator) triggers the disaster recovery runbook.
3. Database Promotion: The replica database in the secondary region is promoted to primary, allowing it to accept writes.
4. Infrastructure Provisioning (Pilot/Warm): If using a Pilot Light or Warm Standby strategy, application servers in the secondary region are scaled up to full capacity.
5. DNS Redirection: The DNS routing configuration is updated (e.g. changing Route 53 records) to point application traffic to the secondary region.
6. Traffic Verification: Health checks verify that incoming requests are being processed successfully in the new region, completing the failover.

8. Internal Architecture

Disaster recovery architectures use global load balancers and replication networks:

• Global DNS Router (Route 53): Uses health checks to route traffic. If the primary IP fails, the DNS router automatically shifts traffic to the secondary IP based on failover policies.
• Cross-Region Replication (CRR): Database engines replicate writes asynchronously across regions. Since cross-region network latency is high (typically $> 100\text{ms}$), synchronous replication is avoided to prevent client-side slowdowns.
• Infrastructure as Code (IaC): Tools like Terraform define the secondary region infrastructure. This allows SREs to rebuild the entire environment from scratch in minutes if the primary region fails.

9. Request Lifecycle

Let's trace a client write request processed during a disaster failover:

1. Write Submission: A client sends a write request: POST /orders.
2. Connection Failure: The request hits the primary region, which is currently down. The connection times out.
3. Client Re-routing: The client's browser queries DNS again. DNS returns the secondary region IP address.
4. Secondary Processing: The client resends the request to the promoted secondary region. The secondary database processes the write and returns success.

10. Deep Dive

A. Recovery Strategies Cost vs. Speed Comparison

Strategy	RTO Target	RPO Target	Relative Cost
Backup & Restore	Hours / Days	24 Hours	$\$$ (Lowest)
Pilot Light	Minutes / Hours	Minutes	$\$\$$
Warm Standby	Minutes	Seconds	$\$\$\$\$$
Active-Active (Multi-Site)	Near-Zero	Near-Zero	$\$\$\$\$\$\$\$$ (Highest)

B. The Failback Challenge

Once the primary region recovers, returning traffic back to it (failback) is often more difficult than the initial failover:

• Data Drift: While the primary was offline, the secondary processed new writes. The recovered primary is now outdated.
• Re-synchronization: Before redirecting traffic, the secondary must replicate the new data back to the primary database.
• Failback Execution: Once replication catches up, the databases swap roles, and DNS records are updated to point traffic back to the primary.

11. Production Examples

• Netflix Simian Army (Chaos Kong): Netflix routinely runs "Chaos Kong" drills, deliberately shutting down entire AWS regions to verify that their active-active multi-site architecture can shift traffic seamlessly without manual SRE intervention.
• Salesforce Disaster Recovery: Uses Warm Standby architectures. Salesforce keeps mirror databases synchronized in secondary locations, ready to take over customer instances during outages.
• AWS Aurora Global Databases: Replicates database updates asynchronously across up to 5 regions, offering an RPO of under 1 second and an RTO of under 1 minute.

12. Advantages

• Protects Business Continuity: Keeps critical systems available during major cloud region outages.
• Prevents Permanent Data Loss: Regular backups and active database replication protect transaction records.
• Maintains SLA Targets: Fast failover processes help operations teams meet customer uptime agreements.

13. Limitations

• High Cost: Running duplicate standby infrastructure increases monthly cloud bills.
• Synchronization Lag: Cross-region network latency makes synchronous replication impractical, meaning some data loss (RPO) is almost unavoidable during sudden failovers.
• Maintenance Overhead: Keeping IaC templates and database configurations synchronized across regions requires ongoing developer effort.

14. Trade-offs

• Recovery Speed vs. Infrastructure Cost: Active-active setups offer near-zero RTO/RPO but require double the infrastructure budget. Backup and restore strategies are highly cost-effective but take hours or days to recover, risking business downtime.
• Sync vs. Async Database Replication: Synchronous replication guarantees zero data loss (RPO = 0) but adds significant write latency. Asynchronous replication keeps write times low but introduces replication lag, meaning some data may be lost during a failover.

15. Performance Considerations

• Optimize Replication Lag: Monitor and tune database replication networks to keep lag under 1 second, minimizing the data loss window (RPO) during outages.
• DNS TTL (Time to Live) Settings: Set low TTL values (e.g. 60 seconds) on failover DNS records. High TTL values cause client browsers to cache dead IPs for hours, delaying recovery.

16. Failure Scenarios

• Failover Loop Cascade: If the secondary region does not have enough capacity to handle the redirected traffic, the sudden surge will overload it, causing the secondary region to crash as well.
  Mitigation: Configure auto-scaling rules to scale up application nodes in the standby region immediately during failover.
• Split-Brain Database Promoted: If database replication partitions, both the primary and replica databases may assume they are the master, accepting writes independently and causing data conflicts.
  Mitigation: Implement consensus algorithms or cluster management software that permits only one master database node to be active.

17. Best Practices

• Test disaster recovery failover processes regularly using automated chaos engineering drills.
• Use Infrastructure as Code (IaC) to define and build standby environments reliably.
• Set low TTL values on DNS failover records to speed up traffic redirection.

18. Common Mistakes

• Storing backups in the same physical region as the primary servers, leaving them vulnerable to the same outage.
• Assuming failover processes work without testing them regularly, leading to configuration failures during real outages.
• Setting DNS TTL values too high, delaying traffic redirection during failover.

19. Implementation (Replication and Failover Simulator)

Below is a complete implementation of a database replication and disaster recovery failover simulator in Java, Python, and C++. The simulator models transaction logs, asynchronous replication lag, primary region crashes (RPO check), database promotion, and traffic redirection (RTO check).

20. Interview Questions & Answers

Q1. Explain the difference between Recovery Time Objective (RTO) and Recovery Point Objective (RPO).

Answer:

• RTO (Recovery Time Objective): The maximum acceptable downtime before the service must be restored (e.g. recovering the system in under 4 hours). It measures system restoration speed.
• RPO (Recovery Point Objective): The maximum acceptable data loss window, measured in time. For example, if database backups run once daily, a disaster just before the backup will result in up to 24 hours of lost transactions. It measures data backup frequency.

Q2. What is the Warm Standby strategy and how does it compare to a Pilot Light setup?

Answer:

• In a Pilot Light strategy, only core database servers run in the secondary region. Application servers are kept turned off (to save costs) and must be provisioned during failover, resulting in an RTO of minutes or hours.
• In a Warm Standby strategy, a scaled-down copy of the entire application environment (including compute nodes) runs continuously in the secondary region. During a disaster, the standby region is scaled up to handle full traffic, offering a faster RTO of minutes.

Q3. Why is active-active database replication across global regions challenging to implement?

Answer: Active-active database replication is challenging due to replication lag caused by speed-of-light propagation delays across global networks.

If two regions accept writes for the same record at the same time:

• Conflict Resolution: Databases must resolve writes using conflict-free replicated data types (CRDTs) or Last-Write-Wins rules, which can lead to data overwrite bugs.
• Consistent Ordering: Guaranteeing a consistent transaction order across regions requires complex global consensus clocks (e.g. Spanner's TrueTime GPS clocks).

21. Practice Exercises

• Exercise 1 (Easy): Trace a diagram showing the request paths of Client A and Client B under sticky session routing compared to stateless shared cache routing.
• Exercise 2 (Medium): Modify the Python database simulator to support Bi-directional Asynchronous replication. Verify that updates synchronize between both nodes when they are online.
• Exercise 3 (Hard): Write a Python module simulating split-brain conflict resolution using a Last-Write-Wins (LWW) resolver based on epoch timestamps.

22. Challenge Problem

The Active-Active Financial Split-Brain Challenge: You operate a high-frequency trading platform deployed in US-East and EU-West regions in active-active mode. Both regions accept cash deposit and withdrawal requests.

A network partition cuts the connection between the two regions for 10 minutes. During this partition:

• A user makes a \$1,000 withdrawal request in US-East.
• The same user makes a \$1,000 withdrawal request in EU-West.
• Their total account balance is only \$1,200.
• Both regions process the withdrawals locally because they cannot check user balances in the other region.

Propose an architecture using database tokens, consistency zones, or fail-fast routing policies to prevent this overdraft when the network partition occurs.

23. Summary

Disaster recovery is a critical component of business continuity planning. System resilience is defined by RTO (tolerable downtime) and RPO (tolerable data loss) targets. Selecting a recovery strategy—from cheap Backups to expensive Active-Active multi-site environments—requires balancing budget constraints with recovery speed requirements.

24. Cheat Sheet

Strategy	RTO (Downtime)	RPO (Data Loss)	Implementation Complexity
Backup & Restore	Hours / Days	Up to 24 hours	Low (standard file copying).
Pilot Light	Minutes / Hours	Minutes	Medium (requires standby DB replication).
Warm Standby	Minutes	Seconds	High (requires running duplicate environments).
Active-Active	Near-Zero	Near-Zero	Very High (requires global write conflict resolution).

25. Quiz

26. Further Reading

• AWS Architecture Blog: Disaster Recovery Strategies in the Cloud.
• Site Reliability Engineering Handbook — Google SRE team (covers disaster readiness).
• Netflix Chaos Engineering Tools and Best Practices.

27. Next Lesson Preview

Disaster recovery strategies rely on virtualizing infrastructure to make nodes portable across clouds. In the next lesson, we will look at VMs & Containers—the core virtualization abstractions that run modern applications.