Availability

Availability is the fraction of time a system remains operational and accessible. It's often expressed as a percentage ("the nines"): more nines means less allowed downtime. For example, 99.9% ("three nines") permits ~8.76 hours of downtime per year, while 99.999% ("five nines") permits only ~5.26 minutes.

1. Learning Objectives

By the end of this lesson, you will be able to:

• Define mathematical availability and explain the standard "nines" of uptime.
• Distinguish between reliability and availability, utilizing metrics like MTBF and MTTR.
• Apply equations to calculate overall system availability for serial (sequential) and parallel (redundant) system layouts.
• Analyze high-availability design patterns including Active-Passive, Active-Active, Failover, Replication, and Health Probes.
• Assess critical trade-offs related to availability, specifically focusing on the CAP Theorem and consistency guarantees.
• Identify common failure modes (split-brain, cascading failures) and apply architectural mitigations like circuit breakers and consensus.

2. Prerequisites

Before diving into this topic, you should have a solid grasp of the following concepts:

• Basic System Architectures: Familiarity with client-server models, load balancers, and databases.
• Basic Probability and Math: Comfort with multiplication, percentages, and simple decimals (e.g., calculating probabilities of independent events).
• Introductory Networking: Understanding of IP routing, DNS, and HTTP request-response flows.

3. Why This Topic Matters

In modern computing, scale is vast and continuous uptime is non-negotiable. Major companies stand to lose millions of dollars for every minute of downtime. For instance, an outage at a large e-commerce platform can cost over $100,000 per minute in direct revenue loss, not to mention reputational damage and diminished user trust.

Hardware will eventually crash, network links will experience packet loss or split-brain partitions, and software updates will occasionally introduce critical bugs. Building highly available systems is the core practice of designing systems that remain functional in spite of these inevitable hardware and software failures. Understanding availability calculations and failover patterns is a fundamental expectation for senior-level engineering interviews and systems design roles.

4. Real-world Analogy

Imagine a critical hospital power supply. A single line from the city's power grid represents a serial layout. If the city's power line goes down, the hospital goes dark instantly. To prevent this, hospitals deploy a backup diesel generator. This represents a parallel (redundant) layout.

In this scenario, if the main power line fails, an automatic transfer switch detects the voltage drop and boots up the generator. This is a failover mechanism. The time it takes for the generator to start and restore electricity is the recovery time. The hospital's power system as a whole is highly available because it has redundant paths (grid + generator) and a mechanism to detect and route around failure.

5. Core Concepts

Let's define the fundamental concepts and terms used when analyzing and discussing availability:

• Availability: The probability that a system is operational and accessible to perform its required function at any given time. Mathematically, it is:
  Availability = Uptime / (Uptime + Downtime)
• Reliability vs. Availability: These two terms are often conflated but have distinct engineering meanings:
  • Reliability: The probability that a system will perform its required function without failure for a specified duration of time. It is measured by Mean Time Between Failures (MTBF). A system that crashes every hour but reboots in 1 millisecond has high availability but very low reliability.
  • Availability: Focuses on whether the system is ready for use, accounting for both how often it fails and how fast it recovers. It is measured using both MTBF and Mean Time To Repair (MTTR):
    Availability (A) = MTBF / (MTBF + MTTR)
• The Nines: Uptime is described by the number of "nines" in its percentage. For example:
  • 99% (Two Nines): Tolerates ~3.65 days of downtime per year.
  • 99.9% (Three Nines): Tolerates ~8.76 hours of downtime per year. Commonly targeted by standard SaaS applications.
  • 99.99% (Four Nines): Tolerates ~52.56 minutes of downtime per year. Targeted by core services (e.g., authentication, payment gateways).
  • 99.999% (Five Nines): Tolerates ~5.26 minutes of downtime per year. Telecommunication, banking, and medical grid systems.
• Redundancy: The duplication of critical components to increase system dependability. Redundancy can be active-active (all instances process traffic) or active-passive (backup instances remain idle until the primary fails).

6. Visualization

Below, we visualize the difference between components configured in series (where any failure breaks the chain) versus components configured in parallel (where redundant nodes handle traffic and bypass failures).

Serial Configuration (Sequence)

If any single component fails, the entire system becomes unavailable. Adding serial components lowers the total availability.

Parallel Configuration (Redundancy)

If one server fails, the load balancer automatically reroutes requests to the healthy instance, preserving system availability.

7. How It Works: The High-Availability Lifecycle

Maintaining high availability in a system is a continuous, automated process. It unfolds through the following key steps:

1. Health Monitoring (Heartbeats & Probes): A monitoring daemon or service registry continuously sends requests (pings, TCP connections, or application health checks) to each active node.
2. Failure Detection: The monitoring system detects a failure if a node fails to respond within a set timeout, or returns a 5xx HTTP status code repeatedly. To avoid false alarms due to transient spikes, the node is typically marked as "unhealthy" only after a specified threshold (e.g., 3 consecutive failed probes).
3. Mitigation and Routing: Upon marking a node unhealthy, the service registry or load balancer updates its routing table. New requests are instantly directed away from the failed node and routed exclusively to the remaining healthy, redundant nodes.
4. Failover / Promotion (For Stateful Services): If the failed node is a primary database or master node, a consensus protocol or orchestrator initiates a promotion. It selects a healthy secondary replica and promotes it to primary. DNS records or virtual IP addresses are updated to point to the new primary.
5. Reconciliation and Recovery: Auto-scaling groups or container orchestrators (like Kubernetes) notice the dead instance, terminate it, and provision a new instance in its place. The new instance downloads the system state or syncs with the data replicas, passes health checks, and is added back to the pool.

8. Internal Architecture

The architecture of a highly available system is comprised of specialized components working together to detect faults, redirect traffic, and recover state.

9. Request Lifecycle

Let's walk through how a request flows through the architecture during normal operation, and what happens when a node fails.

Normal Operation Flow

1. The client initiates an HTTP request. The DNS returns the Virtual IP address of the active Load Balancer.
2. The active Load Balancer receives the request and selects a healthy application node (e.g., App Node A) from its routing pool.
3. App Node A processes the business logic. If it requires a database transaction, it sends a write query to the designated Primary Database Node.
4. The Primary Database commits the change and asynchronously transmits logs to the Read Replica Database Node.
5. App Node A returns the response to the Load Balancer, which forwards it to the client.

Failure and Automatic Recovery Flow

1. App Node A suffers a hardware fault and goes offline mid-operation.
2. A background health probe fails to get a response from App Node A. After 3 consecutive timeouts (e.g., 3 seconds total), the service registry marks App Node A as unhealthy.
3. The Load Balancer updates its internal route configuration to stop routing requests to App Node A.
4. The client (whose request timed out or received a network reset) retries the request. The Load Balancer intercepts this new request and routes it to App Node B.
5. App Node B successfully processes the transaction and returns a response. The client experiences a minor latency spike during the retry, but zero downtime.

10. Deep Dive: The Mathematics of Availability

To design highly available systems, engineers must calculate the probability of failure across multi-tiered architectures. We calculate total system availability based on whether components are wired in series or in parallel.

1. Sequential (Series) Components

In a serial configuration, the system requires every component to be operational to succeed. If any component in the path fails, the request fails. The total availability is the product of the individual availabilities:

Crucial Insight: Adding more components in sequence decreases overall availability. The system is always less available than its weakest link.

2. Redundant (Parallel) Components

In a parallel configuration, the system has backup components. The system only fails if all parallel components fail simultaneously. We calculate this by multiplying the probabilities of failure (which is 1 minus availability) and subtracting the result from 1:

Crucial Insight: Adding components in parallel increases overall availability, allowing you to build highly available systems from less reliable underlying parts.

Worked Mathematical Example

Consider a three-tier web application containing a Web Server (A₁ = 99.9%), an Application Server (A₂ = 99.9%), and a Database (A₃ = 99.0%).

Scenario A: No Redundancy (Pure Serial)

• Calculation: A_total = 0.999 × 0.999 × 0.99 = 0.988011 (approx. 98.8% availability)
• Allowed Downtime: ~4.38 days of downtime per year. Even though two components have 99.9% uptime, the weaker database drags down the entire system.

Scenario B: Redundant Web and Application Servers, Single Database

We add a redundant Web Server and a redundant Application Server in parallel configurations.

• Web Server Tier Availability: A_web = 1 - (1 - 0.999)² = 1 - (0.001)² = 0.999999
• App Server Tier Availability: A_app = 1 - (1 - 0.999)² = 0.999999
• Database Availability (Single Node): A_db = 0.99
• Total System Availability (Sequential Tiers): A_total = A_web × A_app × A_db = 0.999999 × 0.999999 × 0.99 ≈ 0.99 (approx. 99.0% availability)
• Insight: The single database is a critical single point of failure (SPOF) and bottleneck, rendering the redundant application tiers almost useless for overall uptime.

Scenario C: Redundant Tiers at Every Level (Including Database Failover)

We now add a secondary Database Replica in parallel. Database Tier availability increases:

• Database Tier Availability: A_db_tier = 1 - (1 - 0.99)² = 1 - (0.01)² = 1 - 0.0001 = 0.9999
• Total System Availability: A_total = A_web × A_app × A_db_tier = 0.999999 × 0.999999 × 0.9999 ≈ 0.999898 (approx. 99.99% availability, or "four nines")
• Allowed Downtime: ~52.56 minutes per year. We successfully bumped system availability from 98.8% to 99.99% through redundant topologies.

SLA, SLO, and SLI

Engineering availability must align with business commitments. Site Reliability Engineering (SRE) defines these terms:

• SLI (Service Level Indicator): A quantifiable measure of service performance. Example: "HTTP response status code 200 rate over 5 minutes."
• SLO (Service Level Objective): A target reliability/availability level defined by the SLI. Example: "99.9% of requests over 30 days must return HTTP 200."
• SLA (Service Level Agreement): A legal contract with users outlining financial penalties if the SLO is not met. Example: "If uptime falls below 99.9% in a month, customers get a 10% refund."

11. Production Example

Let's look at how world-class systems design for high availability in production:

1. Netflix's Chaos Engineering and Active-Active Regional Failover

Netflix uses AWS infrastructure across multiple geographical locations. To ensure high availability, they operate in an Active-Active multi-region architecture. Traffic is routed dynamically via DNS across regions. If an AWS region goes down (e.g., us-east-1 suffers a major outage), Netflix's traffic controllers can redirect millions of users to us-west-2 or eu-west-1 in under 10 minutes.

To ensure this failover capability works flawlessly, Netflix built Chaos Monkey (and the wider Simian Army). These systems randomly terminate virtual machines and disrupt network links in production, forcing the system to self-heal and prove that its parallel routing and automated failover function reliably under real failure conditions.

2. GitHub's Database Failover (Orchestrator & Raft)

GitHub manages a fleet of MySQL databases. In the past, database primary node crashes caused extensive outages. To mitigate this, they introduced Orchestrator, an open-source MySQL high-availability and replication management tool.

Orchestrator uses Raft consensus to agree on topology changes. When a primary database node crashes, Orchestrator detects the failure, coordinates with other replica nodes to choose the replica with the most up-to-date transaction log, promotes it, updates the proxy layer (e.g., Consul or HAProxy) to point writes to the new primary, and fences the old primary to prevent any stray writes.

12. Advantages

Designing for high availability yields significant business and technical advantages:

• Revenue Protection: Prevents direct financial losses associated with downtime (e.g., missed checkouts, unserved ads).
• User Trust & Brand Equity: Ensures users can access your service when they need it, building confidence in your platform.
• Zero-Downtime Deployments: Redundant structures allow for rolling updates and canary deployments where instances are taken down, upgraded, and brought back online without interrupting the user.
• Resilience to Disasters: Multi-zone and multi-region strategies ensure safety from localized data center issues like fires, floods, or fiber cuts.

13. Limitations

While highly desirable, high availability introduces boundaries and constraints:

• Exponential Cost Increase: Adding "nines" is not linear. Moving from 99.9% to 99.999% requires redundant networking, multi-region database replication, specialized software layers, and dedicated SRE operations. Uptime costs grow exponentially.
• Operational Complexity: Health checks, consensus algorithms, failover scripts, and replication configurations add massive complexity. This increases the chances of human error during configuration updates.
• Data Consistency Sacrifices: Distributed systems cannot guarantee perfect consistency and high availability during network cuts, requiring applications to design for eventual consistency (stale data reads).

14. Trade-offs

Systems design is the art of making calculated trade-offs. Availability forces several critical compromises:

1. Availability vs. Consistency (CAP Theorem)

Under the CAP Theorem, in the presence of a network Partition (P), a distributed system must choose between Availability (A) and Consistency (C):

• Choosing Availability (AP): The system continues to accept writes on both sides of the partition. However, the data will temporarily diverge, meaning clients reading from one side will see stale or inconsistent data.
• Choosing Consistency (CP): The system blocks writes and reads on the minority side of the partition because it cannot guarantee the data is identical. The system becomes unavailable for these operations to maintain a single source of truth.

2. Availability vs. Latency (PACELC Theorem)

Building on the CAP theorem, the PACELC theorem states that even when the system is running normally (Else - E) without partitions, you must trade off Latency (L) against Consistency (C):

• To keep availability and reads fast (Low Latency), systems write asynchronously to replicas. This exposes the system to stale reads.
• To keep reads perfectly consistent (High Consistency), the system must wait for all replicas to acknowledge the write (Synchronous Replication), which increases latency and reduces write availability if any single replica goes down.

15. Performance Considerations

Engineering for high availability impacts system performance in the following ways:

• Replication Overhead: Synchronous replication blocks threads during writes, increasing response times. Asynchronous replication avoids this but requires managing background log ship buffers and potential read-after-write inconsistencies.
• Health Probe Congestion ("Death by Health Check"): If you have 100 microservices checking the health of a shared cache server every 1 second, the cache server spends significant CPU cycles simply answering health probes rather than servicing real traffic. Probing intervals and jitter must be tuned.
• Failover Latency & Storms: When a primary database fails, promoting a secondary takes time (e.g., 5 to 30 seconds). During this window, all database writes fail, causing temporary unavailability. If failover is too sensitive, network jitter can trigger a false failover, creating a connection storm as applications reconnect.

16. Failure Scenarios

Architects must understand how HA systems break. Below are three classic failure modes and how to prevent them:

1. Split-Brain Scenario

The Scenario: A network partition splits a database cluster into two isolated halves (Subnet A and Subnet B). If both halves independently assume the master node is down, Subnet A may elect Node 1 as Master, and Subnet B may elect Node 2 as Master. Both masters begin accepting writes, corrupting the database state irreconcilably.

Mitigation: Implement a consensus quorum. Require that any election or write operation receive confirmation from a strict majority of nodes:
Quorum = floor(N / 2) + 1 If N=3, quorum is 2. The partition containing only 1 node cannot form a quorum, and automatically steps down or goes read-only.

2. Cascading Failures (Herd Effect)

The Scenario: In an active-active cluster of three servers, one server crashes under high load. Its traffic is immediately redistributed to the remaining two servers. Now overloaded, the second server crashes. The entire load falls on the single remaining server, causing it to crash immediately. The system is flatlined.

Mitigation: Implement rate limiting, client-side circuit breakers, and load shedding (gracefully rejecting requests with HTTP 503 instead of crashing). Use autoscaling, but ensure that limits prevent instantaneous overloading.

3. Flapping and Health Probe Jitter

The Scenario: A node undergoes temporary high CPU garbage collection, causing it to delay responding to health probes. The health checker marks it dead and starts a failover. As soon as the garbage collection finishes, the node responds again, triggering a fail-back. This rapid cycle of marking nodes up and down ("flapping") saturates DNS changes and drops active sockets.

Mitigation: Implement hysteresis (requiring a node to pass more health checks to be marked healthy than the number of failed checks required to mark it unhealthy) and debounce timeouts.

17. Best Practices

To achieve resilient high availability, follow these design principles:

• Design for Failure (Assume Faults): Treat server crashes, disk failures, and network splits as normal, expected occurrences.
• Decouple State (Stateless Application Layers): Keep application servers stateless. Store state in databases or external cache stores, allowing application nodes to be terminated and replaced instantly.
• Eliminate SPOFs: Audit every single block of your architecture. If you find a single database, single load balancer, or single API token manager, add a redundant partner.
• Fail-Safe Defaults (Graceful Degradation): If a dependency (like a recommendations service) is down, fail gracefully by returning a static placeholder list rather than failing the entire page load.
• Continuous Chaos Verification: Conduct fire drills and game days. Automate the destruction of production assets to prove recovery systems work.

18. Common Mistakes

Avoid these common pitfalls in high-availability designs:

• Untested Backups and Standbys: Having a cold standby database but never verifying if it can spin up and read the primary's logs. In an outage, the standby fails due to corrupt configuration files.
• Improper Timeout Windows: Setting health check timeouts too low (creating false failovers due to normal GC pauses) or too high (leaving clients hitting a dead node for minutes before failover occurs).
• Ignoring DNS TTL: Updating DNS entries during a failover but setting the Time-To-Live (TTL) to 24 hours. Client operating systems cache the old IP address, rendering the database failover invisible to them.
• Failing to Fence the Old Master: After promoting a database replica, failing to shut down the old master. Once the network partition heals, the old master accepts stray writes, leading to silent data corruption.

19. Implementation

The TypeScript code below demonstrates a fully operational simulation of a High Availability System Manager. It features a health prober, active-passive load routing, and a quorum-backed database leader selection process to mitigate split-brain issues.

20. Interview Questions

Easy Question

Question: What is the difference between Availability and Reliability, and can a system be highly available but unreliable?

Answer: Reliability is the probability that a system performs its function without failure for a specific time window (measured via MTBF). Availability is the percentage of time a system is functional and ready for requests (measured via MTBF and MTTR). Yes, a system can be highly available but unreliable: if a system crashes once every hour (low reliability) but recovers automatically in 50 milliseconds (very low MTTR), its availability is 99.999% despite frequent crashes.

Medium Question

Question: You have a service with three dependent backend microservices, each with 99% availability. If these microservices are called sequentially to fulfill a request, what is the availability of the service? How can you design the system to improve this availability?

Answer: Because the services are in sequence, the total availability is the product of their individual availabilities:
A = 0.99 × 0.99 × 0.99 = 0.970299 (approx. 97%). To improve availability:

• Introduce redundancy: run parallel instances of the microservices behind load balancers.
• Use asynchronous decoupling: place requests in a highly available message queue so they do not rely on synchronous sequential execution.
• Implement graceful degradation (fallback data) if a non-critical microservice fails.

Hard Question

Question: Explain the split-brain problem in database replication during a network partition. How do systems like ZooKeeper or Consul prevent this while maintaining system safety?

Answer: The split-brain problem occurs when a network partition separates a cluster of servers into two or more parts, and each partition thinks the other has failed. If both partitions elect a master/leader, clients writing to different partitions will cause data divergence, creating conflicting records that cannot be easily merged. Systems like ZooKeeper or Consul prevent this using consensus protocols (e.g., Raft, Paxos) that require a strict mathematical majority (quorum) to elect a leader or commit any write. The partition that has a minority of nodes (e.g., 2 nodes out of a 5-node cluster) will recognize it does not have a quorum, refuse to elect a master, and reject writes, keeping data consistent and avoiding split-brain.

21. Practice Exercises

Easy Exercise

Calculate the total annual downtime in minutes allowed for a service targeting "four nines" (99.99%) availability. Show your calculation process.

Medium Exercise

A service uses an API gateway (99.99%), an application server (99.9%), and a relational database (99.0%). Design a modified architecture diagram that pushes the overall system availability above 99.9%. Specify which redundancy techniques are used and recalculate the final availability.

Hard Exercise

Write an architecture proposal for a multi-region active-active database system. Address the strategy for resolving database write conflicts (e.g., CRDTs, Last-Write-Wins, vector clocks) and analyze how network partitions affect the write latency and availability of your chosen design.

22. Challenge Problem

Scenario: You are the lead system architect at a high-volume financial clearing house. You are tasked with designing the core ledger processing pipeline to support 100,000 transactions per second (TPS) with 99.999% ("five nines") availability. The ledger cannot tolerate double-processing (strict idempotency) or loss of transaction history (strong durability), and must recover from any node or availability-zone crash within 5 seconds.

Requirements: Design and document the following aspects of this system:

1. The precise architecture of the database replication protocol (e.g., multi-paxos/Raft, synchronous vs. semi-synchronous logs).
2. The mechanism to detect primary database node failures, prevent split-brain during a 3-way partition, and execute failover within the 5-second window.
3. How you design the app servers to handle database read/write reconnect storms during a primary promotion without collapsing the system.

23. Summary

Availability is a foundational pillar of system design, measuring the percentage of time a system is fully operational. Building highly available architectures requires shifting from serial topologies to parallel, redundant structures. By removing Single Points of Failure (SPOFs) at every tier, utilizing health monitoring, and automating failovers, systems can survive hardware crashes and network partitions. However, high availability forces system designers to navigate the trade-offs of the CAP and PACELC theorems, balancing consistency, cost, and latency against uptime.

24. Cheat Sheet

25. Quiz

1. Which of the following describes a system with 99.9% availability?
   • A) Permits ~5.26 minutes of downtime per year
   • B) Permits ~8.76 hours of downtime per year
   • C) Permits ~3.65 days of downtime per year
   • D) Permits ~52.56 minutes of downtime per year

   Answer: B

   Explanation: 99.9% (three nines) is calculated as 0.001 × 365.25 days × 24 hours ≈ 8.76 hours of allowed downtime per year.

2. If three servers each with 99.9% availability are placed in parallel (redundant) configuration, the combined availability of this tier is:
   • A) 99.7%
   • B) 99.9%
   • C) 99.9999999%
   • D) 99.9999%

   Answer: C

   Explanation: Calculated as 1 - (1 - 0.999)³ = 1 - (0.001)³ = 1 - 0.000000001 = 0.999999999.

3. What is MTTR in availability equations?
   • A) Mean Time to Redirect
   • B) Maximum Tolerance to Recovery
   • C) Mean Time to Repair (or Recover)
   • D) Mean Time to Replication

   Answer: C

   Explanation: Mean Time to Repair (MTTR) represents the average time required to repair or bring a failed component back online.

4. Which design pattern directly mitigates cascading failures by stopping requests to an already struggling downstream server?
   • A) Active-Passive replication
   • B) Load balancer health checking
   • C) Circuit Breaker
   • D) Leader Election

   Answer: C

   Explanation: The Circuit Breaker pattern trips and returns instant errors when a service starts failing, preventing cascading resource exhaustion on callers.

5. A cluster consists of 5 databases. Under a consensus quorum strategy, what is the minimum number of healthy nodes required to commit a write transaction?
   • A) 2
   • B) 3
   • C) 4
   • D) 5

   Answer: B

   Explanation: Quorum is floor(N / 2) + 1. For N=5, quorum = floor(2.5) + 1 = 3 nodes.

6. In a CAP theorem trade-off, choosing Availability (AP) during a network partition means:
   • A) The system rejects all reads and writes until partition heals
   • B) The system accepts writes, but different nodes might return different/stale data
   • C) The partition heals automatically without latency spikes
   • D) Latency is reduced to zero

   Answer: B

   Explanation: An AP system prioritizes serving requests over consistency, allowing nodes on both sides of a partition to accept updates, causing eventual divergence.

7. What does the PACELC theorem add to the CAP theorem?
   • A) It defines how network partitions are formed
   • B) It focuses on cost considerations in high availability
   • C) It describes the Latency (L) vs. Consistency (C) trade-off during normal operations (Else)
   • D) It provides formulas for calculating replica lags

   Answer: C

   Explanation: PACELC states: If there is a Partition (P), trade off Availability (A) vs. Consistency (C); Else (E), trade off Latency (L) vs. Consistency (C).

8. Why is a low DNS Time-to-Live (TTL) critical for database failover?
   • A) It speeds up SQL queries
   • B) It ensures client devices expire cached IP records quickly and query DNS for the new master's IP
   • C) It increases database buffer pool efficiency
   • D) It keeps database replication logs small

   Answer: B

   Explanation: A low TTL prevents clients from caching the IP of the dead primary, directing them to the promoted backup node IP rapidly.

9. The "flapping" of a server refers to:
   • A) Hard drive disks spinning at uneven speeds
   • B) A server rapidly transitioning between healthy and unhealthy states, triggering unstable failovers
   • C) Data moving back and forth between active-active regions
   • D) Application threads blocking during synchronous writes

   Answer: B

   Explanation: Flapping occurs when temporary node hiccups cause it to repeatedly fail and pass health checks, which can confuse routing tables and trigger endless failovers.

10. What is the purpose of database "fencing" during failover?
    • A) Blocking all read requests to replicas
    • B) Forbidding network access to the promoted database node
    • C) Disabling the old master node to prevent it from accepting stray writes
    • D) Securing the cluster against DDoS attacks

    Answer: C

    Explanation: Fencing (or STONITH - Shoot The Other Node In The Head) ensures the old primary cannot write to storage or memory once a new leader is promoted, protecting consistency.

26. Further Reading

• Google Site Reliability Engineering (SRE) Book: Chapters 1-4 cover SLA, SLO, and availability planning extensively.
• Designing Data-Intensive Applications by Martin Kleppmann: Chapter 5 (Replication) and Chapter 9 (Consistency and Consensus).
• The Amazon Builders' Library: "Avoiding fallback in distributed systems" and "Implementing health checks" articles.

27. Next Lesson Preview

In the next lesson, we will transition from keeping systems online to growing them horizontally. We will explore Scalability: analyzing bandwidth bottlenecks, horizontal vs. vertical scaling, stateless systems, and database sharding techniques to handle massive growth in user traffic.