Load Balancing

Load balancing distributes incoming network traffic across multiple servers, sending requests only to healthy resources. It lets us add or remove capacity on demand. A load balancer sits in front of servers and routes requests so no single server is overworked; if one fails, traffic is redirected to the rest. To add scalability and redundancy, you can load balance at every layer of the system.

1. Learning Objectives

By the end of this lesson, you will be able to:

• Explain the fundamental purpose of load balancing in scaling application traffic horizontally.
• Contrast the architecture, operation, and performance characteristics of Layer 4 (Transport) and Layer 7 (Application) load balancing.
• Evaluate and select appropriate routing algorithms (e.g., Round Robin, Least Connections, Consistent Hashing) based on workload requirements.
• Design highly available load balancer tiers using technologies like Virtual IPs (VIP), VRRP/Keepalived, and BGP Anycast.
• Implement secure TLS/SSL termination, and optimize performance through connection pooling, multiplexing, and compression.
• Analyze and mitigate failure modes such as split-brain scenarios, cascading failures, and health check storms.

2. Prerequisites

Before diving into this lesson, you should be comfortable with:

• The OSI Model: A clear understanding of the network layers, specifically Layer 3 (Network), Layer 4 (Transport), and Layer 7 (Application).
• Protocols: Basic familiarity with TCP/IP handshakes, UDP, HTTP/HTTPS, and DNS resolution.
• Basic System Design Concepts: Understanding horizontal vs. vertical scaling, latency, availability, and single points of failure (SPOF).

3. Why This Topic Matters

In distributed systems, a single server hits a scaling wall very quickly. Buying larger servers (vertical scaling) becomes exponentially expensive and ultimately impossible due to physical hardware constraints. The only viable alternative is to scale horizontally by running multiple instances of our application on cheap, commodity servers.

However, horizontal scaling is useless without an intelligent coordinator. If all users hit the same server, the system still crashes. A load balancer is the traffic cop of your architecture: it sits between the clients and your backend instances, distributing traffic evenly, detecting hardware failures instantly to reroute requests, offloading cryptographic computations, and protecting servers from malicious traffic. Without highly available, performant load balancing, building a global-scale system with 99.999% availability is impossible.

4. Real-world Analogy

Imagine a massive, popular bank with dozens of teller counters. If customers entered the bank and rushed to whatever counter they saw first, chaos would ensue. Some tellers would be overwhelmed with a long queue of customers needing complex mortgage setups, while other tellers would sit idle doing nothing.

To fix this, the bank employs a receptionist (the Load Balancer) at the entrance who manages a single, orderly queue. The receptionist monitors all teller counters:

• If Teller A completes their task, the receptionist sends the next customer in line to Teller A (Round Robin).
• If Teller B is handling a very slow, complex corporate account, the receptionist sends incoming simple deposits to Teller C instead (Least Connections).
• If Teller D goes on a lunch break (server failure), the receptionist stops routing customers to Teller D until they return (Health Checks).
• If a customer needs to finish a multi-step transaction with the same teller who started it, the receptionist remembers this and sends them back to that teller (Sticky Sessions).

5. Core Concepts

Understanding these core terms is essential before diving deeper into load balancer architectures:

• Virtual IP (VIP): A single public-facing IP address configured on the load balancer. Clients make requests to this VIP, hiding the physical IP addresses of the backend servers.
• Upstream Pool (or Backend Group): A logical grouping of destination servers that run the application code and receive routed traffic.
• L4 Load Balancing: Traffic distribution operating at the transport layer (TCP/UDP). It makes routing decisions purely based on packet headers (source/destination IP and ports) without looking inside the packet payload.
• L7 Load Balancing: Traffic distribution operating at the application layer (HTTP/HTTPS/gRPC). It decrypts and parses the request, making routing decisions based on HTTP methods, URLs, headers, query parameters, or cookies.
• Health Probes: Periodic automated checks (pings, TCP connections, or HTTP requests) sent by the load balancer to determine if a backend server is running and ready to accept traffic.
• Sticky Sessions (Session Affinity): A mechanism ensuring that all requests from a specific client are routed to the same backend server, maintaining local user sessions.
• TLS Offloading: The decryption of incoming HTTPS traffic at the load balancer level, sending unencrypted HTTP traffic to the backend pool. This removes heavy cryptographic CPU overhead from the application servers.

6. Visualization

Below is a visual representation of how traffic flows from a client browser through multi-layer load balancers down to dedicated backend servers:

7. How It Works

When a user requests a resource from a load-balanced system, the system coordinates several steps to route the traffic safely and efficiently:

1. DNS Query: The client resolves the domain name (e.g., api.example.com). The DNS server returns the Virtual IP (VIP) of the load balancer.
2. Establish Connection: The client initiates a TCP handshake with the load balancer VIP (typically on port 80 or 443).
3. SSL/TLS Handshake (L7 Only): If HTTPS is used, the load balancer and client negotiate cryptography, authenticate certificates, and establish an encrypted session.
4. Read Request Content: The L7 load balancer parses the HTTP request headers, cookies, URL path, and method.
5. Lookup Routing Rules: The load balancer matches the request details against its configuration rules (e.g., path starting with /api/v1/billing routes to the billing server pool).
6. Apply Load Balancing Algorithm: Within the target pool, the load balancer filters out unhealthy servers and applies its algorithm (e.g., Least Connections) to pick a backend server.
7. Forward Request (Reverse Proxy): The load balancer opens a TCP connection (or reuses a pooled connection) to the selected backend server and forwards the request. It injects tracking headers like X-Forwarded-For (client IP) and X-Forwarded-Proto (client protocol).
8. Collect Response: The backend server executes the database query or application logic, returning the response payload back to the load balancer.
9. Return to Client: The load balancer processes the response (compressing headers, stripping internal metadata, caching static content) and writes it back to the client.

8. Internal Architecture

Modern software load balancers split their architecture into two distinct operations to maximize throughput and stability: the Control Plane and the Data Plane.

9. Request Lifecycle

To understand how load balancers operate under the hood, let us trace a single HTTP request packet as it traverses the load balancer:

1. Packet Ingress: The client sends an Ethernet frame containing an IP packet. The Network Interface Card (NIC) of the load balancer receives the frame, verifies the checksum, and triggers a CPU interrupt to copy the packet into kernel memory.
2. TCP Handshake Resolution: The kernel TCP stack processes the SYN packet. If using L4 load balancing, the LB intercept mechanism immediately rewrites the packet headers (NAT) and forwards it. For L7, the LB processes the 3-way handshake fully to establish a client-facing TCP connection.
3. TLS Decryption: Once TCP is established, the TLS handshake starts. The LB negotiates cipher suites, decrypts the client's payload using CPU cryptographic engines (like AES-NI), and retrieves the raw HTTP request stream.
4. Header Parsing & Routing Evaluation: The parser scans the HTTP request, identifying the URI path, headers, and cookies. It queries the routing engine to identify the target backend server pool.
5. Applying the Algorithm: The routing engine selects a target server from the healthy backend list using the configured routing algorithm (e.g., Round Robin).
6. Upstream Dispatching: The LB checks if an idle persistent connection is available in its upstream connection pool. If not, it initiates a new TCP handshake with the backend server. It then writes the HTTP request headers (including X-Forwarded-For) and payload onto this upstream connection socket.
7. Backend Processing & Egress: The backend server finishes execution and sends the response packets back. The LB reads the response, applies modifications (e.g., injecting session cookies or compressing body with Brotli), and sends the finalized response back through the client TCP socket.

10. Deep Dive

A. Routing Algorithms

The core of a load balancer is its routing algorithm. These are split into static and dynamic categories:

• Static Algorithms:
  • Round Robin: Sequentially directs requests to the next server in the pool. It assumes all servers have equal capacity and all requests consume identical resources. This fails if requests have highly variable processing costs.
  • Weighted Round Robin: Assigns a numeric weight to each backend server based on its capacity (e.g., a server with 32 cores gets weight 4, a server with 8 cores gets weight 1). The LB routes 4 requests to the first for every 1 request routed to the second.
  • IP Hash: Hashes the client's IP address (e.g., hash(Client_IP) % N) to select a server. It provides primitive session stickiness but suffers when many clients are masked behind a single NAT/Gateway IP (skewing load to a single server). It also causes massive cache invalidation when servers are added or removed (since N changes).
• Dynamic Algorithms:
  • Least Connections: Tracks active TCP connections on each backend server and routes new requests to the server with the lowest connection count. Extremely useful for databases or long-lived connections (WebSockets, HTTP Long Polling).
  • Weighted Least Connections: Combines active connection counts with server weight metrics to balance resource distribution among heterogeneous hardware pools.
  • Least Response Time (Latency): Periodically monitors response times or ping latencies of each server. Requests are routed to the fastest responding server, ensuring optimal user experience during backend degradation.
• Consistent Hashing:

  Traditional hashing (hash(Key) % N) is terrible when scaling because adding or removing a server invalidates almost all mappings, forcing caches to empty or users to log out. Consistent Hashing solves this by using a logical hash ring (typically $2^{32} - 1$ points):

  • Backend servers are hashed using their IPs or hostnames and placed onto the ring.
  • Incoming requests (hashed by Client ID, Session ID, or Request URI) are placed onto the same ring.
  • The request routes to the first server encountered by moving clockwise from the request's position.
  • Virtual Nodes: To prevent "hotspots" (where one server gets a disproportionate share of the ring due to uneven spacing), each server is mapped to multiple "virtual nodes" scattered across the ring (e.g., Server-A-1, Server-A-2, Server-A-3). This balances the load distribution mathematically.

B. Layer 4 vs. Layer 7 Load Balancing

The architectural difference lies in how deep the load balancer inspects the incoming packets:

• Layer 4 (L4) Transport Routing: Operates at TCP/UDP level. The load balancer receives network packets, modifies the destination IP/Port headers (Network Address Translation - NAT), and immediately routes them to the backend server. It does not inspect the HTTP request body or headers, nor does it perform a TLS handshake (unless configured in a raw TCP-TLS proxy mode). It is computationally lightweight, capable of routing millions of packets per second with minimal CPU and memory overhead, but lacks route optimization intelligence.
• Layer 7 (L7) Application Routing: Operates at HTTP/HTTPS/gRPC level. The load balancer terminates the client's TCP/TLS connection, buffers and parses the HTTP request content, evaluates complex routing policies (e.g., path mapping, cookie checks, header manipulation), and opens a new connection to the backend server. This allows for smart path-based routing (e.g., /images to static storage, /api to application servers), cookie-based session stickiness, and integrated Web Application Firewalls (WAF). However, it requires significantly more CPU, memory, and introduces higher processing latency (1-10ms).

C. Direct Server Return (DSR)

In standard reverse proxy configurations, all request and response traffic flows through the load balancer. Since web responses (HTML pages, media streams, API payloads) are orders of magnitude larger than requests (GET statements), the outbound network link of the load balancer quickly becomes a bandwidth bottleneck.

Direct Server Return (DSR) solves this asymmetry:

1. The client sends a request to the load balancer's VIP.
2. The L4 load balancer modifies only the destination MAC address of the packet to match the selected backend server. It leaves the destination IP address set to the VIP.
3. The backend server is configured with the VIP assigned to its local loopback interface (making it accept the packet instead of dropping it).
4. The backend processes the request and sends the response directly back to the client over its own internet gateway, bypassing the load balancer completely. The source IP of the response packets is set to the VIP, so the client's TCP connection remains valid.

DSR dramatically increases egress throughput, making it ideal for video streaming services, CDNs, and high-volume media sites.

D. High Availability Configurations

Because all traffic passes through the load balancing tier, a single load balancer instance is a major Single Point of Failure (SPOF). To eliminate this risk, load balancers must be deployed in redundant setups:

• Active-Passive (Failover): Two load balancers share a single Virtual IP (VIP) using protocols like VRRP (Virtual Router Redundancy Protocol) or CARP. The Active node periodically sends "heartbeat" packets to the Passive node. If the Active node crashes or fails to send heartbeats, the Passive node detects the failure and claims the VIP, taking over traffic routing in seconds. Daemons like keepalived are commonly used for this.
• Active-Active (Anycast / DNS): Both load balancers actively accept traffic simultaneously. This can be achieved via:
  • DNS Round Robin / GeoDNS: The DNS server returns multiple load balancer IPs, directing different clients to different load balancer clusters.
  • BGP Anycast: Multiple load balancers advertise the exact same IP address to upstream routers using the Border Gateway Protocol (BGP). The network routers direct packets to the topologically closest load balancer. If one node fails, the router recalculates the shortest path and sends traffic to the other active load balancer.

11. Production Example

Let us look at how scale-leading tech architectures implement load balancing:

Google Maglev

Google developed Maglev, a software-based L4 load balancer running on commodity Linux servers, to handle all Google traffic. Key architectural elements include:

• Kernel Bypass: Maglev runs entirely in user-space, bypassing the Linux kernel TCP/IP network stack using DPDK (Data Plane Development Kit). Packets are read directly from the NIC ring buffer to user-space memory, allowing a single Maglev node to saturate a 10Gbps link.
• Consistent Hashing Lookup Table: Instead of dynamic state sharing across nodes, Maglev computes a global consistent hashing lookup table. If a Maglev node fails, incoming packets are routed by upstream routers to a sibling Maglev node, which computes the same hash and sends the packet to the exact same backend server, ensuring no connections are broken.

AWS Elastic Load Balancer (ELB)

AWS separates load balancing into specialized tiers managed by their distributed SDN platform (Hyperplane):

• Network Load Balancer (NLB): A Layer 4 load balancer capable of handling millions of requests per second with ultra-low latency. It uses static IP addresses per Availability Zone and Anycast routing. It does not parse application data, preserving packet integrity.
• Application Load Balancer (ALB): A Layer 7 load balancer that operates at the application layer. It evaluates advanced routing rules, parses HTTP/HTTPS protocols, manages cookie-based sticky sessions, terminates SSL/TLS certificates, and integrates with AWS Web Application Firewall (WAF) for request inspection.

12. Advantages

• Horizontal Scalability: Allows system administrators to scale server capacity dynamically up or down based on traffic spikes without impacting users.
• Fault Tolerance & High Availability: Automatically isolates unhealthy nodes from the active server pool, redirecting traffic to running nodes to maintain application uptime.
• Resource Offloading: Handles computationally expensive operations like SSL/TLS handshake decryption, Gzip/Brotli compression, and caching at the edge, freeing backend CPU cycles.
• Centralized Security Control: Serves as a central gatekeeper to implement rate-limiting rules, Web Application Firewalls (WAF), IP white/blacklisting, and DDoS mitigation policies.
• Zero-Downtime Deployments: Facilitates blue-green deployments, canary releases, and rolling upgrades by seamlessly routing traffic away from servers scheduled for updates.

13. Limitations

• Single Point of Failure: If a load balancer cluster itself is not configured correctly for high availability, its outage will take down the entire system.
• Latency Overhead: Decrypting packets, parsing headers, running routing logic, and re-encrypting (in L7 scenarios) adds a latency penalty of 1 to 10 milliseconds.
• Configuration Complexity: Managing DNS settings, SSL/TLS certificate updates, route path mappings, and session synchronization across clusters requires significant DevOps overhead.
• Restricted Backend Statelessness: Encourages session stickiness when applications are stateful, which limits the flexibility of scaling out or replacing nodes dynamically.
• Financial Cost: High-performance hardware appliances or managed cloud load balancers (like AWS ALB) accumulate substantial operational costs under heavy usage.

14. Trade-offs

When designing a load balancing layer, architects must navigate several system design trade-offs:

L4 vs. L7 Routing

L4 operates with extreme speed, low latency, and low CPU usage because it does not inspect packet payloads. However, it cannot perform header modifications, path-based routing, or inspect HTTP payloads for security threats. L7 offers rich, intelligent routing, security filtering, and certificate offloading, but consumes significantly more memory/CPU and introduces higher latency.

Session Stickiness vs. Stateless Backend Scaling

Using sticky sessions allows application servers to cache user data locally in memory, speeding up subsequent requests. However, this binds a user to a specific server. If that server fails, the user's session is lost. Furthermore, if a server gets loaded with many intensive user sessions, the load becomes unbalanced. Shifting to stateless backend nodes with shared session stores (like Redis) makes scaling seamless, but adds network round-trips and DB lookup latency for every HTTP request.

Active-Passive vs. Active-Active HA

Active-Passive setups are easy to configure, write, and manage, but leave expensive hardware or cloud instances sitting idle. Active-Active setups maximize infrastructure usage and increase throughput but require complex routing protocols (Anycast, BGP), and a failure in one node can cause sudden, massive traffic shifts to the surviving node, potentially overloading it.

15. Performance Considerations

To maintain microsecond or low-millisecond latencies at scale, the load balancing tier must be carefully tuned:

• Non-Blocking I/O Multiplexing: Software load balancers must utilize event-driven, non-blocking I/O multiplexing systems (like Linux epoll or BSD kqueue) to handle hundreds of thousands of concurrent client connections on a single CPU core without thread-switching overhead.
• Upstream Connection Pooling: Setting up and tearing down TCP connections is expensive due to the 3-way handshake. The load balancer should maintain a pool of long-lived, idle TCP connections (Keep-Alives) to backend servers, eliminating handshake latency.
• SSL Session Resumption: Support TLS Session IDs or Session Tickets (RFC 5077) so returning clients can skip the full cryptographic handshake on subsequent connections, reducing latency by one network round-trip.
• TCP Tuning: Optimize TCP kernel parameters such as tcp_max_syn_backlog (to handle SYN floods) and tcp_tw_reuse (to quickly reclaim sockets in the TIME_WAIT state).

16. Failure Scenarios

Load balancers are prone to specific operational failures that can lead to catastrophic outages if left unmitigated:

Split-Brain Scenario

In Active-Passive configurations, if the heartbeat communication link between the two load balancers is severed (due to switch failure or packet loss), both instances will assume the other is dead. Both nodes will attempt to bind to the same Virtual IP (VIP), resulting in IP conflicts, packet loss, and flapping traffic. Mitigations include using multiple redundant physical paths for heartbeats and fencing systems (STONITH - "Shoot The Other Node In The Head").

Cascading Failure (The Silent Killer)

If a backend server crashes due to overload, the load balancer detects this and stops sending traffic to it. The load balancer immediately redistributes the failed server's traffic to the remaining healthy servers. If these servers are already running near capacity, the added load causes another server to crash, starting a domino effect that takes down the entire backend pool. Mitigations include rate-limiting at the LB tier, circuit breakers, and scaling up the backend pool automatically.

Health Check Storm (Thundering Herd)

When multiple load balancer nodes actively poll the health of a small backend pool, the total volume of health checks can consume all available connection sockets or CPU resources on the backend servers, effectively executing a self-inflicted DDoS attack. Mitigations include adding randomized jitter to health check intervals, increasing the check interval, and utilizing passive health checks (monitoring real client connection success rates instead of active polling).

Thundering Herd on New Instances

When a new server is added to the backend pool, it starts with zero active connections. If the load balancer is using the Least Connections algorithm, it will aggressively direct all incoming traffic to the new server until its connection count matches the others. This massive, sudden surge will crash the server immediately. Mitigations include implementing a "Slow Start" algorithm, which ramps up the traffic directed to a newly added server gradually over several minutes.

17. Best Practices

• Always Deploy in High Availability: Never run a single load balancer instance. Ensure active-passive failover with VRRP or active-active routing with BGP Anycast/DNS routing.
• Use Connection Draining (Graceful Shutdown): When removing a server for maintenance, configure the load balancer to enter draining mode. It will complete active user requests while immediately stopping new connections from routing to the node.
• Separate Internal and External Balancers: External load balancers should face the internet, terminate client TLS, and filter threats. Internal load balancers should route traffic between microservices within a private subnet, preventing public exposure of database and inner RPC layers.
• Optimize Health Probe Endpoints: Create a dedicated /healthz or /ping endpoint that executes minimal checks (e.g., checking if database connections are active) but is lightweight enough to run without impacting CPU. Never route health checks to heavy pages or index endpoints.
• Implement Timeout Parity: Ensure that the load balancer's client-side timeout is longer than its backend-side timeout. This prevents orphaned backend processes from executing when the client has already disconnected.

18. Common Mistakes

• Pointless Heavy Health Checks: Running health checks that execute deep SQL database transactions or filesystem write checks every few seconds. Under load, this causes false negatives and crashes database connections.
• Forgetting Sticky Session Expirations: Setting cookie-based stickiness without timeouts, resulting in users permanently bound to specific backend instances and breaking horizontal scale efficiency.
• Ignoring Load Balancer Capacity: Assuming the load balancer is a magical entity that never saturates. Load balancers have throughput, memory, and packet-per-second limits that must be monitored and scaled.
• Hardcoding Upstream Server IPs: Directly hardcoding server IPs in the load balancer config instead of integrating with a dynamic service registry (e.g., Consul, AWS Cloud Map) or DNS-based discovery.

19. Implementation

The following is a complete, working software load balancer written in TypeScript using Node.js. It features a Round Robin algorithm, dynamic active health checking, and automatic retry logic when a backend connection fails:

20. Interview Questions

Easy Question: What is the difference between Layer 4 (L4) and Layer 7 (L7) load balancers? When would you use each?

Answer:

• Layer 4 Load Balancers: Operate at the transport layer (TCP/UDP). They make routing decisions based purely on source/destination IP addresses and port numbers. They do not decrypt TLS or read application payload. Use cases: When you need raw speed, low memory usage, and high throughput (e.g., millions of concurrent connections at the entry point of your cloud).
• Layer 7 Load Balancers: Operate at the application layer (HTTP/HTTPS/gRPC). They terminate client connections, decrypt SSL/TLS, and read request parameters (paths, cookies, headers). Use cases: When you require intelligent application routing (e.g., path-based routing like /api vs /static), sticky sessions, SSL termination, and security filtering.

Medium Question: How does Consistent Hashing work in load balancing caching servers? What are virtual nodes?

Answer: Consistent Hashing maps both servers and keys (e.g., client IDs or request URIs) to a logical 360-degree circle (the hash ring). A key is hashed and placed on the ring, then routed to the first server found moving clockwise.

When a server is added or removed, instead of redistributing all keys (as in hash(key) % N), only a small fraction of keys (roughly $K/N$, where $K$ is total keys and $N$ is servers) are re-mapped. This preserves cache hits and prevents database overloads.

Virtual Nodes: Simple server hashes can lead to uneven distribution on the ring, creating "hotspots." To solve this, each physical server is mapped to multiple logical "virtual nodes" (e.g., Server-A-1, Server-A-2, etc.) placed at different hashes around the ring. This averages out the segment sizes on the ring, ensuring uniform load distribution across the cluster.

Hard Question: How does BGP Anycast routing fit into global load balancing, and how do you handle state preservation during router shifts?

Answer: BGP Anycast allows multiple geographically distributed load balancer clusters to advertise the exact same IP address to the internet using the Border Gateway Protocol (BGP). Upstream ISP routers direct packets to the topologically closest cluster according to routing path metrics.

However, if BGP paths recalculate due to link flapping, a client's packets mid-session can suddenly route to a different Anycast data center that has no record of the client's TCP/TLS connection state, resulting in a connection reset (RST).

State Preservation Mitigation: Production systems handle this using:

• Consistent Hashing Connection Tracking (e.g., Google Maglev): Every load balancer node across regions utilizes the exact same deterministic consistent hashing lookup table. If a packet shifts to a new node, the node recalculates the target backend server identically, forwarding the packet to the original backend server which still holds the connection state.
• IPv6 Session Mapping: Encapsulating connection information inside packet headers to allow stateless reconstruction of connection state at the edge.

21. Practice Exercises

Easy Exercise: Weighted Round Robin

Implement a function in Python or JavaScript that accepts a list of servers with different weights (e.g., ServerA: 3, ServerB: 1) and returns the correct server sequence for a stream of 10 incoming requests.

Medium Exercise: Nginx Configuration Modeling

Draft a virtual Nginx config block that defines an upstream group of 3 servers. Configure the group to use the least_conn routing algorithm, set a connection timeout of 3 seconds, specify that a server is marked down after 2 failures within a 10-second window, and configure a backup server.

Hard Exercise: Global Anycast Architecture

Draw a network architecture diagram showing how a multi-region deployment (US-East, EU-West, AP-South) uses BGP Anycast for ingress, L4 Maglev load balancers inside each region, and L7 Envoy proxies to route traffic to stateless microservices. Detail where TLS is terminated and how dynamic updates are sent to the routing tables.

22. Challenge Problem

Scenario: You are the lead systems architect at a fast-growing multiplayer gaming company. The company is launching a real-time multiplayer card game that requires active WebSocket connections from 15 million concurrent players. The game engine is highly stateful, maintaining active match state in-memory on individual server nodes for matches lasting up to 45 minutes.

System Constraints & Requirements:

• Clients must maintain socket affinity to the exact backend server where their match is running.
• If an active backend server node crashes, all active matches on that node are lost, but we must protect the other 99.9% of matches running on neighboring servers from cascading failure.
• A tournament starts every 30 minutes, causing a sudden influx of 1 million connection requests within a 30-second window.

Design the load balancing topology. Write a detailed architectural proposal explaining:

1. How you will prevent connection drops during tournament spikes.
2. Which load balancing layer and algorithm you will select, and how you will handle game session stickiness.
3. How health checks will be configured to detect dead game servers in under 5 seconds without triggering a health check storm.
4. Your strategy to prevent a "thundering herd" effect when a newly launched game server node enters the pool.

23. Summary

Load balancing is a foundation of modern web scale, routing traffic dynamically to prevent server overload, eliminate single points of failure, and maximize hardware efficiency. Operability ranges from Layer 4 network forwarding, which prioritizes speed and raw packet volume, to Layer 7 application proxying, which trades CPU cycles for routing intelligence and payload inspection. Achieving reliability at scale requires combining these two tiers, running active-passive failovers or active-active Anycast configurations to protect the load balancers themselves, and applying optimized health probes, circuit breakers, and connection pools to safeguard backends from cascading outages.

24. Cheat Sheet

25. Quiz

1. Which layer does an Application Load Balancer (ALB) operate at, and what is its main capability?

   • A. Layer 4 (Transport); routes based on IP hash
   • B. Layer 7 (Application); inspects HTTP request headers and URL paths
   • C. Layer 3 (Network); forwards raw IP packets via BGP
   • D. Layer 2 (Data Link); routes frames based on MAC addresses

   Correct Answer: B

   Explanation: L7 load balancers operate at the application layer, allowing them to inspect HTTP/HTTPS payloads to make path-based routing decisions.

2. What is a primary advantage of Direct Server Return (DSR) in load balancing?

   • A. It simplifies SSL/TLS decryption on the load balancer.
   • B. It eliminates the need for backend health checks.
   • C. It routes heavy response payloads directly to the client, bypassing the load balancer.
   • D. It guarantees that client sessions are sticky.

   Correct Answer: C

   Explanation: DSR avoids the LB bandwidth bottleneck by having the backend server send responses directly to the client.

3. Under what conditions does Consistent Hashing outperform simple hashing algorithms like hash(IP) % N?

   • A. When backend servers are scaled out or scaled in frequently.
   • B. When CPU utilization on the load balancer must be minimized.
   • C. When all requests are distributed sequentially.
   • D. When there are only two servers in the pool.

   Correct Answer: A

   Explanation: Adding or removing servers in consistent hashing minimizes key reallocation, preventing massive cache misses.

4. What does the term "SSL/TLS Termination" mean at the load balancer level?

   • A. The load balancer blocks all SSL/TLS requests for security.
   • B. The load balancer decrypts incoming HTTPS traffic and forwards unencrypted HTTP traffic to the backend.
   • C. The load balancer terminates connections that take too long to perform handshakes.
   • D. The load balancer rejects expired certificates.

   Correct Answer: B

   Explanation: Decrypting HTTPS at the load balancer offloads heavy mathematical computations from backend servers.

5. What is a major risk in an Active-Passive load balancer setup if the heartbeat connection is severed?

   • A. Cascading failure on the backend servers
   • B. Split-brain scenario where both load balancers attempt to claim the VIP
   • C. Instant termination of all active client sessions
   • D. DNS record expiration

   Correct Answer: B

   Explanation: Split-brain occurs when both nodes assume the other is dead and actively bind to the same IP, causing packet conflicts.

6. Which routing algorithm is most appropriate for a system with long-lived WebSocket connections?

   • A. Round Robin
   • B. IP Hash
   • C. Least Connections
   • D. Path-based Routing

   Correct Answer: C

   Explanation: Least Connections distributes new sockets to servers with the fewest active WebSockets, balancing long-lived loads effectively.

7. What mechanism prevents a newly added server from being instantly overwhelmed by the Least Connections algorithm?

   • A. Health check timeout
   • B. Slow Start (or Warm-up) algorithm
   • C. Connection Draining
   • D. Rate Limiting

   Correct Answer: B

   Explanation: Slow Start gradually ramps up traffic to a newly added server, letting caches warm up and avoiding crashes.

8. What header is commonly injected by reverse proxy load balancers to preserve the client's actual IP address?

   • A. Client-IP-Address
   • B. X-Forwarded-For
   • C. X-Client-Host
   • D. Via

   Correct Answer: B

   Explanation: The standard HTTP extension header X-Forwarded-For lists the client IP since the IP packet source changes to the LB's IP.

9. How does Google Maglev bypass kernel processing limits to achieve high packet routing speeds?

   • A. Utilizing physical hardware ASIC chips
   • B. Running in user-space using DPDK to bypass the kernel TCP/IP stack
   • C. Restricting routing to the IP-address level only
   • D. Storing all routing rules inside the client's DNS

   Correct Answer: B

   Explanation: Bypassing kernel networking stack using DPDK allows Maglev to read and route packets directly in user-space, maximizing throughput.

10. What is a Health Check Storm?

    • A. A massive storm damaging physical load balancer hardware
    • B. Too many backend servers failing at once, causing alert fatigue
    • C. Excessive polling by load balancers that overloads backend server capacities
    • D. Unencrypted health check packets causing security warnings

    Correct Answer: C

    Explanation: Health Check Storms happen when a large number of load balancers probe a small backend pool, saturating their sockets.

26. Further Reading

• Maglev: A Fast and Reliable Software Network Load Balancer - Google Research (2016) Paper detailing Google's production L4 LB architecture.
• HAProxy Architecture Guide - Official HAProxy documentation explaining event-driven connection models and thread mapping.
• AWS Whitepaper on Elastic Load Balancing - Best practices on configuring high availability and auto-scaling tiers.
• The Envoy Proxy Threading Model - Deep dive into Envoy's non-blocking execution loop architecture.

27. Next Lesson Preview

In our next lesson, we will explore API Gateways. We will discuss how they differ from standard load balancers, how they handle edge concerns like authentication, rate limiting, and request transformation, and how to combine them with load balancers to build secure, scalable entry points for microservice architectures.