Latency vs Throughput

Latency measures the time to complete a single operation — for example, the milliseconds between sending a request and receiving the response. Lower is better. Throughput measures the volume of work done over time — for example, requests per second or bytes per second. Higher is better. They're related but distinct: a system can have low latency and low throughput (a fast but single-lane road) or high latency and high throughput (a slow but very wide highway).

1. Learning Objectives

By the end of this lesson, you will be able to:

• Define mathematical latency and throughput along with their standard metrics and units.
• Analyze why averages/means fail to represent real-world performance, and evaluate systems using percentiles (p50, p95, p99, p99.9).
• Map network, hardware, and logical delays that govern performance bounds.
• Formulate capacity plans using Little's Law to calculate concurrent connections and sizing.
• Assess the latency vs. throughput trade-offs involved in buffering, batching, socket options (TCP_NODELAY), and data compression.
• Implement an asynchronous, latency-bounded Batch Processor in TypeScript that optimizes throughput under operational constraints.

2. Prerequisites

Before diving into this topic, you should have a solid grasp of the following concepts:

• The Client-Server Model: Understanding how requests flow through networks, load balancers, and application processes.
• Basic Operating Systems Concepts: Familiarity with concurrency, threads, processes, and network socket connections.
• Basic Statistics: Comfort with calculations involving averages, medians, and percentiles.

3. Why This Topic Matters

In systems engineering, latency and throughput are the foundational metrics that define performance. If you fail to design for latency, your application will feel sluggish, causing users to abandon your platform. If you fail to design for throughput, your system will buckle and crash under high concurrent traffic, such as during a flash sale or marketing event.

A common mistake in systems design is assuming that optimizing one automatically improves the other. In reality, they are often in direct opposition. Optimization techniques like request batching, socket buffering, and compression can dramatically boost throughput while actively degrading latency. For senior engineers, understanding these trade-offs and knowing how to measure them accurately is critical for sizing hardware, preventing cascading timeouts, and designing production-ready systems.

4. Real-world Analogy

To distinguish these concepts, consider two classic physical analogies:

1. The Highway Analogy

Imagine a 10-mile stretch of highway connecting two cities:

• Latency: The time it takes a single car to travel from the start to the end of the highway. At a speed limit of 60 mph, this takes 10 minutes.
• Throughput: The number of cars that pass under a toll gate per minute. If 15 cars pass the gate every minute, the throughput is 15 cars/minute.
• Increasing Throughput: If we add two more lanes to the highway, more cars can travel in parallel. The throughput increases to 45 cars/minute. However, the latency remains exactly 10 minutes because the speed limit hasn't changed.
• Decreasing Latency: If we raise the speed limit to 120 mph, the trip takes only 5 minutes (lower latency). As a side effect of cars traveling faster, throughput will also increase because cars clear the highway quicker.

2. The Water Pipe Analogy

Imagine water flowing through a pipe from a reservoir to a house:

• Latency: The time it takes for a single drop of water to travel the length of the pipe.
• Throughput: The volume of water flowing out of the pipe per second (e.g., 5 gallons per second).
• If the pipe is very narrow but short, a water drop travels through quickly (low latency), but total flow volume is low (low throughput). If the pipe is extremely wide but long, a drop takes minutes to travel through (high latency), but once full, it delivers a massive volume of water per second (high throughput).

5. Core Concepts

Let's define the metrics and mathematical relationships used to analyze latency and throughput:

Latency

Latency is the time it takes for a single operation to complete from start to finish, measured from the client's perspective.

• Units: Milliseconds (ms) for network operations, microseconds (µs) or nanoseconds (ns) for CPU and RAM access.
• Components: Latency is the sum of network propagation delay, serialization delay, processing delay at the server, and queuing delay in buffer pools.
• Statistical Percentiles: Rather than averages, latency must be monitored using percentiles:
  • p50 (Median): 50% of requests are faster than this value.
  • p95: 95% of requests are faster than this value. This represents standard bad performance.
  • p99: 99% of requests are faster than this value. Critical for capturing tail latency and outlier experiences.

Throughput

Throughput is the rate at which a system processes successful operations or transactions over a given time duration.

• Units: Requests Per Second (RPS), Queries Per Second (QPS), Transactions Per Second (TPS), or data rate transfer units like Megabits per second (Mbps) and Gigabytes per second (GB/s).
• Limits: Bound by physical resource ceilings (e.g., maximum network card bandwidth, disk write IOPS, or maximum database connection limits).

The Concurrency Relationship (Little's Law)

In a purely sequential system (where only one request can be processed at a time), throughput is the exact inverse of latency:

However, real-world systems process multiple operations in parallel. If a system can process N concurrent requests, the maximum throughput is governed by Little's Law:

This formula reveals that we can increase throughput in two ways: either by reducing latency (W) or by increasing concurrency (L) via scaling workers or threads.

6. Visualization

Below we visualize the request latency breakdown and compare low-throughput (sequential) environments with high-throughput (parallel) architectures.

Latency Breakdown of a Single Request

This sequence diagram shows where delays accumulate as a request makes its way from client to database and back.

Throughput and Concurrency

In a single-threaded configuration, requests queue behind each other, yielding low throughput. Adding horizontal scaling and concurrent workers increases throughput, allowing multiple requests to process simultaneously, even if individual processing latency remains unchanged.

7. How It Works: The Performance Lifecycle

When an HTTP request is fired, its journey is divided into several distinct stages that determine overall latency. The slowest stage in this chain establishes the system's performance bottleneck.

1. DNS Resolution & TCP Handshake: The client resolves the domain name (latency: 10–100ms) and initiates a TCP/TLS handshake. This handshake requires multiple network round trips, establishing a latency floor before any data is sent.
2. Network Propagation (WAN): Data packets travel across fiber optic cables. Speed-of-light limits govern propagation delays (approx. 1ms per 100 miles).
3. Ingestion and Queueing (OS Buffers): The physical network interface card (NIC) receives the packet. If the operating system's connection queue (TCP backlog) is full, the packet sits in buffer space, adding queueing delay.
4. Application Processing: The application server reads the request. It parses the JSON payload, checks authorization tokens, and runs business rules (using CPU cycles).
5. Downstream I/O (Database & Cache): If the database must retrieve a record from an SSD or hard drive, it incurs disk read latency. Reading from a cache (like Redis) takes <1ms, while disk seek can take 1–10ms.
6. Serialization and Response Routing: The server serializes the result back to raw bytes and streams it over the network card. The packets route back to the client device.

The Bottleneck Principle: According to the Theory of Constraints, the maximum throughput of a system is capped by the slowest stage in the lifecycle. If your database can only process 100 queries per second (QPS), adding 1,000 application servers will not increase throughput; it will only increase queueing delays and drive up tail latency.

8. Internal Architecture

Different hardware and software components exhibit distinct latency profiles and throughput capabilities. Sizing a system requires balancing these limits.

9. Request Lifecycle

To see how latency accumulation affects overall system throughput, let's examine the detailed request lifecycle under two different system load states.

State A: Under Normal Load (Low Queueing Delay)

A client calls a user profile service:

• Network transport takes 25ms.
• The request is immediately accepted by a free application thread. Processing time is 5ms.
• The application makes a database read query. The database is idle and completes the read in 15ms.
• The application serializes and sends back the output, taking 25ms to return.
• Total Latency: 25 + 5 + 15 + 25 = 70ms.
• System Throughput: System handles 50 requests per second easily. Threads are available, so queueing time is 0ms.

State B: Under Heavy Load (Saturation & Queueing)

The same profile service receives 10,000 requests per second, exceeding its capacity:

• Network transport takes 25ms.
• Because all worker threads are busy, the request sits in the operating system's socket queue (TCP backlog) for 400ms.
• Once picked up, application processing takes 5ms.
• The database connection pool is exhausted. The request waits another 200ms for a free database connection.
• The query is executed. Because the database disk I/O queue is saturated, the read takes 120ms (up from 15ms).
• The output is returned over the network, taking 25ms.
• Total Latency: 25 + 400 + 5 + 200 + 120 + 25 = 775ms.
• System Throughput: The system throughput plateaus (caps at 1,200 RPS) while queueing delays continue to escalate. Latency balloons, eventually triggering client-side timeouts.

10. Deep Dive: Percentiles, Little's Law, and Measurement Bias

Why Averages (Means) are Dangerous

In production monitoring, average latency is a deceptive metric. Consider a system processing 100 requests:

• 99 requests take exactly 10ms to complete.
• 1 request takes exactly 1,000ms (1 second) due to a garbage collection pause.
• Average Latency: ((99 × 10) + 1000) / 100 = 19.9ms.

An average of 19.9ms looks healthy. However, this hides the fact that 1% of your users experienced a painful 1-second delay. In a modern microservices architecture, a single user request might fan out to 100 downstream services in parallel. If each downstream service has a p99 latency of 10ms, the probability that the overall user request hits at least one slow p99 response is:

Thus, more than 63% of your users will experience the worst-case tail latency. Systems engineers must monitor p95, p99, and p99.9 metrics to protect the user experience.

Applying Little's Law

We use Little's Law (L = λ × W) to size infrastructure. Suppose your application needs to handle a peak load of 2,500 RPS (λ) with a target response latency of 80ms (W = 0.08s). How many concurrent requests must your system support?

Your application servers must maintain a minimum thread pool capacity of 200 active slots to prevent queuing. If your database takes 20ms to execute queries under this load, the database connection pool must support at least 2500 × 0.02 = 50 active connections.

Coordinated Omission

A subtle and dangerous bias in latency measurement occurs when load testing tools wait for a response before sending the next request. If the server hits a blockage (e.g., a 10-second freeze):

• The load test client blocks, waiting for the active request to complete.
• No new requests are sent during these 10 seconds.
• When the server unblocks, the client resumes and records one very slow request (10s) and then returns to recording fast requests.
• In reality, hundreds of requests that would have been sent during those 10 seconds were omitted from the test. This is Coordinated Omission. It makes latency metrics appear significantly better than they are under load. Accurate load testing requires decoupled, constant-throughput request generators (like wrk2 or JMeter configured for open-loop generation).

11. Production Example

Modern scale architectures demonstrate how latency and throughput are prioritized and tuned in the industry:

1. Business Latency Penalties (Amazon & Google)

In 2006, Amazon conducted experiments where they intentionally introduced latency in 100ms increments. They discovered that every 100ms of latency cost them 1% in sales. Similarly, Google found that a 500ms delay in search results page generation dropped user traffic and ad revenue by 20%. These findings proved that latency is a direct driver of business conversion rates.

2. Apache Kafka: Throughput Maximization

Apache Kafka is designed to process millions of messages per second. To achieve this high throughput, Kafka sacrifices real-time latency by prioritizing batching and sequential disk I/O. Instead of sending each message over the network immediately, Kafka producers collect messages in memory and send them in batches. This amortizes network packet header overhead and allows the OS on the broker to perform sequential page-cache writes. The individual message latency is slightly higher, but the total volume of messages handled per second is exponentially greater.

3. Cloudflare: Decoupling Network Latency via Anycast CDN

To bypass speed-of-light limitations, CDNs like Cloudflare deploy edge nodes globally. Using Anycast routing, client requests route to the physically closest CDN edge server (typically <10ms away). Static assets are served directly from edge memory, circumventing the need to traverse the global WAN to the main database origin center, minimizing user-perceived latency.

12. Advantages

Advantages of Optimizing for Latency

• Better User Experience: Applications feel snappy and responsive, boosting engagement.
• Higher Search Rankings: Search engines (like Google) explicitly use site speed (Core Web Vitals) as a ranking factor.
• Mitigation of Tail Latency: Low base latency prevents tail latency from fanning out and ruining complex page loads.

Advantages of Optimizing for Throughput

• Cost Efficiency: Packing more transactions onto fewer servers reduces cloud infrastructure costs.
• Spike Resilience: High throughput capacity allows systems to weather sudden traffic surges without falling over.
• Efficient Resource Utilization: Techniques like batching maximize CPU and disk efficiency by reducing operations overhead.

13. Limitations

Systems optimization is bounded by physical laws and logical limits:

• Speed of Light in Fiber: Light travels through silica glass at approximately 124,000 miles per second. This sets a hard physical latency floor. A round trip between New York and London cannot be faster than ~60ms, regardless of software optimization.
• Amdahl's Law: The latency of a program is limited by its sequential fraction. If 10% of a task must run sequentially (e.g., acquiring a database lock), you cannot reduce latency below that 10% limit, even if you throw infinite CPU cores (concurrency/throughput) at the remaining 90%.
• Hardware Bandwidth Limits: Physical network cards are capped (e.g., 10Gbps interfaces). Once this bandwidth is fully saturated, throughput cannot increase without upgrading physical hardware.

14. Trade-offs

Optimizing a system requires choosing where to compromise. Here are the four classic latency vs. throughput trade-offs:

1. Batching vs. Real-time Delivery

Writing data to storage in batches increases throughput by grouping multiple disk writes into a single sequential operation, saving CPU context-switching and disk head movement. However, batching degrades latency because individual records must wait in a buffer until the batch is full or a timeout expires.

2. Compression vs. CPU Overhead

Compressing payloads (e.g., using Brotli or Gzip) reduces packet sizes, which lowers network transmission latency. However, compression requires CPU cycles to compress on the server and decompress on the client. On fast networks, the CPU cost can make latency worse; on slow networks, compression is a massive win.

3. Nagle's Algorithm (TCP_NODELAY)

Nagle's algorithm increases network throughput by buffering small TCP packets and sending them only when a full-sized packet is ready, or when an acknowledgement is received. This prevents networks from being saturated by small packets. However, this introduces a delay (up to 40ms) for small, time-sensitive packets. Disabling this via the TCP_NODELAY socket option optimizes for low latency at the cost of network throughput efficiency.

4. Buffer and Queue Capacities

Providing deep queues at the entry point of your server prevents packet drops during traffic spikes (protecting throughput). However, deep queues under sustained overload lead to Bufferbloat — requests sit in queues for seconds, causing latency to skyrocket. Sizing queues small causes early packet drops but keeps latency low for the requests that are accepted.

15. Performance Considerations

When tuning network and application layers, several protocol features impact latency and throughput:

• HTTP/1.1 vs. HTTP/2 Multiplexing: HTTP/1.1 suffers from Head-of-Line (HoL) blocking on TCP connections; only one request can be active at a time. HTTP/2 allows multiplexing multiple requests over a single TCP connection, significantly increasing throughput and reducing connection latency.
• HTTP/3 (QUIC) over UDP: HTTP/3 replaces TCP with QUIC (built on UDP). By bypassing the rigid TCP connection state and handling packet recovery independently per stream, HTTP/3 mitigates HOL blocking at the packet level, dramatically reducing latency on lossy networks.
• Connection Pooling: Establishing database connections is slow (TCP handshake + TLS + authorization). Reusing a fixed pool of pre-established connections bypasses this setup latency, directly improving throughput and speed.
• Garbage Collection (GC) tuning: In languages like Java or Go, GC pauses freeze all application threads. Tuning GC algorithms (e.g., using ZGC or Shenandoah in Java) optimizes for minimal pause times (lowering tail latency) at the expense of slightly reduced total CPU throughput.

16. Failure Scenarios

Failure to balance latency and throughput leads to three classic systems collapse patterns:

1. Bufferbloat & Queue Contention

When incoming request rates exceed processing capacity, queues swell. Requests sit in memory waiting for execution. If client timeouts are set to 2 seconds, but requests take 5 seconds to get through the queue, the client will time out and retry. The server spends CPU cycles processing requests that have already been abandoned, wasting throughput and compounding latency.

2. Tail Latency Amplification

In a large microservice web, a single page request might trigger 50 concurrent microservice calls. If one node experiences a latency spike due to disk serialization or network congestion, the entire parent request waits for that single slow block. The slowest node determines the user-perceived performance.

3. Congestion Collapse

Under extreme load, network packet drops occur. Protocols like TCP respond by backing off, reducing the congestion window. Clients, seeing timeouts, aggressively retry their operations. This retry storm consumes all remaining CPU and network capacity, dropping useful system throughput to near-zero.

17. Best Practices

Implement these patterns to manage latency and throughput effectively:

• Track Percentile SLOs: Avoid averages. Set clear objectives for your p95 and p99 metrics (e.g., "p99 latency must remain under 150ms at 5,000 QPS").
• Load Shedding: When queue lengths exceed safety limits, reject excess incoming requests early with an HTTP 503 status code. This protects the latency of requests already inside the system.
• Enable TCP_NODELAY: For real-time, interactive services (like gaming, messaging, or financial trading), disable Nagle's algorithm to ensure immediate packet delivery.
• Use Asynchronous Pipelines: Move heavy writes or analytics updates out of the user request path. Queue them in an asynchronous broker (e.g., RabbitMQ or Kafka) for background ingestion.
• Bypass Latency with CDN Caching: Keep static and semi-static assets at the edge, physically close to users, minimizing travel times.

18. Common Mistakes

Avoid these pitfalls in systems design:

• Benchmarking on a Local Host: Running performance tests on a single machine or local network hides WAN network transit latency. Tests must match real-world client distributions.
• Over-allocating Thread Pools: Believing that adding more threads increases throughput. If your CPU has 8 physical cores, allocating 1,000 threads leads to extreme context-switching overhead, driving up latency and reducing throughput.
• Unbounded Queues: Using in-memory queues without size limits. Under high traffic, queues consume all RAM, triggering Out-Of-Memory (OOM) crashes.
• Ignoring Downstream Timeouts: Calling third-party APIs without timeouts. If the vendor API blocks, your application threads will hang, exhausting thread capacity and causing cascading outages.

19. Implementation

The TypeScript code below demonstrates a fully operational, memory-safe BatchProcessor. This class implements the latency-throughput trade-off: it groups incoming requests into a single batch to maximize database write throughput, while using a flush timer to guarantee that no request waits longer than a defined latency threshold (maxDelayMs).

20. Interview Questions

Easy Question

Question: Define latency and throughput, and explain their relationship using a physical analogy.

Answer: Latency is the time taken to complete a single operation (e.g., milliseconds for a page load). Throughput is the volume of work processed per unit of time (e.g., requests per second). Using the highway analogy: Latency is the travel time of a single car from A to B (determined by speed limit and distance). Throughput is the number of cars passing a point per hour (determined by the number of lanes). Adding lanes increases throughput but does not make a single car's trip faster (latency remains constant).

Medium Question

Question: Why is average latency an ineffective metric for checking application health in production? What should you use instead, and why does this matter for modern microservice architectures?

Answer: Average latency hides tail performance. If 99 requests take 10ms and 1 request takes 1,000ms, the average is ~20ms, which looks healthy but masks that 1% of users had a slow 1-second experience. Engineers should monitor percentiles (p95, p99, p99.9). In microservice architectures where a user request triggers multiple downstream calls in parallel, tail latency dominates. If a page makes 100 sequential or parallel microservice calls, and each has a 1% chance of taking 1 second (p99), over 63% of overall user requests will experience that 1-second delay.

Hard Question

Question: State Little's Law. How would you apply it to size a database connection pool for an application with a peak throughput of 5,000 queries per second, given that the database executes queries in 15ms under load?

Answer: Little's Law is defined as L = λ × W, where L is the average number of active requests in the system, λ is the throughput (arrival rate), and W is the average response time (latency). Here:
λ = 5,000 QPS
W = 15ms = 0.015 seconds
Applying the law: L = 5000 × 0.015 = 75.
This means at any given millisecond, there are an average of 75 queries executing in parallel. To prevent queueing delay at the database driver layer, the application's database connection pool must have a capacity of at least 75 active connections.

21. Practice Exercises

Easy Exercise

A high-performance trading server receives a trade execution request exactly every 4 microseconds. Calculate the equivalent throughput of the server in operations per second.

Medium Exercise

An application ingests logs at a steady rate of 20,000 logs per second. To save network header overhead, the logs are batched into groups of 1,000 before being pushed to storage. Assuming a constant ingestion rate, calculate the minimum queuing latency added to the very first log in each batch, and explain the latency difference between the first and last log in the batch.

Hard Exercise

Analyze the interaction of Nagle's algorithm and TCP Delayed Acknowledgements. Explain mathematically how they combine to cause a 40ms tail latency spike in microservice environments sending small JSON RPC messages, and outline how the TCP_NODELAY socket flag changes this behavior.

22. Challenge Problem

Scenario: You are the principal architect at an ad-tech analytics company. You need to design an ingestion pipeline that handles 1,500,000 ad-click telemetry events per second (throughput). However, the business requirements dictate that the time between a user clicking an ad and that data appearing on the live customer dashboard must not exceed 150 milliseconds (latency ceiling).

Requirements: Design and document the following aspects of this system:

1. Detail the buffering strategy at the ingestion servers: select the maximum buffer size and flush interval that guarantees the 150ms latency window is preserved under all traffic loads.
2. Explain how you configure client socket settings (e.g., TCP flags), queue structures (e.g., ring buffers vs. disk queues), and DB batch writes to handle this throughput without triggering Out-Of-Memory (OOM) errors.
3. Analyze how your design handles a sudden database replication lag event where database writes take 90ms instead of the normal 5ms, without dropping client packets.

23. Summary

Latency and throughput are distinct but deeply coupled performance metrics. Latency is the round-trip execution time of a single request, whereas throughput is the total volume of requests processed per second. Optimizing system performance requires balancing the trade-offs between these two metrics: techniques like batching and compression maximize throughput at the expense of introducing buffering latency. Monitoring performance requires tracking high percentiles (p95, p99) rather than averages to capture slow tail requests. By applying Little's Law and setting appropriate buffer limits, architects can design systems that handle massive traffic spikes without compromising user experience.

24. Cheat Sheet

25. Quiz

1. Which of the following defines latency?
   • A) The total data storage capacity of a database node
   • B) The number of requests processed by a network card per second
   • C) The time interval taken to complete a single transaction/operation
   • D) The memory consumption limit of an application worker

   Answer: C

   Explanation: Latency measures the time spent from initiating an action to receiving the result.

2. If a server has an average latency of 15ms but its p99 latency is 800ms:
   • A) 99% of requests take exactly 15ms
   • B) 1% of requests take 800ms or longer
   • C) 99% of requests take 800ms or longer
   • D) The server's average latency is incorrect

   Answer: B

   Explanation: The p99 percentile indicates that 99% of requests are faster than 800ms, meaning the slowest 1% take 800ms or longer.

3. How does adding lanes to a highway affect latency and throughput?
   • A) Reduces latency, throughput remains unchanged
   • B) Increases throughput, latency remains unchanged
   • C) Decreases throughput, increases latency
   • D) Increases both latency and throughput

   Answer: B

   Explanation: Adding lanes (parallelism) allows more cars to pass simultaneously (throughput), but does not reduce the time it takes a single car to travel the distance (latency).

4. Under Little's Law (L = λ × W), if a system has a constant latency of 100ms (0.1s) and processes 500 RPS, what is the average number of concurrent requests?
   • A) 5
   • B) 50
   • C) 500
   • D) 5,000

   Answer: B

   Explanation: L = 500 RPS × 0.1s = 50 active concurrent requests.

5. Which design pattern optimizes database throughput at the cost of increasing single-request latency?
   • A) Read Replicas
   • B) Indexing
   • C) Write Batching
   • D) Connection Pooling

   Answer: C

   Explanation: Batching caches updates in memory and writes them together, which reduces total I/O transactions (higher throughput) but makes early requests wait in the buffer (higher latency).

6. What is "Coordinated Omission" in systems performance testing?
   • A) Synchronizing database updates across regions
   • B) A measurement bias where testing tools wait for a request to complete before sending the next one, underrepresenting tail latency
   • C) Dropping duplicate metrics packets to save CPU
   • D) Excluding healthy nodes from load testing logs

   Answer: B

   Explanation: Tools that block waiting for responses fail to issue new requests during server pauses, neglecting the queueing latency that would occur in a real system.

7. What does Nagle's algorithm do?
   • A) Compresses HTTP payloads on the fly
   • B) Groups small TCP packets into larger segments to increase network throughput, introducing minor latency delays
   • C) Distributes connections across load balancers
   • D) Resolves database lock contention

   Answer: B

   Explanation: Nagle's algorithm buffers packets to maximize packet payload efficiency (throughput), at the cost of a delay up to 40ms (latency) for small messages.

8. Why is tail latency amplification critical in fan-out microservice architectures?
   • A) Microservices execute faster when fanned out
   • B) A single request calling multiple microservices in parallel must wait for the slowest service, raising the probability of hitting a slow tail response
   • C) Microservices bypass DNS resolution entirely
   • D) Fan-out limits total system throughput

   Answer: B

   Explanation: Since a parent request blocks until all parallel child requests complete, the probability of experiencing a slow response increases with the number of dependencies.

9. Which protocol eliminates Head-of-Line blocking at the transport layer by using UDP-based streams?
   • A) HTTP/1.1
   • B) HTTP/2
   • C) HTTP/3 (QUIC)
   • D) WebSockets

   Answer: C

   Explanation: HTTP/3 uses QUIC over UDP, handling lost packets independently per stream so that packet loss on one stream does not block other streams.

10. How does "Load Shedding" protect a system from congestion collapse?
    • A) By adding more hardware resources automatically
    • B) By caching all queries in memory
    • C) By dropping incoming requests early when the server is overloaded, keeping latency low for accepted requests
    • D) By writing data directly to disk without checking constraints

    Answer: C

    Explanation: Rejecting excess requests early stops queues from swelling, allowing the server to dedicate its remaining CPU to finishing accepted transactions quickly.

26. Further Reading

• Designing Data-Intensive Applications by Martin Kleppmann: Chapter 1 provides a comprehensive overview of Reliability, Scalability, and Performance metrics.
• Systems Performance: Enterprise and the Cloud by Brendan Gregg: Covers physical machine latency indicators and kernel scheduling delays.
• High Performance Browser Networking by Ilya Grigorik: Explains network transmission delays, TCP windows, and HTTP/2/3 protocols.

27. Next Lesson Preview

In the next lesson, we will explore Scalability. We will study horizontal vs. vertical scaling, stateless app design, and database sharding patterns to understand how to scale our systems to support millions of concurrent users.