Long Polling, WebSockets & SSE

Traditional web communication follows a strict request-response lifecycle: the client makes an HTTP request, the server responds, and the connection is closed. This model is stateless and unidirectional, making it difficult to build real-time interactive features. If you are building a chat application, a stock dashboard, or a live collaboration tool, the client needs updates immediately when they occur on the server. To achieve this, system designers rely on three push-based architectures: Long Polling, WebSockets, and Server-Sent Events (SSE).

1. Learning Objectives

• Differentiate between HTTP short polling, long polling, WebSockets, and Server-Sent Events.
• Understand the mechanics of the WebSocket HTTP Upgrade Handshake.
• Analyze the text/event-stream format used in Server-Sent Events (SSE).
• Evaluate the memory and scaling constraints of holding millions of persistent TCP connections (solving the C10K/C10M connection problem).
• Understand how to route event updates to specific server nodes holding client sockets using Redis Pub-Sub.
• Implement connection simulators for Long Polling, WebSockets, and SSE in Java, Python, and C++.

2. Prerequisites

To get the most out of this lesson, you should be familiar with:

• Client-Server Architecture: Socket configurations, HTTP framing, and TCP handshakes.
• Message Brokers: Basic publisher-subscriber message routing concepts.

3. Why This Topic Matters

In modern web applications, real-time functionality is a core requirement. If users must refresh their screens to see new chat messages, or if a stock dashboard displays data that is 10 seconds stale, the system fails to meet user expectations.

However, holding persistent connections open for millions of concurrent users introduces significant architectural challenges. Standard thread-per-connection servers will crash due to memory exhaustion if they try to allocate a thread for every open socket. Choosing the right real-time protocol—and scaling the backend using non-blocking I/O event loops—is critical for building highly available, low-latency applications.

4. Real-world Analogy

Think of different ways to get News Updates from a friend:

Long Polling (The Persistent Call): You call your friend and say: "Let me know when the election results are out. I'll stay on the line." You both sit in silence, tying up the phone line, until they finally read the results. Once they speak, you hang up, call them right back, and repeat the process to wait for the next update.

WebSockets (The Live Phone Call): You dial your friend, they pick up, and you keep the call active. You can both speak, listen, and interrupt each other in real-time, keeping the line open continuously.

Server-Sent Events (The Radio Broadcast): You tune your radio receiver to a specific news station. The host streams live updates to you continuously. You cannot speak back to the host through the radio; the data flows only one-way.

5. Core Concepts

• Full-Duplex: A communication channel that supports simultaneous, bi-directional message transfer in both directions (e.g. WebSockets).
• Unidirectional: A communication channel that flows in one direction only (e.g. Server-Sent Events, where data only flows from server to client).
• Upgrade Handshake: The initial HTTP request sent by a client to request transitioning the connection from standard HTTP to a persistent WebSocket TCP socket.
• text/event-stream: The specific MIME type response header required to establish a Server-Sent Events (SSE) connection, telling the browser to keep the connection open and read incoming lines.
• The C10K Problem: The system design challenge of optimizing web servers to handle 10,000 concurrent connection sockets on a single server node.
• Reconnection Storm: A failure scenario where a server restart causes thousands of clients to attempt to reconnect at the exact same moment, overloading the server.

6. Visualizations

Real-time Connection Lifecycles

7. How It Works Step-by-Step

WebSocket Handshake & Upgrading Path

1. HTTP Request: The client sends a standard HTTP request to the server, including switching headers:
   • Upgrade: websocket
   • Connection: Upgrade
   • Sec-WebSocket-Key: [random-base64]
2. Handshake Check: The server receives the request, hashes the Sec-WebSocket-Key with a globally defined GUID key, and returns an HTTP response:
   • HTTP 101 Switching Protocols
   • Upgrade: websocket
   • Connection: Upgrade
   • Sec-WebSocket-Accept: [calculated-hash]
3. Protocol Transition: Both client and server bypass the HTTP parser and switch to reading raw TCP frames over the same socket.
4. Bidirectional Flow: Client and server send text or binary frames back and forth without connection setup latency.
5. Heartbeat Pings: The server sends periodic ping frames, and the client replies with pong frames to verify the connection is healthy.

8. Internal Architecture

Exposing persistent connections requires a specialized backend gateway architecture:

• Connection Manager Service: A stateless microservice that holds open client TCP connections in an event loop (e.g. Netty). It maps clientId to specific active socket objects.
• Pub-Sub Message Broker (e.g. Redis Pub-Sub): When a user sends a chat message, it is written to the broker. The broker broadcasts the message to the specific server holding the recipient's WebSocket connection, which pushes it to the recipient's client.
• Sticky Sessions Routing: Ensures that if a client reconnects, the load balancer routes them to the same server node if needed, or coordinates routing tables globally.

9. Request Lifecycle

Let's walk through receiving real-time stock ticks on a dashboard using Server-Sent Events (SSE):

• t0: Dashboard client makes a GET /stocks/stream request with header Accept: text/event-stream.
• t1: The Gateway checks cookies, validates the user session, and forwards the request to the SseService server node.
• t2: SseService returns HTTP 200 OK with header Content-Type: text/event-stream and leaves the connection stream open.
• t3: The client browser parses the header and instantiates an EventSource listener object.
• t4: An external stock ticker service writes an update to a Redis Pub-Sub channel: AAPL: $192.50.
• t5: SseService reads the update from the Redis channel, formats the payload as a string data: {"symbol":"AAPL", "price":192.50}\n\n, and writes it to the client's TCP socket.
• t6: The browser's EventSource listener detects the stream frame, triggers the onmessage callback, and updates the dashboard UI.

10. Deep Dive

WebSocket HTTP Upgrade Handshake

The transition from standard stateless HTTP to stateful WebSockets is done using the HTTP Upgrade handshake. The client sends a header string containing Upgrade: websocket. The server validates the request and returns 101 Switching Protocols. This tells both sides to stop parsing HTTP headers and start sending binary or text frames directly over the established TCP socket. This allows bidirectional communication with minimal overhead.

Server-Sent Events (SSE) Protocol

Unlike WebSockets, which uses a custom protocol, Server-Sent Events (SSE) runs over standard HTTP. The server returns a response with header Content-Type: text/event-stream and leaves the connection open.

Data is pushed to the client using a simple text-based format:
data: {"message": "hello"}
\n\n
The double newline (\n\n) tells the browser's EventSource parser that the event payload is complete, triggering the client-side event handler.

Scaling Persistent Connections (C10K Event Loop)

Traditional web servers allocate a thread to handle each client connection. If a thread consumes 1MB of memory, holding 10,000 concurrent client sockets open will consume 10GB of memory just for thread stacks, causing the server to run out of memory (the C10K Problem).

To scale to millions of connections, real-time servers use Non-Blocking I/O Event Loops (e.g., Netty, Node.js). Instead of dedicating a thread to each connection, a small pool of worker threads manages thousands of connections. Sockets register interest in reading/writing, and the event loop notifies threads only when data is ready to be processed, minimizing resource usage.

Reconnection Storms & Backoff Jitter

If a real-time server with 50,000 open client sockets restarts, all 50,000 clients will lose their connection and attempt to reconnect at the same time. This thundering herd can crash the server during boot.

To prevent this, clients must implement Exponential Backoff with Jitter when reconnecting. Instead of reconnecting immediately, clients wait a randomized time before retrying, distributing the connection load.

11. Production Examples

• ChatGPT Text Streaming (SSE): ChatGPT uses SSE to stream text responses back to the browser character-by-character as they are generated, giving a real-time response feel over standard HTTP.
• Slack App Notifications (WebSockets): Slack keeps a persistent WebSocket connection open to push user online statuses, channel messages, and typing indicators in real-time.

12. Advantages

• Long Polling: Globally compatible; works on old legacy browsers and corporate firewalls.
• WebSockets: Ultra-low latency, full-duplex bi-directional messaging, and supports binary payloads.
• Server-Sent Events: Standard HTTP compliant, native browser auto-reconnects, and simple to implement.

13. Limitations

• Long Polling: High overhead (repeated HTTP connection setup), and wastes server resources.
• WebSockets: Bypasses HTTP caching, requires custom proxies, and does not support native browser reconnection.
• Server-Sent Events: Unidirectional (server-to-client only), and limited to text payloads.

14. Trade-offs

• Bidirectional vs. Unidirectional: If the client must send data back to the server frequently (e.g. multiplayer gaming), WebSockets is required. If the client only reads data (e.g. news feeds), SSE is simpler and more resource-efficient.
• Custom Protocol vs. HTTP Standards: WebSockets requires a custom protocol and proxy support, while SSE runs over standard HTTP, making it simpler to deploy.

15. Performance Considerations

• Heartbeat Tuning: Set pings/pongs to fire every 30-60 seconds to detect dead connections without consuming excessive bandwidth.
• Socket Memory Footprint: Optimize socket memory buffers to minimize the RAM consumed by each open connection.
• Proxy Buffering: Disable proxy buffering in NGINX or Envoy to ensure updates are pushed to clients immediately.

16. Failure Scenarios

• Proxy Connection Drops: Corporate firewalls or load balancers often close idle TCP connections after 60 seconds of inactivity.
  Mitigation: Configure the server to send periodic ping frames or SSE comment lines (: keep-alive\n\n) to keep connections active.
• Browser Connection Starvation: Browsers limit the number of open HTTP/1.1 connections to a single domain to 6. If a user opens multiple tabs to an SSE endpoint, they will block other page assets from loading.
  Mitigation: Force the system to use HTTP/2, which supports multiplexing up to 100 streams over a single connection.

17. Best Practices

• Use SSE for Unidirectional Flows: If you only need to push data to the client, choose SSE over WebSockets for simplicity.
• Use Non-Blocking Event Loops: Run real-time servers on non-blocking event loop frameworks to scale connections efficiently.
• Implement Reconnect Jitter: Add random jitter to client reconnection timers to prevent thundering herd crashes.

18. Common Mistakes

• Keeping Database Connections Open: Holding database connections open while a socket waits for updates. This exhausts connection pools immediately, crashing the system.
• Choosing WebSockets by Default: Implementing WebSockets for simple notifications where SSE or even polling would be simpler and more reliable.

19. Implementation (Real-time Connection Simulator)

The code tabs below showcase a complete simulation comparing Long Polling, WebSockets, and Server-Sent Events (SSE) in Java, Python, and C++. It demonstrates request holding, handshakes, and event streams.

20. Interview Questions

Easy

Q: What is the main difference between WebSockets and Server-Sent Events (SSE)?

A: WebSockets provide a Full-Duplex (bi-directional) connection where both client and server can transmit messages simultaneously over a custom TCP socket. Server-Sent Events (SSE) provide a Unidirectional (server-to-client) stream over standard HTTP, meaning only the server can push updates, and clients must use standard HTTP requests to send data back.

Medium

Q: Why is it difficult to load-balance WebSocket connections, and how do you route a message to a user connected to a different server instance?

A: WebSockets are stateful, persistent TCP connections held in memory by a specific server instance. Standard load balancers distribute requests dynamically, but once a WebSocket is established, it cannot be easily moved. If User A is connected to Server 1, and User B sends a message to User A via Server 2, Server 2 must route the message to Server 1.
Resolution: Use a Pub-Sub Message Broker (like Redis Pub-Sub or RabbitMQ). Every server instance subscribes to a channel matching the IDs of the clients it is currently holding. When Server 2 receives a message for User A, it publishes it to the broker, which broadcasts it. Server 1 reads the message and pushes it to User A's open socket.

Hard

Q: Explain the C10K connection problem and detail how event-driven, non-blocking I/O architectures solve it compared to traditional thread-per-connection servers.

A: The C10K problem is the challenge of handling 10,000 concurrent socket connections on a single server node.
Traditional Thread-per-Connection: Allocate a thread to handle each client socket (e.g. standard Apache/Tomcat). If 10,000 clients connect, the server must run 10,000 threads. Since each thread allocates memory for stack space (typically 1MB) and incurs thread context-switching overhead, the server will crash from memory exhaustion and high CPU usage.
Event-Driven Non-Blocking I/O: Uses operating system multiplexing APIs (like epoll in Linux or Kqueue in BSD). A small, fixed pool of worker threads manages all connections. Sockets register interest in read/write events. The event loop monitors the sockets and assigns a thread only when a socket has data ready to be processed, releasing the thread immediately after. This allows a single server instance to hold open millions of connections with minimal RAM and CPU.

21. Practice Exercises

• Easy: Modify the WebSocketSimulator to print a warning if a client attempts to send a message when their socket status is not "WEBSOCKET" (e.g., trying to send over a regular HTTP connection).
• Medium: Extend the SseSimulator to include support for custom event names in the output stream format (e.g. event: stock-tick\ndata: ...\n\n), and show that the client parser routes events correctly.
• Hard: Implement a keep-alive ping worker simulation inside the WebSocketSimulator. Spawns a background thread that periodically sends ping frames to active sockets every 100 milliseconds. If the client fails to respond with a pong within 50 milliseconds, close the socket and remove it from the active map.

22. Challenge Problem

Problem Statement: Design a real-time multiplayer card game server (like Poker). The system has three main requirements:
1. Players: Send coordinates, bets, and actions rapidly (latency must be under 50ms) and receive updates in real-time.
2. Spectators: Watch game streams (reads only) without participating.
3. Legacy clients: Connect to the game server using old web browsers that do not support modern socket upgrades.

Detail the combination of protocols (WebSockets, SSE, Long Polling) you would choose to optimize server scalability and bandwidth. Explain how you load-balance these connections across multiple server nodes while keeping game state consistent.

23. Summary

• Standard HTTP request-response is unsuitable for real-time applications requiring push notifications.
• Long Polling emulates server push by holding standard requests open until updates occur, but wastes resources.
• WebSockets upgraded from HTTP provide a persistent, full-duplex TCP socket connection for bidirectional messaging.
• Server-Sent Events (SSE) establish a long-lived HTTP stream for unidirectional server-to-client pushes.
• Event-driven architectures using non-blocking I/O event loops are required to hold open millions of connections.

24. Cheat Sheet

25. Quiz

1. Which protocol provides full-duplex, bi-directional communication over a single TCP connection?

• HTTP Long Polling
• Server-Sent Events (SSE)
• WebSockets (Correct)
• DNS Query Routing

Explanation: WebSockets establish a persistent, full-duplex connection that allows both client and server to send messages at any time.

2. What MIME type header is required to establish an SSE stream?

• application/json
• text/event-stream (Correct)
• application/octet-stream
• text/html

Explanation: The text/event-stream header tells the browser to keep the connection open and parse incoming data as SSE events.

3. How do event-driven web servers solve the C10K concurrent connection problem?

• By spawning a thread for each connection.
• By closing client sockets immediately.
• By using event loops and non-blocking I/O to manage multiple sockets with a small thread pool. (Correct)
• By switching connections to UDP protocols.

Explanation: Event-driven event loops use non-blocking I/O to handle thousands of open connections using a small thread pool, reducing memory and CPU overhead.

4. What is a "Reconnection Storm"?

• A database connection pool deadlock.
• A scenario where thousands of disconnected clients attempt to reconnect to a restarted server at once, overloading it. (Correct)
• A network route loop.
• An API Gateway scaling crash.

Explanation: Reconnection storms occur when a server restarts, causing all disconnected clients to try to reconnect simultaneously, which can crash the server.

5. How should client applications mitigate reconnection storms?

• By retrying continuously without delays.
• By disabling reconnect features in the application.
• By implementing exponential backoff with random jitter. (Correct)
• By using a different DNS resolver.

Explanation: Adding random jitter to reconnection timers distributes client reconnect attempts over time, preventing thundering herd crashes on the server.

6. Which real-time protocol natively supports automatic browser reconnection out-of-the-box?

• WebSockets
• Server-Sent Events (SSE) (Correct)
• HTTP Long Polling
• gRPC Stream

Explanation: The browser's native EventSource API handles SSE reconnects automatically, whereas WebSockets require custom JavaScript code to handle reconnections.

7. Why are WebSocket connections difficult to load-balance across multiple server nodes?

• Because they require HTTPS encryption.
• Because they are stateful TCP connections tied to the memory of a specific server instance. (Correct)
• Because browsers block socket upgrades.
• Because load balancers do not support Layer 7 routing.

Explanation: Since WebSockets are persistent TCP connections, the connection state is held in the memory of the specific server node it was established on, requiring careful message routing.

8. How can a server route a message to a client connected to a different server instance in a cluster?

• By closing and recreating the socket.
• By using a Pub-Sub message broker to broadcast messages across the server cluster. (Correct)
• By querying the API gateway database directly.
• By using sticky sessions routing.

Explanation: A Pub-Sub broker broadcasts messages across the cluster, allowing the server holding the target client's connection to receive and push the message.

9. What double newline character sequence (\n\n) is used for in SSE message formatting?

• To encrypt the message payload.
• To indicate that the current event frame payload is complete, triggering client callbacks. (Correct)
• To close the stream connection.
• To request a ping handshake.

Explanation: The double newline character sequence (\n\n) tells the client's EventSource parser that the event payload is complete, triggering the handler.

10. What is a common mistake when designing persistent connection architectures?

• Keeping stateless API instances.
• Implementing ping/pong heartbeats.
• Keeping database connections open while holding client sockets. (Correct)
• Using HTTP/2 multiplexing.

Explanation: Keeping database connections open while waiting for updates exhausts connection pools immediately, crashing the system.

26. Further Reading

• High Performance Browser Networking by Ilya Grigorik (Chapters 14, 16, 17).
• Mozilla MDN Web Docs: WebSockets API & Server-Sent Events API guides.
• RFC 6455 - The WebSocket Protocol specifications.

27. Next Lesson Preview

This completes Module 4 — Architecture & Communication. In Module 5 — Distributed System Concerns, we will start by exploring Geohashing & Quadtrees, learning how to represent and index geographic coordinates for location-based search engines.