Publish-Subscribe

In point-to-point message queuing, each message is processed by exactly one consumer. While this is ideal for processing distinct tasks like transactional jobs, many architectures require broadcasting a single system event to multiple independent services simultaneously. Publish-Subscribe (Pub/Sub) is a messaging pattern where Publishers send messages to logical Topics without knowing who the receivers are, and multiple independent Subscribers listen to these topics to receive copies of every matching message.

1. Learning Objectives

• Understand the publish-subscribe pattern and how it enables one-to-many broadcasting.
• Differentiate between Point-to-Point Queuing and Publish-Subscribe.
• Analyze the mechanics of Transient vs. Durable (Persistent) subscriptions.
• Understand message filtering at the broker level based on attributes.
• Evaluate the trade-offs of delivery reliability, order processing, and network fan-out latency.
• Implement a fully functional, thread-safe Pub/Sub broker engine in Java, Python, and C++.

2. Prerequisites

To fully grasp this lesson, you should review the following topics first:

• Message Brokers: Asynchronous exchanges and AMQP concepts.
• Message Queues: Point-to-point competing consumer workflows.
• Multithreading Concurrency: Thread pools and dynamic lists mapping.

3. Why This Topic Matters

In a microservice architecture, a single user action often has side effects across different domains. Consider a ride-sharing app where a passenger completes a ride:

• The Billing Service must charge the user's card.
• The Driver Service must credit the driver's earnings and update their statistics.
• The Notifications Service must send a receipt email to the user.
• The Analytics Service must log the trip coordinates for mapping analysis.

If you use standard message queues, the Ride Dispatcher service must write 4 separate messages to 4 separate queues. This couples the Dispatcher service to the requirements of the downstream services. If you introduce a new Promo Service that issues coupons for completed rides, you must update the Dispatcher service to write to a 5th queue.

Under Pub/Sub, the Ride Dispatcher service publishes a single RideCompleted event to a topic. The billing, driver, notifications, analytics, and promo services subscribe to this topic. The broker automatically copies (fans out) the message to all subscribers, enabling complete decoupling.

4. Real-world Analogy

Think of subscribing to a Newspaper/Magazine Publication:

The publisher (writer) prints a single issue of a magazine and sends it to the distribution hub. They do not know who the individual readers are, how many there are, or where they live.

Readers (subscribers) register their addresses with the subscription office (the broker) for the specific magazine topic (e.g., "Tech News").

When the new issue is printed, the sorting hub copies and delivers a physical copy to every registered subscriber. If a new subscriber signs up, the publisher's daily writing routine remains completely unchanged.

5. Core Concepts

• Topic: A logical channel or feed to which publishers send messages. Topics group related events (e.g. user-events, sensor-data).
• Subscription: The link between a subscriber and a topic. It registers a consumer's intent to receive copies of all messages published to that topic.
• Fan-Out Pattern: The broker process of copying a single incoming message from a topic and sending it to multiple destination queues or sockets concurrently.
• Durable (Persistent) Subscription: The broker tracks subscriber state. If a subscriber goes offline, the broker buffers messages in a queue on disk. When the subscriber reconnects, the broker delivers the missed messages.
• Transient Subscription: Fire-and-forget. The broker only broadcasts messages to subscribers that are currently online. If a subscriber disconnects, any messages published during that window are lost (e.g., Redis Pub/Sub).
• Message Filtering: The ability to configure subscription rules so that subscribers only receive a subset of a topic's messages based on attributes (e.g., subscribing to user-events but filtering for WHERE action = 'login').

6. Visualizations

One-to-Many Fan-Out Flow

Transient vs. Durable Subscription States

Message Publication Flow

7. How It Works Step-by-Step

1. Topic Registration: The system defines a logical channel named order-events on the broker.
2. Subscription Bindings: Downstream worker queues subscribe to the order-events topic, establishing a link.
3. Event Publish: The order service producer writes a JSON payload to the order-events topic:

Fan-Out Replication: The broker reads the message, retrieves the subscription list, and duplicates the payload, routing copies to each subscribed queue.

Message Filtering (Optional): If a subscription has a filter policy (e.g. WHERE total > 100), the broker checks the message attributes and discards the copy for that subscription if the condition is not met.

Asynchronous Consumption: The individual service workers read their local subscriber queues independently. If the Billing worker is busy, the Notification worker can still consume and process its copy instantly.

8. Internal Architecture

A Pub/Sub broker maintains several internal lookup directories:

• Topic Directory: Maps logical topic names to active subscription pointers.
• Subscription Registry: Tracks subscriber endpoint details (e.g. HTTP webhooks, AMQP queues, TCP sockets) and filter schemas.
• Fan-Out Pipeline Router: A multi-threaded replication engine that copies payloads and routes them to target delivery queues in parallel.

9. Request Lifecycle

Let's trace a publish-subscribe request lifecycle:

10. Deep Dive

A. Publish-Subscribe vs. Point-to-Point Queue

Metric	Point-to-Point Queue	Publish-Subscribe Topic
Workload Division	1:1 (One message $\rightarrow$ One consumer).	1:Many (One message $\rightarrow$ Many subscribers).
Decoupling axis	Decouples execution timing.	Decouples execution timing AND service domains.
Message Lifetime	Removed immediately after ACK.	Duplicated and routed to all active subscription queues.

B. Subscription Durability Models

Managing offline subscribers requires selecting the correct durability model:

• Transient Subscriptions (In-Memory Broadcast): Best for low-latency notifications where missing messages is fine (e.g. stock price updates, chat rooms, Redis Pub/Sub). Sockets deliver directly to active connections; if a client drops, the message is discarded.
• Durable Subscriptions (SNS + SQS Hybrid): Best for critical business events (e.g. order checkouts). The broker fans out messages to durable message queues bound to each subscription. If a consumer crashes, the queue buffers the messages on disk until the consumer recovers.

C. Attribute-Based Message Filtering

To prevent subscribers from fetching messages they do not care about, modern pub/sub engines support Filter Policies. The publisher attaches metadata attributes to the message envelope:

The broker reads these attributes and evaluates them against subscription filter rules (e.g., country == 'US'). If a match is found, the broker delivers the message; otherwise, it discards the copy, saving network bandwidth and processing cycles.

11. Production Examples

• AWS SNS (Simple Notification Service): A managed pub/sub service. You publish to SNS topics, and SNS fans out copies to AWS SQS queues, Lambda functions, or HTTPS webhook endpoints.
• Google Cloud Pub/Sub: GCP's scalable messaging engine. It uses a pull-based subscription model where consumers poll subscription queues, scaling to billions of events per second.
• Redis Pub/Sub: An in-memory, transient pub/sub engine. It is fast (sub-millisecond) but does not persist messages, making it ideal for real-time chat apps and live dashboards.

12. Advantages

• Loose Coupling: Publishers do not know who the subscribers are, allowing teams to develop and deploy services independently.
• System Extensibility: You can add new features (e.g. a new Audit Logging Service) by subscribing to existing topics without modifying the publishing services.
• Parallel Processing: A single event triggers actions across billing, inventory, and notifications concurrently, reducing processing times.

13. Limitations

• Network Bandwidth Bloat: High fan-out ratios (e.g., 100 subscribers for a high-traffic topic) can saturate network interfaces at the broker level.
• Lack of Delivery Insights: Publishers have no visibility into whether or when subscribers process messages, complicating error handling.
• Schema Versioning Conflicts: Changing the payload structure can break downstream subscribers that are expecting older fields.

14. Trade-offs

Topic Granularity: Wide vs. Narrow Topics

Wide Topics (e.g. system-events): A single topic handles all system events. This simplifies broker configuration but forces subscribers to receive and filter out a large volume of irrelevant messages.

Narrow Topics (e.g. user.orders.checkout.success): Specialized topics for specific events. This saves subscriber processing cycles but results in topic sprawl, increasing configuration and monitoring complexity. Most architectures compromise by grouping events by domain (e.g. order-events, user-events).

15. Performance Considerations

• Replication Fan-Out Speed: Duplicating and serializing messages to dozens of subscribers consumes CPU cycles on the broker. Use thread pools and asynchronous non-blocking network writes to prevent replication bottlenecks.
• Metadata Caching: Keep active subscription registries and filter rules cached in memory on the broker to prevent database lookups on every published message.

16. Failure Scenarios

• The Slow Subscriber Bottleneck: If a subscription uses a push-based model without flow control, a slow or failing consumer will back up messages. The broker must store these pending messages in memory or on disk, which can exhaust resources.
  Mitigation: Bound subscription queues, set strict TTLs, and use rate limiting to drop messages for slow transient subscribers.
• Schema Drift Crashes: A team updates a publisher's payload schema (e.g., renaming a JSON key) without updating the subscribers, causing downstream parsers to crash.
  Mitigation: Implement a central Schema Registry (like Confluent Schema Registry) to validate payloads before publication.

17. Best Practices

• Keep event payloads immutable and backward-compatible.
• Use unique message IDs to support consumer-side deduplication.
• Leverage broker-side filtering to save subscriber bandwidth and processing cycles.
• Use durable subscriptions with DLQs for critical business workflows.

18. Common Mistakes

• Using transient pub/sub (like Redis Pub/Sub) for transactional states, risking data loss during consumer disconnects.
• Creating a tight schema coupling by passing large, complex database entity models instead of light key-based event envelopes.

19. Implementation (Pub/Sub Engine)

Below is a complete, production-grade Asynchronous Publish-Subscribe Engine. It supports topic registration, dynamic client subscriptions (both durable and transient), message attribute filtering, and concurrent event fan-out replication.

20. Interview Questions & Answers

Q1. What is message "fan-out" in publish-subscribe systems?

Answer: Fan-out is the process where a single message published to a topic is replicated and delivered to multiple independent subscribers.

The broker acts as the replicator. When a message reaches a topic, the broker queries its subscription table, duplicates the payload, and sends copies to all active queues, sockets, or webhooks bound to that topic. This isolates downstream subscribers, allowing a single event to trigger actions across multiple domains in parallel.

Q2. What is the difference between a durable subscription and a transient subscription?

Answer:

• Durable Subscription: The broker registers the consumer's identity. If the consumer goes offline, the broker writes incoming messages to a queue on disk. When the consumer reconnects, the broker delivers the buffered messages. Use this for critical business updates (e.g., payment logs).
• Transient Subscription: Fire-and-forget. The broker only broadcasts messages to consumers that are currently online. If a consumer is offline, the message is discarded. Use this for real-time dashboards or telemetry.

Q3. How does topic-level message filtering save network bandwidth in high-volume systems?

Answer: Message filtering allows subscribers to register attribute rules (e.g. WHERE region = 'US') along with their subscription.

Instead of sending all messages to all subscribers and forcing them to discard irrelevant payloads, the broker evaluates the filter rules locally. The broker only copies and transmits messages that match the criteria. This saves network bandwidth, reduces serialization CPU costs on the broker, and minimizes processing loads on downstream services.

21. Practice Exercises

• Exercise 1 (Easy): Sketch a diagram mapping a single order-created topic fanning out messages to three separate microservices: Billing, Shipping, and Notifications.
• Exercise 2 (Medium): Modify the provided Python simulation to add Regex-Based Filtering. Subscribers should be able to filter topic bodies using regular expressions (e.g., matching only messages that contain the word "Error").
• Exercise 3 (Hard): Implement a Python script simulating a Durable Subscription Buffer limit. If a durable subscriber goes offline and its buffer queue exceeds a set limit (e.g. 5 messages), the broker should drop older messages or send them to a DLQ to prevent memory leak crashes.

22. Challenge Problem

Designing a Partitioned Pub/Sub Event Hub: You are designing a high-throughput event hub (similar to Apache Kafka) processing 1,000,000 events per second.

To handle this volume on a single topic, you decide to partition the topic across 3 physical nodes.

• Explain how the publisher uses a partition key (e.g. user_id) to decide which partition node to send the message to.
• Explain how a consumer group coordinate subscription read indexes (offsets) to ensure that messages within a partition are processed in order.
• Detail what happens when a partition node crashes and how the broker coordinates consumer rebalancing.

23. Summary

Publish-Subscribe (Pub/Sub) is a fundamental pattern for broadcasting messages to multiple independent receivers asynchronously. By fanning out copies to topics subscriptions, it decouples publishers and subscribers in both execution time and domain dependency. Properly selecting transient or durable subscription models and utilizing attribute filters is critical to building a high-performance messaging pipeline.

24. Cheat Sheet

Metric	AWS SNS	Redis Pub/Sub	Apache Kafka
Durability Model	Durable (when bound to SQS queues).	Transient (in-memory only).	Highly Durable (persistent partition logs on disk).
Consumption Axis	Push (dispatched to endpoints).	Push (pushed to active connection sockets).	Pull (consumers actively poll partitions).
Message Replay	No (deleted once sent/ACKed).	No (fire-and-forget).	Yes (consumers can reset offsets to replay logs).
Attribute Filtering	Yes (Broker parses subscription rules).	No (Wildcard channel matching only).	No (Filtering occurs at consumer code level).

25. Quiz

26. Further Reading

• Enterprise Integration Patterns (Chapter 3: Messaging Channels) — Gregor Hohpe.
• Amazon SNS Developer Guide: Fan-out and filter policies tutorial.
• Designing Data-Intensive Applications (Chapter 11: Stream Processing) — Martin Kleppmann.

27. Next Lesson Preview

We have seen how Pub/Sub broadcasts events to multiple independent queues. However, as the number of queues, topics, and microservices in an enterprise grows, managing these connections individually becomes chaotic. In the next lesson, we will explore the Enterprise Service Bus (ESB)—the enterprise integration pattern that introduces a centralized middleware bus to translate, orchestrate, and route communications across heterogeneous enterprise systems.