Pub/Sub
Broadcasting transactions to topics where multiple subscribers consume duplicates concurrently.
What you'll learn
- Topic-Based vs Content-Based Routing
- Fan-Out Pattern (SNS → SQS)
- Message Durability
- At-Least-Once Delivery & Idempotency
- Push vs Pull Model
- Dead Letter Topic (DLT)
TL;DR
Broadcasting transactions to topics where multiple subscribers consume duplicates concurrently.
Visual System Topology
Pub/Sub Network Handshake Flow
Concept Overview
Publish-Subscribe (Pub/Sub) is a messaging pattern where publishers send messages to named channels (topics) without knowing which subscribers will receive them, and subscribers express interest in topics without knowing which publishers produce messages. This decouples producers and consumers both in time and space.
Pub/Sub differs from a point-to-point message queue in a critical way: in a queue, one consumer processes each message; in pub/sub, every active subscriber receives a copy of every message. This enables fan-out patterns where a single event (e.g., "order shipped") simultaneously triggers multiple downstream workflows (send email, update inventory, push mobile notification, generate invoice).
Key implementations: Amazon SNS (serverless fan-out to SQS, Lambda, HTTP), Google Pub/Sub (at-least-once, scalable), Apache Kafka (log-based, durable, replay), Redis Pub/Sub (in-memory, no durability, extremely fast), MQTT (IoT protocol for constrained devices).
Key Architectural Pillars
Topic-Based vs Content-Based Routing
Topic-based: subscribers subscribe to named topics; all messages on a topic are delivered to all subscribers. Content-based: the broker filters messages based on message attributes/content; subscribers declare predicates, not topics. More flexible but more complex.
Fan-Out Pattern (SNS → SQS)
A single SNS topic fans out to multiple SQS queues. Each downstream service has its own SQS queue, processed independently. If one service is slow or down, it doesn't affect others. Each service processes at its own pace from its own queue.
Message Durability
Redis Pub/Sub: zero durability — if no subscriber is connected when a message is published, the message is lost. SNS: messages are buffered if the subscriber is an SQS queue or Lambda. Kafka: messages are durably stored in a replicated log for a configurable retention period.
At-Least-Once Delivery & Idempotency
Pub/Sub systems typically guarantee at-least-once delivery — a message may be delivered more than once (on retries). Subscribers must be idempotent: processing the same message twice must produce the same result as processing it once.
Push vs Pull Model
Push: the broker delivers messages to subscribers automatically (SNS, Google Pub/Sub HTTP). Pull: subscribers explicitly poll the broker for messages (SQS, Kafka consumer groups). Push provides lower latency; pull provides better backpressure control.
Dead Letter Topic (DLT)
Messages that fail delivery after N retries are published to a dead letter topic. Enables isolation of poison messages and debugging without blocking the main topic.