ReviseAlgo Logo

Event Streaming Fundamentals

Pub/Sub Model Explained

Deep dive into the Publish-Subscribe pattern, message brokers, topics, and subscription semantics.

Introduction

The Publish-Subscribe (Pub/Sub) model is a messaging pattern where senders (publishers) do not programmatically address messages directly to specific receivers (subscribers). Instead, published messages are characterized into channels or categories called topics. Senders publish events to these topics without knowing which subscribers—or how many of them—are listening. Subscribers express interest in one or more topics and automatically receive all messages published to those topics.

Why It Matters

In modern distributed systems, tight coupling between services creates fragile, slow, and hard-to-scale architectures. If an Order Service needs to directly call the Inventory, Billing, and Shipping APIs synchronously, any failure or slow response in downstream systems blocks checkout. The Pub/Sub model eliminates this coupling. The Order Service simply publishes an OrderPlaced event to a topic, and immediately responds to the user. Downstream services consume the event asynchronously.

Real-World Analogy

Imagine subscribing to a podcast. The creator (publisher) records an episode and uploads it to their feed (topic). They do not know your email, location, or name, nor do they send individual emails with audio files to millions of listeners. As a listener (subscriber), your podcast player checks the feed and pulls down the new episode. When a new listener subscribes, they get the same feed. The creator publishes once, and an unlimited number of listeners consume it independently at their own convenience.

How It Works

In a classic Pub/Sub model, the workflow follows these steps:

  1. Publish: Senders write events containing keys, values, and timestamps to the broker.
  2. Categorize: Senders specify a target topic for each message.
  3. Subscribe: Consumers register their interest in one or more topics with the broker.
  4. Deliver: The broker ensures the message is broadcasted to all active subscriptions matching that topic.

Internal Architecture

Kafka modifies the traditional pub/sub model to achieve extreme scalability. In traditional pub/sub systems (like RabbitMQ or ActiveMQ), the broker keeps track of subscriptions and pushes messages to consumers. Once a message is acknowledged, the broker deletes it.

Kafka uses a log-centric pull model:

  • Message Persistence: Events are written to an append-only commit log on disk and persist even after consumption (determined by retention policies).
  • Consumer Groups: Multiple consumers can join a single group to share the workload of a topic (acting like a queue), while different consumer groups read the exact same message stream independently (acting like pub/sub).
  • Smart Consumers: Consumers track their own reading position via Offsets stored in a special internal topic (__consumer_offsets), making the broker stateless and highly performant.

Visual Explanation

The diagram below illustrates how multiple publishers stream events to a topic, and how Kafka load-balances partitions among consumers inside a group while broadcasting the complete event stream across different consumer groups.

Practical Example

Below is a simplified Java implementation using the official Apache Kafka Client library, demonstrating how a Producer publishes an order event, and how two independent Consumers (representing different services) subscribe to receive it.

1. Publisher (Order Service):

2. Subscriber A (Inventory Service):

3. Subscriber B (Notification Service):

Common Mistakes

1. Sharing the same Group ID when broadcasting is intended

If you deploy the Inventory Service and Notification Service with the same group.id, Kafka treats them as a single logical application. They will split the messages between them (queuing model) instead of both receiving every message (pub/sub model).

2. Misconfiguring auto.offset.reset

If a new subscriber group is registered and there is no active offset, using the default latest config will make the consumer miss all historic messages. Set it to earliest if processing historical events is necessary.

Quick Quiz

Q1: In Kafka, if you want two downstream microservices to receive the same published order event, how should their consumer groups be configured?

They must use completely different group.id values (e.g., inventory-group and email-group). This instructs Kafka to track offsets for each service independently.

Q2: How does Kafka support historical message playback for new subscribers, unlike traditional brokers like RabbitMQ?

Kafka saves events as an immutable log on disk. When a new consumer group joins, it can set its offset to the beginning of the log (offset 0) and consume the historical stream from the start.

Scenario-Based Challenge

The Challenge: Designing a Promotion Engine Fan-Out

You are designing a high-traffic gaming platform. When a user levels up, a LevelUpEvent is published. Three engines must process this event: a Rewards Engine (awards coins), a Leaderboard Engine (updates rankings), and a Push Notification Engine (sends alerts). The Push Notification Engine is slow and occasionally goes offline. Explain how you would structure the Kafka topics, partitions, and consumer groups to ensure that the slow Notification Engine does not lag or block updates in the Rewards or Leaderboard systems.

Solution Tip: Give each engine its own group.id. Because Kafka uses a pull model and stores offsets separately per consumer group, the latency or offline status of the Push Notification group has absolutely no impact on the read position or speed of the other two consumer groups.

Debugging Exercise

A developer reports that they set up a local testing consumer to watch messages in payments-topic. However, as soon as they started their test client, their production billing application stopped receiving half of the transaction events.

The Broken Config:

The Solution: Since the developer's client shared the group.id with the production billing service (billing-service), Kafka treated the new client as another instance of the billing cluster. Kafka rebalanced the partitions, sending half of the payments to the developer's terminal (where they were lost/ignored) instead of the billing service. The fix is to change the developer client's group ID to a unique name (e.g. local-dev-billing-group-xyz).

Interview Questions

1. What is the fundamental difference between point-to-point queuing and Publish-Subscribe?

Point-to-point queuing delivers a message to exactly one consumer (one-to-one). Publish-subscribe broadcasts a message to all active subscriptions (one-to-many).

2. How does Apache Kafka support both queuing and Pub/Sub simultaneously?

Through the Consumer Groups mechanism. Multiple consumer instances in the same group load-balance partitions (queuing). Multiple distinct consumer groups receive all events from the same topic partitions independently (pub/sub).

3. What is the impact of a stateless broker in Kafka's Pub/Sub model?

In traditional systems, the broker must track which consumers have received which messages, causing scaling issues as subscriber count grows. In Kafka, consumers track their own offsets, allowing the broker to remain lightweight, handle high concurrency, and stream data at disk write speed.

Production Considerations

When using Kafka Pub/Sub in production, remember:

  • Fan-Out Broker Network Limits: Every new consumer group reading from a topic increases the read I/O on the partition leaders. If you have 50 consumer groups fan-outing a high-volume topic, brokers can face network saturation. Use caching or consider intermediate aggregations.
  • Schema Evolution & Poison Pills: Because events are immutable, once a publisher sends a malformed record, it remains in the log. If a subscriber crashes trying to parse it, it will block partition progress. Use a Schema Registry (e.g., Confluent/Apicurio with Avro or Protobuf) to enforce schema contracts at the publisher level.
  • Log Retention Policies: Configure retention bytes and time wisely (e.g., log.retention.hours=168 for 7 days) to ensure subscribers have enough time to recover and catch up on offsets in case of prolonged downtime.