Message Queues vs Event Logs — Key Differences

Introduction

Distributed messaging systems generally fall into two categories: Message Queues (like RabbitMQ) and Event Logs (like Apache Kafka). While both serve to decouple producers and consumers, their underlying storage models and ingestion workflows differ fundamentally. Understanding these differences is critical for choosing the right system architecture.

Why It Matters

Choosing the wrong messaging system causes data loss, limits concurrency, and creates operations headaches. For example, if you require the ability to audit historical transactions or rebuild app state by replaying old messages, a standard message queue is useless because it deletes messages immediately upon delivery. Conversely, using an event log to route individual tasks with complex priority rules can lead to partition blockages and high latency.

Real-World Analogy

How It Works

The lifecycle of a message differs under each system:

1. Message Queue: Senders push messages to an exchange. The exchange routes messages to a queue. The queue pushes messages to workers. Once a worker acknowledges processing, the broker deletes the message immediately.
2. Event Log: Senders write events sequentially to a partition log. The broker appends the records to disk and keeps them. Consumers pull data sequentially, keeping track of their own read position (offset) without changing the broker's data.

Internal Architecture

The contrasting design philosophies dictate different internal structures:

• Smart Broker vs Dumb Broker: Message queues use smart brokers that track consumer connections, manage message locks, acknowledge state, and coordinate garbage collection. Event logs use dumb brokers that simply manage disk files, leaving offset tracking to smart consumers.
• Memory vs Sequential Disk: Queues maintain transient indexes in memory. Logs write directly to sequential disk structures, leveraging page caches and the sendfile system call for zero-copy data transfer.

Visual Explanation

The diagram below highlights the structural differences: a transient queue deleting records post-acknowledgment vs an immutable, offset-driven log retaining history.

Practical Example

Below is a Java snippet showing how to reset a consumer's offset programmatically in Kafka to replay historical events, something impossible in standard queues:

Common Mistakes

• Treating Kafka like RabbitMQ: Attempting to delete individual messages on demand. Kafka logs are immutable; you cannot target and delete a single record in the middle of a partition.
• Allowing slow consumers to block logs: Since log processing is sequential, if one record takes long to process, it blocks all records behind it in that partition (Head-of-Line blocking). Traditional queues avoid this by distributing items out-of-order.

Quick Quiz

Scenario-Based Challenge

Debugging Exercise

A consumer group is falling behind in an event log. The developer tries to scale worker instances to 10, but notices only 4 instances are active, while the other 6 remain completely idle.

The Fix: Event logs allocate partition ownership to consumers in a group. If the topic has only 4 partitions, Kafka will assign one partition to each of the 4 consumers. The remaining 6 consumers have no partitions to read from and will sit idle. To resolve this, increase the partition count of the topic to 10.

Interview Questions

Production Considerations

In production, monitor disk storage closely when using event logs since data accumulates over time. Ensure proper log compaction is set up for key-value streams, or configure appropriate retention intervals (e.g. 7 days or 50GB) to prevent disk saturation.