ReviseAlgo Logo

Event Streaming Fundamentals

When to Use Kafka (and when NOT to)

Understand the specific use cases where Kafka shines, and where traditional systems are a better fit.

Introduction

Apache Kafka is a powerhouse for real-time event streaming, but it is not a direct replacement for traditional relational databases, key-value stores, or lightweight task queues. Utilizing Kafka in scenarios it was not designed for introduces high architectural complexity and scaling challenges.

Why It Matters

Implementing Kafka requires managing clustering, partition sizing, disk provisioning, offsets, and consumer coordination. Choosing Kafka for low-throughput, simple point-to-point messaging creates immense operational overhead. Conversely, not using Kafka for streaming telemetry or massive audit trails leads to performance bottlenecks on relational databases.

Real-World Analogy

Using Kafka is like operating a high-speed commercial cargo railway. It is incredibly efficient for moving massive tons of goods continuously between major hubs. However, you wouldn't lay railway tracks to deliver a single pizza to a customer's house; a local scooter courier (lightweight queue/database) is cheaper and faster.

How It Works

When reviewing a candidate architecture, evaluate the system requirements against Kafka's strengths:

  • High Throughput: Are you processing millions of events per second?
  • Multi-Consumer Fanout: Do multiple services need to read the same event stream?
  • Order Preservation: Do events need to be processed in strict chronological order per key?
  • Replay Capability: Do you need to replay historical states?

Internal Architecture

Kafka's internal design prioritizes sequential disk writes and block file handling over individual message operations:

  • No Random Indexing: Kafka does not index individual messages by key for random point lookups like a database. To find a message, you must seek to an offset and scan sequentially.
  • Partition Locking: Concurrency is bound to partitions. You cannot have more active consumers in a group than partitions, limiting granular worker scaling.

Visual Explanation

The flowchart below outlines the typical decision paths for selecting Kafka versus traditional relational/NoSQL databases or transient message brokers.

Practical Example

Below is a configuration snippet demonstrating a production-grade high-throughput Producer setup in Java, optimized for batching telemetry data:

Common Mistakes

1. Using Kafka as a database

Attempting to query Kafka partitions directly for specific user accounts like a SQL database. Kafka is a log, not an indexed datastore; scanning partitions for single keys is extremely slow.

2. Using Kafka for slow worker tasks

If processing a single message takes several minutes (e.g. video rendering), it blocks the entire partition partition from moving forward, leading to consumer heartbeats timing out and causing partition rebalances.

Quick Quiz

Q1: Which of the following is an anti-pattern for Kafka?

Using Kafka for direct, random key-value queries from an external client application (use Redis or DynamoDB instead).

Q2: Why does Kafka struggle with massive partition counts (e.g. 100,000 partitions on a single broker)?

Each partition maps to physical files on disk. Too many partitions lead to high metadata management overhead, file descriptor exhaustion, and slow cluster recovery times.

Scenario-Based Challenge

The Challenge: Selecting Architecture for IoT Metrics

You are building a fleet tracking system. 50,000 delivery trucks send GPS coordinates every 5 seconds. The analytics engine must process these updates in real-time, while a database stores the latest known location of each truck. How do you incorporate Kafka in this architecture?

Solution Tip: The raw stream of GPS logs represents high-throughput event data, which is ideal for a Kafka topic. An analytics engine (like Flink or Kafka Streams) consumes this stream. However, to display the "current location" to users, write the computed states to a key-value store (e.g., Redis) which supports rapid query lookups.

Debugging Exercise

A consumer group keeps disconnecting and reconnecting continuously. The logs show: CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member.

The Fix: The consumer is taking too long to process a batch of records, exceeding the max.poll.interval.ms configuration. Kafka assumes the consumer has died, removes it from the group, and triggers a rebalance. To fix this:

  1. Increase max.poll.interval.ms.
  2. Or decrease max.poll.records (the batch size per poll).
  3. Or delegate the record processing to a separate thread pool instead of blocking the main poll loop.

Interview Questions

1. When would you choose RabbitMQ over Kafka?

When you need flexible routing keys (e.g., wildcard exchanges), fine-grained message priority, individual message acknowledgments, and don't need replay capability.

2. Can Kafka handle transactions?

Yes, Kafka supports transactional writes (Exactly-Once Semantics/EOS) via a transaction coordinator, allowing producers to write atomically across multiple partitions.

Production Considerations

Always size your clusters with KRaft/Zookeeper limitations in mind. Do not over-provision partitions (keep them under 4,000 per broker broker) to avoid long replication recovery times during broker failures.