ReviseAlgo Logo

Kafka Architecture Deep Dive

Topics and Partitions — How Data is Organized

Understanding partition scaling, order guarantees, and load balancing.

Introduction

Topics are the logical streams in Kafka, while Partitions are the physical implementation on disk. Understanding how data is partitioned, replicated, and ordered across brokers is critical for designing scalable event streaming pipelines.

Why It Matters

Kafka guarantees message ordering only within a single partition. If you require chronological event sequencing (e.g., processing order edits after creations), you must design a partition strategy that maps related events to the exact same partition. Failing to do so leads to out-of-order execution bugs downstream.

Real-World Analogy

Imagine a busy grocery store checkout. The store represents a Topic. To process high customer volume, the store sets up 10 registers (Partitions). Senders are customers choosing a line. If a family splits their items across different registers, the items will check out out-of-order. If they stay in the same line, ordering is guaranteed.

How It Works

When a producer sends a record, the partitioner assigns it to a partition:

  • Keyed Messages: Senders apply a hash function (Default: MurmurHash2) on the key: partition = hash(key) % totalPartitions. Messages with matching keys land in the same partition.
  • Keyless Messages: Senders use round-robin or sticky partitioning (batching efficiency) to distribute load evenly.

Internal Architecture

Partitions scale out by being spread across multiple brokers. To prevent data loss, partitions are replicated:

  • Replication Factor: The total number of copies of a partition. If RF = 3, 1 broker holds the leader replica, while 2 brokers hold follower replicas.
  • In-Sync Replicas (ISR): The set of follower brokers that are fully caught up with the leader. Only nodes in this list are eligible to become partition leader if the current leader fails.

Visual Explanation

The diagram on the page illustrates partition distribution: Senders route events via key hashes to individual broker partition leaders, while followers sync data in the background.

Practical Example

Below is a Java custom Partitioner implementation, routing VIP customer orders to partition 0 and other customer orders round-robin:

Common Mistakes

  • Changing partition count on active topics: Since key-based routing relies on the modulus of total partitions, increasing partitions breaks the hash calculation. Subsequent writes for key user-123 will land in a different partition, scrambing transaction order!
  • Uneven partition distribution (Hot Partitioning): Using keys with low cardinality (e.g. country codes). If 90% of your users are from US, one partition will receive 90% of the traffic, creating hot broker spots.

Quick Quiz

Q1: How does Kafka guarantee record ordering across the entire topic?

It doesn't. Kafka only guarantees ordering within a single partition. If topic-wide order is required, you must use a single partition (which limits concurrency to 1).

Q2: What occurs if a partition leader fails and the ISR list is empty?

If unclean leader election is disabled, the partition becomes unavailable. If enabled, a non-synced follower is elected leader, causing data loss.

Scenario-Based Challenge

The Challenge: Designing a Multi-Tenant Log

You are building a chat application. You have a topic called room-messages. Some rooms have millions of messages (e.g., announcement feeds), while others have only a few (private chats). If you partition by room_id, how do you prevent partition skew?

Solution Tip: Using room_id directly as the key will cause hot spots. To solve this, implement key salting (e.g. prefixing keys room-id with a random hash) for massive rooms to spread their updates across partitions, while using a different topic or metadata index for aggregation.

Debugging Exercise

A billing system reports that invoice updates are being processed before invoice creation events. Investigation shows both events were sent with the invoice ID as the message key.

The Fix: Check the partition count of the topic. If it was recently increased, the hash calculation for the invoice key shifted. The creation was routed to partition 1, while the update was routed to partition 2. The fix requires either resetting offsets to replay or ensuring key-to-partition bindings are maintained during partition migrations.

Interview Questions

1. Explain the relationship between Partitions and Consumer instances.

Partitions map to consumers. Within a group, a partition can belong to only one consumer instance. A single consumer can read from multiple partitions.

2. What is the role of the Partition Leader?

The Leader handles all read and write requests from clients. Replicas act as consumers, pulling log logs from the leader to stay synchronized.

Production Considerations

Keep partition counts balanced. Avoid exceeding 4,000 partitions per broker, as excessive partitions increase replication overhead and degrade latency.