ReviseAlgo Logo

Producers & Message Delivery

Partitioning Strategies — Round Robin, Key-Based, Custom

How keys map to partitions and how custom partitioners work.

Introduction

Partitioning determines which broker node handles a message. Selecting the right partition assignment strategy allows developers to balance data distribution evenly across nodes while enforcing strict order execution guarantees.

Why It Matters

Message ordering in Kafka is only guaranteed within a single partition. If transaction records (e.g., account debits and credits) are routed to different partitions, a consumer might process a credit before a debit, violating business rules.

Real-World Analogy

Imagine a sorting conveyor belt in a distribution center. If you sort packages onto lanes randomly, workers at all lanes are equally busy (even distribution). If you sort packages by state codes (keys), packages from high-density states (e.g., California) will flood a single lane, causing a bottleneck (skew), while other lanes sit idle.

How It Works

When a producer sends a record, the partitioner assigns it to a partition:

  • Keyed Routing: Senders compute the partition index using MurmurHash2 on the message key modulo the number of partitions.
  • Default Partitioner (Sticky): If no key is provided, the producer groups messages into batch files per partition. The Sticky Partitioner keeps sending records to the active partition batch until it is filled, then switches partitions, reducing network connection swaps.
  • Custom Partitioner: Overrides the default selection algorithm by evaluating custom header fields or message contents.

Internal Architecture

The hashing partitioner uses MurmurHash2 on the key bytes. Crucially, the mathematical formula is Math.abs(Utils.murmur2(keyBytes)) % numPartitions. If the partition count changes, the modulus changes, routing keys to new nodes.

Visual Explanation

Below is an ASCII visualization of key-based hashing versus sticky round-robin distribution: 

Message Key "User-A" ──> Hash("User-A") % 3 = Partition 0 ──> Broker 1
Message Key "User-B" ──> Hash("User-B") % 3 = Partition 1 ──> Broker 2

Null-Key Batch 1 ────> Sticky Assign ────> Partition 2 ──> Broker 3
Null-Key Batch 2 ────> Sticky Assign ────> Partition 0 ──> Broker 1

Practical Example

Below is a Java implementation of a custom partitioner routing specific priority clients (e.g. VIP clients) to a dedicated partition channel:

Common Mistakes

  • Increasing partitions on active topics: Adding partitions changes the division denominator, breaking key consistency. Historical records for user User-123 remain in partition 1, while new ones are written to partition 3, scrambling ordering guarantees.
  • Using Low-Cardinality Keys: Using a key like Gender or Status (which has only 2 or 3 states) causes intense hot-spotting, routing all records to just 2 or 3 partitions while leaving others unused.

Quick Quiz

Q1: What is the danger of increasing a topic's partition count?

It scrambles key-based routing, routing messages with identical keys to different partitions.

Q2: How does the Sticky Partitioner improve performance?

It groups null-key records into partition batches in memory, minimizing network requests and reducing latency.

Scenario-Based Challenge

The Challenge: Multi-Tenant skew

You route tenant logs using tenant_id as the key. One enterprise tenant generates 80% of the logs, causing partition 4 to run out of disk space while other nodes are empty. How do you resolve this?

Solution Tip: Implement a custom partitioner that detects high-traffic tenants and routes their logs round-robin without keys, or split tenants into separate dedicated topics.

Debugging Exercise

Billing reports show invoices are being paid before they are created. You verify that all producer records use the invoice ID as the message key.

The Fix: Check if the partition count of the topic was recently increased, or verify if the producer is configured to retry writes while max.in.flight.requests.per.connection is greater than 1 without idempotency.

Interview Questions

1. How does Kafka partition null-key messages?

Modern Kafka uses sticky partitioning to batch null-key messages, filling up a single partition's batch buffer before rotating to another partition.

2. Can you write to a specific partition number directly?

Yes. Senders can specify the partition number directly when building the ProducerRecord constructor, bypassing partitioners entirely.

Production Considerations

Always keep key size minimal to avoid memory waste in the index files. Monitor key distribution to prevent broker hot-spots.