ReviseAlgo Logo

Consumers & Consumer Groups

Consumer Groups and Partition Assignment

Horizontal scaling and partition sharing algorithms.

Introduction

A Consumer Group is a collection of consumers sharing the workload of reading messages from a topic. Kafka coordinates partition assignments across members of a group to ensure that each partition is consumed by only one group member at any given time, enabling horizontal scale.

Why It Matters

Properly designing consumer groups and scaling partition counts is fundamental for high-performance streaming. If a group has more consumers than there are partitions in a topic, the extra consumers sit idle, wasting hardware resources.

Real-World Analogy

Imagine a team of mail sorting clerks (Consumer Group) processing letters from a cabinet containing 10 drawers (Partitions). If you hire 5 clerks, each clerk is assigned 2 drawers. If you hire 10 clerks, each clerk gets 1 drawer. If you hire 12 clerks, 2 clerks sit idle with no drawers assigned, since only one clerk can access a drawer at a time to prevent conflicts.

How It Works

The group assignment protocol flows as follows:

  1. Registration: When a consumer starts, it registers with the Group Coordinator broker using its group.id.
  2. Leader Election: The Group Coordinator elects the first consumer to join the group as the "Group Leader."
  3. Assignment Protocol: Senders delegate assignment calculations to the Group Leader. The leader runs the configured partition assignment strategy (e.g., Range, Round Robin, or Sticky) and reports back to the Coordinator.
  4. State Sync: The Group Coordinator propagates assignments to all group members.

Internal Architecture

The group assignment strategies run client-side. The coordinator broker manages group state changes (e.g. Empty, PreparingRebalance, Stable), while the actual partition sharing logic runs inside the client library itself.

Visual Explanation

Below is an ASCII diagram showing partition sharing among members of a consumer group: 

Topic "Orders" (4 Partitions):
[Partition 0]   [Partition 1]   [Partition 2]   [Partition 3]
      │               │               │               │
      ▼               ▼               ▼               ▼
[Consumer 1]    [Consumer 1]    [Consumer 2]    [Consumer 3]

(Consumer Group "orders-processor": Consumer 1 gets 2 partitions, others get 1)

Practical Example

Below is a configurations setup detailing how to register a consumer under a group and choose the CooperativeStickyAssignor strategy:

Common Mistakes

  • Deploying more consumers than partitions: If your topic has 4 partitions and you scale your Kubernetes consumer deployment to 8 pods, 4 pods will remain completely idle, consuming memory and network resources without doing work.
  • Running unrelated microservices with the same group.id: Assigning the identical group.id to separate apps (e.g., payment logs and inventory logs) causes them to share partition lanes, leading to missing events in both services.

Quick Quiz

Q1: What happens if a consumer group has 10 members, but the topic only has 8 partitions?

8 members will be assigned one partition each, and 2 members will sit completely idle.

Q2: Which partition assignment strategy minimizes partition migrations when consumer instances are restarted?

The Sticky (or Cooperative Sticky) assignor, as it preserves existing assignments where possible.

Scenario-Based Challenge

The Challenge: Multi-Topic subscription alignment

A consumer group subscribes to two topics: orders (3 partitions) and shipments (3 partitions). The group coordinator runs the default RangeAssignor. You notice that Consumer 1 is assigned partition 0 of BOTH topics, leaving other nodes under-utilized. How do you rebalance this?

Solution Tip: Change partition.assignment.strategy to org.apache.kafka.clients.consumer.RoundRobinAssignor to distribute partitions evenly across all members.

Debugging Exercise

You view metrics and see partition 2 is lagging behind, while partitions 0 and 1 are caught up. Senders verify keys are distributed evenly, and all consumer instances are running.

The Fix: This suggests the consumer instance assigned to partition 2 is running slow (due to high CPU, garbage collection pauses, or database thread exhaustion). Thread dumps and host metrics should be checked on that specific worker node.

Interview Questions

1. What is the role of the Group Coordinator broker?

It is the broker designated to manage a consumer group's state, receive heartbeats, coordinate partition reassignments, and store committed offsets.

2. Can a consumer read from multiple partitions of a topic?

Yes. If there are fewer consumers than partitions, some consumers will be assigned multiple partitions.

Production Considerations

Run with CooperativeStickyAssignor to enable rolling deployments without interrupting the entire cluster pipeline.