ReviseAlgo Logo

Consumers & Consumer Groups

Consumer Rebalancing — Causes, Types, and Impact

Eager vs Cooperative Sticky assignment and rebalance storms.

Introduction

A Rebalance is the process where the Group Coordinator reassigns partition ownership across group members. Rebalances occur when membership changes, ensuring that all available partitions are distributed evenly.

Why It Matters

Traditional rebalances are Stop-the-World events. When a member joins or leaves, all consumers must stop polling, release their partition assignments, and wait for reassignment. In large clusters, this rebalance storm can halt processing for minutes, creating latency spikes.

Real-World Analogy

Think of a school classroom where students are reading books. If a new student enters the room, the teacher shouts "Stop reading! Everyone hand in your books!" (eager rebalance). The teacher redistributes the books and tells everyone to resume. Under cooperative rebalancing, the teacher tells only one student to hand over a spare book, leaving others to read uninterrupted.

How It Works

The lifecycle of a group rebalance triggers as follows:

  1. Trigger: A consumer joins the group, crashes, or fails to send heartbeats within session.timeout.ms.
  2. Eager Rebalance: All members stop consuming and release their partitions. The group reassigns partitions from scratch.
  3. Cooperative Sticky Rebalance: The coordinator identifies which partitions need to migrate and instructs only the affected consumers to release them. Unaffected members keep reading.

Internal Architecture

Cooperative rebalances work by executing multiple incremental rounds. The group coordinator designates partition migration paths, letting active nodes remain online during rolling deployments.

Visual Explanation

Below is a comparison of Eager vs Cooperative Sticky rebalancing: 

[Eager Rebalance]
Consumer A (Reading P0) ──> STOP ──> Release P0 ──┐
Consumer B (Reading P1) ──> STOP ──> Release P1 ──┼──> Reassign ──> Resume Both

[Cooperative Sticky]
Consumer A (Reading P0) ──> Keeps Reading P0 ────────────────────────> Continues
Consumer B (Reading P1, P2) ──> Releases P2 ──> Reassign P2 to C ──> Resume C

Practical Example

Below is a Java configuration enabling the cooperative sticky assignor protocol to prevent stop-the-world pauses:

Common Mistakes

  • Triggering False Rebalances via long Processing delays: If your business logic takes longer than max.poll.interval.ms (default 5 minutes), the consumer client library will voluntarily leave the group, triggering a rebalance.
  • Not listening for rebalance notifications: When using manual commits, failing to register a ConsumerRebalanceListener to commit pending offsets before partition release leads to duplicate processing.

Quick Quiz

Q1: What parameter defines the maximum time allowed between poll calls before a consumer is evicted from the group?

max.poll.interval.ms.

Q2: How does Cooperative Sticky assignment improve rolling deployments?

It migrates only the partitions belonging to the shutting-down pod, leaving other pods to process logs.

Scenario-Based Challenge

The Challenge: Eliminating rebalance storms in a Kubernetes cluster

Every time you roll out a microservice deployment, lag surges. How do you optimize configurations?

Solution Tip: Configure cooperative sticky assignor and increase group.initial.rebalance.delay.ms on the broker to let pods register before calculating assignments.

Debugging Exercise

You observe that a consumer node is repeatedly evicted from the group, rejoins, and gets evicted again, creating a continuous rebalance loop.

The Fix: The consumer is hitting garbage collection pauses that block heartbeats for longer than session.timeout.ms, or batch processing exceeds max.poll.interval.ms. Inspect GC logs and increase timeout settings.

Interview Questions

1. What triggers a consumer group rebalance?

A new consumer joining the group, an existing consumer leaving or crashing, or a consumer group subscribing to a topic whose partition count changes.

2. What is the purpose of ConsumerRebalanceListener?

To execute tasks (like saving offsets or closing database connections) when partitions are revoked or assigned.

Production Considerations

Transition legacy clusters to CooperativeStickyAssignor to avoid latency spikes during deployments.