ReviseAlgo Logo

Kafka Architecture Deep Dive

Kafka Core Components — Brokers, Topics, Partitions, Offsets

The anatomy of a Kafka cluster and how messages are stored.

Introduction

An Apache Kafka cluster is made up of several key abstractions: Brokers (physical servers), Topics (logical categories), Partitions (parallel commit log lanes), and Offsets (sequential record pointers). Together, these components allow Kafka to operate as a distributed, partition-tolerant, and high-throughput streaming engine.

Why It Matters

Without these core components, scaling a real-time system is impossible. If a topic was not divided into partitions, it could only live on a single broker disk, limiting your write throughput and storage capacity to a single machine's physical limits. Partitions enable horizontal scale, brokers handle physical storage, and offsets track subscriber progress.

Real-World Analogy

Think of a Kafka topic like a large municipal archive room. - Brokers are the physical archive buildings containing the records. - Topics are the broad index categories (e.g., Land Deeds). - Partitions are the individual filing cabinet drawers within that category. Multiple clerks can check out files from different drawers simultaneously (parallel scaling). - Offsets are the numbered index cards inside each drawer. Each file reviewer has a personal notepad (committed offset) recording the last index number they reviewed.

How It Works

The lifecycle of data routing flows as follows:

  1. Publishing: Senders publish events to a topic, supplying an event key and payload.
  2. Partition Routing: Senders hash the key to assign the event to a specific partition.
  3. Appending to Log: Senders write the event to the partition log leader broker.
  4. Consuming: Receivers pull events sequentially from the partition, updating their offset.

Internal Architecture

Topics are divided into multiple partitions distributed across brokers. Each partition is replica-backed:

  • Partition Leaders: Every partition has one active broker designated as "Leader." Senders write to and receivers read from this leader.
  • Partition Followers: Standby brokers that pull logs from the leader to stay synchronized. If the leader crashes, a follower becomes the new leader.

Visual Explanation

The diagram on the page illustrates the connection between publishers, partitions distributed across brokers, and independent consumers tracking offsets.

Practical Example

Below is a Java snippet showing how to use the AdminClient to programmatically describe the partitions, leaders, and replicas of a topic:

Common Mistakes

  • Using a Single Partition: Fails to distribute data across brokers, defeating the purpose of horizontal scalability and limiting concurrency to one consumer.
  • Publishing without Keys: Causes round-robin routing which scrambles message ordering across partitions. Always use a key if transaction sequence matters.

Quick Quiz

Q1: If a topic has 5 partitions, what is the maximum number of active consumers in a single consumer group?

5 consumers. Any additional consumers in that group will sit idle since Kafka enforces a maximum of one consumer per partition within a group.

Q2: Where are the actual partition logs stored on a broker?

As raw append-only files on the broker's local filesystem, located under the directory specified by log.dirs in configuration.

Scenario-Based Challenge

The Challenge: High-Frequency Stock Telemetry Sizing

You are building a stock price ingest stream handling 100,000 updates per second. Senders hash updates using stock ticker symbols (e.g. AAPL, MSFT) as keys. The consumer processing engine has a capacity limit of 10,000 updates/sec per worker. How many partitions do you design the topic with?

Solution Tip: With 100,000 writes/sec and worker capacity of 10,000 writes/sec, you require a minimum of 10 parallel consumers. Therefore, you must provision at least 10 partitions. To handle future scaling overhead, setting it to 12 or 16 partitions is recommended.

Debugging Exercise

An application logs show consumer lag growing continuously. The developer tries to scale the consumer pool from 4 instances to 8 instances, but lag does not decrease.

The Solution: Describe the topic using AdminClient or CLI. If the topic has only 4 partitions, adding 4 more consumer instances will result in those new workers sitting idle, leaving the processing speed unchanged. To fix this, increase the topic's partition count or optimize the consumer's throughput (e.g., batch processing or thread pools).

Interview Questions

1. How does a broker coordinate partition leader election when a node crashes?

The active Controller broker elects a new partition leader from the partition's In-Sync Replica (ISR) list and writes this metadata state change to the cluster log.

2. Can an offset number be reused within a partition?

No. Offsets are strictly sequential and monotonically increasing. Once offset 100 is written, that offset is immutable. If a message is deleted via retention, offset 100 is retired.

Production Considerations

In production, always configure a replication factor of at least 3 for critical data topics. Ensure that the brokers are distributed across different availability zones to prevent single points of failure.