ReviseAlgo Logo

Kafka Streams

KStream vs KTable — Key Differences

Understanding streams of facts vs tables of latest updates.

Introduction

In Kafka Streams, data streams are modeled using two primary abstractions: KStream and KTable. A KStream represents an append-only stream of independent events, whereas a KTable represents a changelog stream representing the latest state of keys.

Why It Matters

Failing to distinguish between KStream and KTable lead to serious design errors. If you use a KStream to track user profile status, you must scan millions of obsolete historical update events to retrieve a user's current city. Using KTable for transactional payments, on the other hand, risks dropping intermediate transfers if updates arrive too fast.

Real-World Analogy

Think of a checking account transaction history: "+100", "-30", "+20". This append-only sequence is a KStream. Think of your current account balance: "90". This represents the cumulative state of updates and is a KTable. You can reconstruct the balance (KTable) from transactions (KStream), showing the stream-table duality.

How It Works

The dual systems behave as follows:

  • KStream (Facts): If you publish (Alice, Boston) followed by (Alice, Paris), a consumer reading a KStream sees both records as distinct facts.
  • KTable (States): If you publish the same updates, a consumer querying the KTable only sees the active state (Alice, Paris). The previous city is overwritten.

Internal Architecture

Under the hood, a KTable leverages a local RocksDB state store to perform key lookups and handle updates (upserts/tombstones). If a record has a key and a null value, the KTable treats it as a deletion (Tombstone) and removes the key from the state store.

Visual Explanation

Below is the interactive visual diagram showing how the KStream and KTable dualities process updates over time:

Practical Example

Here is a Java snippet showing how to create a KStream and a KTable from topics and join them to enrich transactions with active user profiles:

Common Mistakes

  • Joining KStreams without windowing: Attempting to join two KStreams without defining a time window causes an exception, because streams are infinite and cannot be kept in memory forever.
  • Using KStream for master data tables: Reading a static configurations topic into a KStream instead of a KTable forces your application to duplicate lookup memory and write complex lookup logic.

Quick Quiz

Q1: What is the primary functional difference between a KStream and a KTable?

KStream represents an append-only stream of updates (inserts), while KTable represents the latest update status per key (upserts).

Q2: How does a GlobalKTable differ from a regular KTable?

A KTable only loads partitions assigned to the local instance's tasks, while a GlobalKTable replicates all topic partitions to every instance.

Scenario-Based Challenge

The Challenge: Joining location logs with delivery transactions

You receive courier location coordinates every 10 seconds, and delivery requests periodically. You want to tag every delivery request with the courier's most recent location. How do you design this?

Solution Tip: Read courier location coordinate events into a KTable (since you only care about the latest location coordinates per courier ID). Read delivery request events as a KStream. Perform a Stream-Table join to look up location states instantaneously on transaction arrival.

Debugging Exercise

You execute a Stream-Table join, but no events are outputting to the enriched topic, even though match keys are published to both topics.

The Fix: Check partition counts. For co-partitioning rules in Kafka Streams, both join topics must have the same number of partitions. If they differ, the record key will hash to different broker node tasks, causing join failures. Re-create topics with matching partition counts.

Interview Questions

1. Explain what Stream-Table Duality means.

It is the concept that a stream can be viewed as a table (by taking the sum/latest updates over time), and a table can be viewed as a stream (by capturing its modification logs).

2. What is co-partitioning, and why is it a requirement for regular KStream-KTable joins?

Co-partitioning means that both topics have matching partition counts and use the same key partitioner. This ensures matching keys reside on the same broker task, avoiding cross-network join calls.

Production Considerations

Use cache.max.bytes.buffering to control KTable output updates. High buffer cache values deduplicate records before writing them to downstream changelogs, reducing traffic.