ReviseAlgo Logo

Kafka Streams

KStream vs KTable — Key Differences

Understanding streams of facts vs tables of latest updates.

Introduction

In Kafka Streams, data streams are modeled using two primary abstractions: KStream and KTable. A KStream represents an append-only stream of independent events, whereas a KTable represents a changelog stream representing the latest state of keys.

Why It Matters

Failing to distinguish between KStream and KTable lead to serious design errors. If you use a KStream to track user profile status, you must scan millions of obsolete historical update events to retrieve a user's current city. Using KTable for transactional payments, on the other hand, risks dropping intermediate transfers if updates arrive too fast.

Real-World Analogy

Think of a checking account transaction history: "+$100", "-$30", "+$20". This append-only sequence is a KStream. Think of your current account balance: "$90". This represents the cumulative state of updates and is a KTable. You can reconstruct the balance (KTable) from transactions (KStream), showing the stream-table duality.

How It Works

The dual systems behave as follows:

  • KStream (Facts): If you publish (Alice, Boston) followed by (Alice, Paris), a consumer reading a KStream sees both records as distinct facts.
  • KTable (States): If you publish the same updates, a consumer querying the KTable only sees the active state (Alice, Paris). The previous city is overwritten.

Internal Architecture

Under the hood, a KTable leverages a local RocksDB state store to perform key lookups and handle updates (upserts/tombstones). If a record has a key and a null value, the KTable treats it as a deletion (Tombstone) and removes the key from the state store.

Visual Explanation

Below is the interactive visual diagram showing how the KStream and KTable dualities process updates over time:

Practical Example

Here is a Java snippet showing how to create a KStream and a KTable from topics and join them to enrich transactions with active user profiles:

Common Mistakes

  • Joining KStreams without windowing: Attempting to join two KStreams without defining a time window causes an exception, because streams are infinite and cannot be kept in memory forever.
  • Using KStream for master data tables: Reading a static configurations topic into a KStream instead of a KTable forces your application to duplicate lookup memory and write complex lookup logic.

Quick Quiz

Q1: What is the primary functional difference between a KStream and a KTable?

KStream represents an append-only stream of updates (inserts), while KTable represents the latest update status per key (upserts).

Q2: How does a GlobalKTable differ from a regular KTable?

A KTable only loads partitions assigned to the local instance's tasks, while a GlobalKTable replicates all topic partitions to every instance.

Scenario-Based Challenge

The Challenge: Joining location logs with delivery transactions

You receive courier location coordinates every 10 seconds, and delivery requests periodically. You want to tag every delivery request with the courier's *most recent* location. How do you design this?

Solution Tip: Read courier location coordinate events into a KTable (since you only care about the latest location coordinates per courier ID). Read delivery request events as a KStream. Perform a Stream-Table join to look up location states instantaneously on transaction arrival.

Debugging Exercise

You execute a Stream-Table join, but no events are outputting to the enriched topic, even though match keys are published to both topics.

The Fix: Check partition counts. For co-partitioning rules in Kafka Streams, both join topics must have the *same number of partitions*. If they differ, the record key will hash to different broker node tasks, causing join failures. Re-create topics with matching partition counts.

Interview Questions

1. Explain what Stream-Table Duality means.

It is the concept that a stream can be viewed as a table (by taking the sum/latest updates over time), and a table can be viewed as a stream (by capturing its modification logs).

2. What is co-partitioning, and why is it a requirement for regular KStream-KTable joins?

Co-partitioning means that both topics have matching partition counts and use the same key partitioner. This ensures matching keys reside on the same broker task, avoiding cross-network join calls.

Production Considerations

Use cache.max.bytes.buffering to control KTable output updates. High buffer cache values deduplicate records before writing them to downstream changelogs, reducing traffic.