Event Streaming

A topic is a named stream of events. Topics are divided into partitions — each partition is an independent, ordered, immutable log. Partitions are the unit of parallelism: more partitions = more consumers can read concurrently. The partition count determines max throughput.

A consumer group is a set of consumer instances sharing the same group_id. Kafka assigns each partition to exactly one consumer in the group. Each consumer tracks its offset (last processed message position) in Kafka. Restarting a consumer resumes from the last committed offset.

Kafka stores events for a configurable retention period (hours to forever). Any consumer can seek to any offset — including offset 0 (the beginning of time). This enables: debugging by replaying production traffic, rebuilding materialized views from scratch, and bootstrapping new consumers.

Kafka guarantees strict ordering within a partition, but not across partitions. Use a consistent partition key (e.g., user_id, order_id) to ensure all events for the same entity go to the same partition — maintaining entity-level ordering.

Kafka Transactions (KIP-98) enable exactly-once processing: read-process-write cycles that commit atomically. The producer can write to multiple partitions atomically; if the producer crashes mid-write, the partial write is rolled back. Critical for financial and inventory systems.

Frameworks for processing Kafka streams in real time: filtering, aggregation, joining, windowing. Kafka Streams runs inside the application (no separate cluster). Apache Flink is a separate cluster with more powerful stateful processing.