Event Streaming Fundamentals
What is Event Streaming?
An introduction to event streaming, comparing it to traditional databases and request-response patterns.
Interview: Commonly asked in distributed systems and system design interviews to evaluate your understanding of real-time data flows.
Introduction
In traditional software development, we think of data as static records sitting in a database. When we want information, we query it. When we want to perform an action, we send a request.
Event Streaming flips this paradigm. In an event-driven system, data is treated as a continuous, never-ending stream of events happening in real-time. An event is simply a statement of fact: "a user clicked a button," "a payment was processed," or "a temperature sensor read 72 degrees." Event streaming is the practice of capturing these events from source systems in real-time, storing them durably, and routing them to destination systems instantaneously.
Why It Matters
Modern businesses run on real-time decisions. Waiting for a nightly batch job to process transactions or calculate fraud is no longer acceptable. Event streaming enables systems to react to changes instantly, scale to billions of events per day, and decouple services completely.
Real-World Analogy
Think of Kafka like a news ticker tape or a continuous conveyor belt in a factory. Just as a conveyor belt continuously transports items from builders (producers) to inspectors (consumers) without stopping, Kafka acts as an append-only log that continuously receives and delivers event messages in real-time.
How It Works
- Event Capture: An application or system generates an event representing an action (e.g., "Order #123 Placed").
- Publishing to Kafka: The Producer serializes the event data and publishes it to a specific named channel called a Topic.
- Durable Storage: Kafka writes this event to an append-only commit log on disk, distributing it across partitions for fault tolerance.
- Real-time Subscription: Consumers subscribe to the topic, continuously reading new events from the log as soon as they arrive.
Internal Architecture
| Component | Role | What Happens Without It |
|---|---|---|
| Producer | Publishes event records to topics. | No data enters the stream. |
| Broker | A server in the cluster that stores the log partitions and handles requests. | Events cannot be stored or distributed. |
| Consumer | Reads and processes events from topics. | Events sit in the log unprocessed. |
The internal architecture enables complete decoupling. Producers do not care who reads the data, how many consumers exist, or when they consume it. Kafka retains the events on disk, allowing consumers to read them at their own pace.
Practical Example
Here is a basic example of writing and producing an event in Java using the official Kafka client:
What just happened? We configured a Kafka client pointing to a local broker, created an event payload (click_login_button) labeled with key (user_123), and published it to the user-clicks topic.
Common Mistakes
| Mistake | Why Beginners Make It | How to Avoid It |
|---|---|---|
| Treating Kafka like a database | Expecting query capabilities (like SELECT SQL). | Use Kafka for streaming/pipelines; use an external DB for queries. |
| Not closing the producer | Forgetting to call close(), leading to resource leaks. | Always use try-with-resources or call close() explicitly. |
Quick Quiz
Q1: Which of the following best describes an event in Kafka?
A) A mutable row in a database table.
B) An immutable statement of fact about something that happened.
C) A request-response API payload.
D) A temporary queue message that is deleted after reading.
Answer: B — Events in Kafka are immutable records representing statements of fact.
Scenario-Based Challenge
Production Scenario:
Your microservices architecture relies on HTTP request-response. During flash sales, the inventory service crashes due to a massive spike in order requests, causing cascading failures in the billing and shipment services. How would you solve this using event streaming?
View SolutionImplement Kafka as an intermediary event log. When a user places an order, the Order service publishes an "OrderCreated" event to Kafka. The inventory, billing, and shipment services subscribe to this topic and consume events at their own pace (rate limiting / load leveling), resolving the spike issue and decoupling the systems.
Debugging Exercise
Identify the bug in this properties configuration:
The configuration lacks a value serializer (value.serializer). Kafka requires both key and value serialization classes to convert the payload to bytes. Without it, the producer will throw a ConfigException at initialization.
Interview Questions
1. Conceptual: What is the main difference between event streaming and traditional request-response?
Request-response is synchronous, point-to-point, and tightly coupled, where the caller blocks waiting for a response. Event streaming is asynchronous, decoupled, and event-driven, where producers publish events to an append-only log, and consumers process them independently.
2. Comparison: How does Kafka compare to a message queue like RabbitMQ?
RabbitMQ uses smart brokers and dumb consumers, deleting messages immediately after they are acknowledged. Kafka uses dumb brokers and smart consumers, persisting events in an append-only commit log on disk, allowing multiple consumers to replay events independently.
Production Considerations
In production, pay close attention to event schema design. Once an event is published to a log, it is immutable and cannot be updated. Implement schema validation (e.g., Avro or Protobuf) early to prevent "poison pills" (malformed messages) from crashing downstream consumers.