ReviseAlgo Logo

Real-World System Design with Kafka

Event-Sourced Order Management System

Rebuilding transaction states using compacted order streams and microservices.

Introduction

Traditional order tables are prone to consistency drifts when database triggers or API calls fail. An Event-Sourced Order Management System designs order state as a sequence of immutable facts (e.g., OrderCreated, OrderPaid) recorded durably in Kafka. The active order state is built by materializing this log.

Why It Matters

When order modifications fail, diagnosing historical edits in CRUD tables is nearly impossible. Storing order steps in Kafka compacted topics creates an audit trail. Log compaction keeps the cluster clean by pruning intermediate events for closed transactions while guaranteeing retention of the absolute latest state of active orders.

Real-World Analogy

Think of an event-sourced order system like a construction project blueprint registry. Builders do not edit a single master blueprint directly. They file revisions (e.g., add partition wall, paint wall red). To see the current layout, you apply all historical revisions sequentially. Log compaction is like a cleanup clerk archiving intermediate paint changes and retaining only the final structural layout specifications for each room.

How It Works

Rebuilding order status views from compacted event streams operates as follows:

  1. Event Ingestion: Order services publish state events to an orders topic configured with a compact cleanup policy.
  2. Partition Keying: Events are keyed by orderId, placing all lifecycle changes for a specific order on the same partition.
  3. View Materialization: Consumer services read the partition from offset 0, aggregating the states in local caches (e.g., Redis or memory) to build active views.
  4. Compaction Runs: Kafka cleaners periodically scan log segments and delete older duplicate keys, retaining only the final lifecycle state.

Internal Architecture

Compacting Kafka logs relies on the **Log Cleaner** thread. A log segment is divided into a read-only **clean** section and an active **dirty** section. The cleaner builds an in-memory SkimpyOffsetMap of the latest keys in the dirty section. It then copies the clean segment, skipping any record whose key has a newer offset in the map. This keeps storage footprint minimal while preserving key timelines.

Visual Explanation

Below is an ASCII representation showing how log compaction purges intermediate keys to retain only the latest state of each order: 

Before Compaction:
[Key: A, v1 (Created)] [Key: B, v1 (Created)] [Key: A, v2 (Paid)] [Key: A, v3 (Shipped)]
                                                                           
After Compaction (Intermediate A states deleted):
[Key: B, v1 (Created)] [Key: A, v3 (Shipped)]

Practical Example

Here is a Spring Boot Java consumer configuration illustrating how to read a compacted topic and construct an in-memory synchronized map of active orders:

Common Mistakes

  • Publishing events without a key on compacted topics: Forgetting to set the message key. Compaction requires key attributes to identify duplicates. Null-key records cannot be compacted and will result in broker thread failures.
  • Publishing null values without understanding Tombstones: Sending a null value payload deletes the key from Kafka's log entirely during the next compaction pass, which should only be done for hard deletes.

Quick Quiz

Q1: How does a Kafka broker cleaner identify tombstones in compacted topics?

By checking for records with a null value payload. The broker retains this "tombstone marker" for a configured period before purging it completely.

Q2: Why are compacted topics useful for rebuilding in-memory cache states?

Because they store only the final state for each key, minimizing log read size and drastically reducing startup reconstruction durations.

Scenario-Based Challenge

The Challenge: Implementing a hard-delete compliance flow under GDPR

A customer requests that their billing records be deleted under GDPR. The order management events are stored on a compacted Kafka topic. If compaction only deletes old keys, how do you guarantee that historical entries for this specific customer are physically removed from all log segments?

Solution Tip: Publish a record with the customer's ID as the key and a null payload (a Tombstone record). Configure delete.retention.ms to define how long brokers retain the tombstone before purging it. When the log cleaner runs, it will delete the tombstone and all older customer keys from the active log segments.

Debugging Exercise

You notice that your compacted topic's size is growing continuously, and checking the logs reveals that the cleaner thread has skipped segments, leaving multiple duplicate keys intact.

The Fix: The active segment cannot be compacted. Adjust the segment properties: set segment.ms or segment.bytes to force segment roll rotations more frequently. Ensure log.cleaner.enable is set to true in server.properties and increase log.cleaner.threads to allocate processing resources.

Interview Questions

1. Can you change a topic's cleanup policy from delete to compact dynamically in production?

Yes, using the admin config command: kafka-configs.sh --alter --entity-type topics --entity-name orders --add-config cleanup.policy=compact. The cleaner thread will start processing segments on its next cycle.

2. What are the key configuration properties regulating log compaction intensity?

Monitor log.cleaner.min.cleanable.ratio (default 0.5; controls minimum dirty ratio before compaction triggers) and log.cleaner.dedupe.buffer.size (size of the key deduplication memory pool).

Production Considerations

Avoid using compaction on high-cardinality transient topics where keys never repeat, as this adds processing overhead without saving space. Allocate sufficient memory to the cleaner's dedupe buffer to handle large keys.