ReviseAlgo Logo

Event-Driven Architecture Patterns

Outbox Pattern — Guaranteed Event Publishing

Combining database operations and Kafka publication reliably without distributed transactions.

Introduction

In microservice architectures, a service often needs to update its local database and simultaneously publish an event to a message broker (like Kafka). The Transactional Outbox Pattern is a design pattern that guarantees atomic execution of both database updates and event publication, resolving consistency issues.

Why It Matters

Updating a database and calling Kafka are two independent network operations. If the service updates the database but crashes before sending the event, downstream consumers remain unaware of the change. If it sends the event first but the database commit fails, the event represents false data. The Outbox pattern guarantees at-least-once event publication.

Real-World Analogy

Think of the Outbox pattern like a physical office mail tray. Instead of the accountant running to the local post office every single time they type up an invoice (and risking finding the post office closed), they place the stamped envelope in the office "Outbox" tray. A dedicated mail runner regularly collects letters from the tray and delivers them safely to the post office.

How It Works

Guaranteed event propagation via the Outbox pattern works in the following order:

  1. Local Transaction: The service creates or mutates a business entity (e.g. saves an Order row) and inserts an event record into a dedicated outbox table within the same database transaction.
  2. Relational Commit: The database commits both rows atomially. If either write fails, the entire transaction rolls back.
  3. Event Harvesting: A separate, independent process (e.g. Debezium or a database poller) reads new records from the outbox table.
  4. Broker Publication: The harvester publishes the event to Kafka, then flags or deletes the processed record from the outbox table.

Internal Architecture

To harvest events without introducing polling lag, production systems use Change Data Capture (CDC) tools like Debezium. Debezium acts as a database replica, reading database binary transaction logs (e.g., PostgreSQL WAL or MySQL binlog) directly. When it sees an insert in the outbox table, it stream-publishes the event to Kafka immediately, keeping database overhead extremely low.

Visual Explanation

Below is an ASCII representation of the Outbox flow displaying how the database transaction boundary encapsulates the outbox write: 

[Application Service] ──> (Single Transaction Start)
                               │  - Write Order Table
                               │  - Write Outbox Table
                               ▼
  [Database Commit]    ──> Writes written atomically to WAL log
                               │
                               ▼
  [Debezium CDC Engine]──> Reads WAL log insert ──> Publishes event to Kafka

Practical Example

Here is a Spring Boot Java implementation showing a transactional business logic write saving both entities within a single database session boundary:

Common Mistakes

  • Polling tables via custom SELECT queries at high frequency: Issuing SELECT * FROM outbox every 100ms. This introduces heavy lock contention and degrades core database transaction performance. Use CDC tools instead.
  • Assuming exactly-once delivery on consumers: The Outbox pattern guarantees at-least-once delivery. Harvester retries can publish duplicates to Kafka. Consumers must be designed to be idempotent.

Quick Quiz

Q1: Why is CDC (Change Data Capture) preferred over SQL polling for reading the outbox table?

CDC reads directly from database transaction logs asynchronously, bypassing the query execution engine and avoiding resource-heavy table locks.

Q2: What happens if the outbox harvester crashes after publishing to Kafka but before deleting the outbox row?

The next harvester instance will read the row again and publish it, resulting in duplicate events. Downstream consumers must be idempotent to handle this.