Distributed Transactions & Resilience
The Outbox Pattern — Guaranteed Event Publishing Without 2PC
Saving entities and events locally in a single database transaction, with separate poll publishes.
Interview: Transactional messaging patterns. Expect questions on dual-write issues, Outbox table design, Change Data Capture (CDC) integrations, and ensuring at-least-once delivery.
Introduction
In event-driven microservices, applications must update local database tables and publish corresponding events to a message broker (like Kafka). Executing these two actions sequentially is risky: if the database commit succeeds but the network fails before publishing the event, downstream services remain out of sync.
The **Transactional Outbox Pattern** resolves this. Instead of publishing directly to the broker, the application saves both the entity and the event payload to the same database inside a single local ACID transaction. An independent event relay then reads the outbox table and publishes events to the broker.
Why It Matters
The Outbox pattern guarantees **At-Least-Once Delivery** without the latency and resource locking overhead of distributed 2PC/XA transactions.
The Outbox Pattern Workflow
The process relies on database transaction boundaries:
• 1. Atomic Store: The application service updates the primary table (e.g. ORDERS) and inserts an event record into an OUTBOX table inside the same transaction block.
• 2. Event Relay Ingestion: An independent publisher process (e.g. a polling scheduler or a Change Data Capture tool like Debezium) reads new records from the OUTBOX table.
• 3. Publish & Mark: The relay publishes events to the message broker (e.g. Kafka). Upon receiving confirmation from the broker, it deletes or marks the outbox record as processed.
Practical Example
Let's write a Spring Service method that saves an order and its outbox event atomically:
Quick Quiz
Q1: Why is publishing events to a message broker inside a @Transactional block considered an anti-pattern?
A) It is not supported by Spring Boot.
B) It holds database connections open during network calls to the broker, reducing throughput, and can result in publishing events even if the transaction rolls back.
C) It encrypts database logs.
D) It forces the application to use NoSQL databases.
Answer: B — Network calls inside transactions delay commits and hold connections open. If the transaction rolls back after the network call, the event is still sent, causing inconsistency.
Scenario-Based Challenge
Production Scenario:
You deploy a polling scheduler to read unprocessed records from the Outbox table and publish them to Kafka. Under load, you scale your application to 4 replicas. Downstream services complain they are receiving duplicate events. What caused this, and how do you resolve it?
View Solution
To prevent duplicate publishing from multiple replicas:
1. **Identify the Cause:** A simple polling query (e.g. SELECT * FROM OUTBOX WHERE PROCESSED = FALSE) executed by multiple replicas concurrently returns the same set of records. Each replica tries to publish the same events, causing duplication.
2. **Implement Pessimistic Locking:** Update the query to lock rows during selection, preventing other replicas from reading them:
SELECT * FROM OUTBOX WHERE PROCESSED = FALSE FOR UPDATE SKIP LOCKED
This assigns rows exclusively to a single replica, skipping already locked rows.
3. **Migrate to Log-Based CDC (Debezium):** Use Debezium to read database transaction logs (WAL/Binlog) directly. Debezium stream acts as a single reader, eliminating read locks and polling query overhead.
Interview Questions
1. How does Change Data Capture (CDC) optimize the Outbox pattern over polling queries?
Polling queries execute SQL statements (like SELECT ... FOR UPDATE) repeatedly, increasing database CPU load and I/O latency. CDC tools (like Debezium) read raw database transaction logs (e.g., PostgreSQL WAL, MySQL Binlog) directly. This is non-blocking, consumes minimal database resources, and processes updates with sub-millisecond latencies.
Production Considerations
Because the Transactional Outbox pattern guarantees At-Least-Once delivery, message consumers must be designed to be idempotent to handle duplicate events safely.