Event-Driven Architecture Patterns
The Saga Pattern — Distributed Transactions Without 2PC
Orchestrating multi-service payments and orders via event compensation workflows.
Introduction
In monolithic architectures, ensuring transactional safety across multiple database tables is achieved using standard ACID transactions. However, in microservice architectures, databases are distributed. The Saga Pattern is an architectural pattern designed to manage distributed transactions using a sequence of local transactions, avoiding the blockages of traditional two-phase commit protocols.
Why It Matters
Standard Two-Phase Commit (2PC) blocks databases during execution, rendering microservices slow and prone to deadlocks. The Saga pattern executes local transactions on separate microservice databases step-by-step. If any local transaction fails, the Saga coordinates execution of a series of Compensating Transactions to roll back changes in reverse order.
Real-World Analogy
Think of the Saga pattern like booking a holiday trip. You book a flight, then book a hotel, and finally book a rental car. If the rental car booking fails because no cars are available, you cannot simply perform a database rollback. You must contact the hotel and flight companies to cancel your existing bookings (compensating actions) to get a refund.
How It Works
Sagas can be structured using two core design approaches:
- Choreography: Services publish events, and other services react to them. There is no central controller, keeping services highly decoupled but harder to track.
- Orchestration: A central coordinator service tells each participant which local transaction to execute, managing success, errors, and compensating steps.
Internal Architecture
Because Sagas do not lock database rows globally, they do not guarantee ACID Isolation (the "I" in ACID). Other microservices can read temporary updates before the entire saga completes. To prevent anomalies, developers must implement countermeasures such as semantic locks, versioning tokens, or executing transactions in a non-reversible sequence until the pivot point.
Visual Explanation
Below is an ASCII representation of an Orchestration Saga executing a sequence of payments and order approvals, with a compensation loop on failure:
Successful Flow:
[Order Service] ──> (Start Saga) ──> [Payment Service] ──> [Inventory Service] (Complete)
Compensating Flow (Failure at Inventory Step):
[Inventory Fail] ──> [Orchestrator] ──> (Refund Payment) ──> (Cancel Order Booking)
Practical Example
Here is a Java example demonstrating a simplified Order Saga Orchestrator coordinate workflow step sequences:
Common Mistakes
- Not implementing Idempotency in compensating handlers: A compensation step (e.g. refunding money) must be idempotent. If network issues trigger multiple retries of a refund, the consumer could refund the payment multiple times.
- Designing non-undoable steps before pivot points: Sending a confirmation email to a customer before confirming that the payment and inventory reservation steps have fully succeeded.
Quick Quiz
Q1: What is the primary difference between Choreography and Orchestration Sagas?
Choreography uses decentralized, reactive event flows. Orchestration relies on a central controller managing transaction states explicitly.
Q2: What is a compensating transaction in a Saga workflow?
A transaction designed to logically undo the mutations performed by a previous successful transaction when subsequent steps fail.
Scenario-Based Challenge
The Challenge: Avoiding double refunds in a billing compensation service during network blips
Your payment service receives RefundPayment events to compensate for failed inventory checks. During network disruptions, Kafka retries delivering the refund events. How do you design your payment database model to prevent sending multiple refunds to the bank API?
Solution Tip: Maintain a processed_refunds log table with a unique constraint on saga_id + transaction_id. Before executing the bank API call, insert the log record in the database transaction to prevent concurrent duplicate updates.
Debugging Exercise
An inventory service fails to reserve items and emits a compensation trigger event, but the payment service fails to receive it, leaving the customer charged for items they will not receive.
The Fix: Implement a Saga State Machine with persistent storage. Save the saga's state (IN_PROGRESS, FAILED, COMPENSATING) in a database before executing operations. Run a background cleanup scheduler that finds Sagas stuck in COMPENSATING or IN_PROGRESS state and re-dispatches the compensation events until they succeed.
Interview Questions
1. Why do Sagas lack database isolation, and how do you handle it?
Because changes are committed in local transactions immediately before the whole saga finishes. You handle it using semantic locks (setting order state to "Pending Approval" instead of "Completed") or versioned checks.
2. What is a pivot transaction in a Saga?
The transaction that acts as the "point of no return". If the pivot transaction succeeds, the saga cannot be compensated; it must proceed forward to completion.
Production Considerations
Use industrial-strength orchestrator engines (like Temporal, Camunda, or Spring State Machine) for complex sagas to avoid building custom retry state managers.