Distributed Transactions

In a monolithic system, maintaining data consistency is simple: the database locks tables and processes changes atomically within a single database transaction. But in a distributed microservices architecture, a single business action (like booking a trip) spans multiple independent services and databases. The challenge of keeping these operations all-or-nothing (atomic) over unreliable networks is the core problem of Distributed Transactions.

1. Learning Objectives

• Identify the "dual-write" problem and why standard database transactions fail in microservices.
• Understand the mechanics, drawbacks, and blocking states of Two-Phase Commit (2PC) and Three-Phase Commit (3PC).
• Learn the Saga Pattern and compare Choreography vs. Orchestration approaches.
• Master the Transaction Outbox Pattern for publishing reliable event updates without distributed locking.
• Implement a fully functional Saga Orchestrator with rollback compensations.

2. Prerequisites

To fully grasp this lesson, you should be familiar with:

• Databases and DBMS: Basic knowledge of database operations and record updates.
• ACID & BASE: Understanding of atomicity and eventual consistency.
• Transactions: Understanding of single-database transactions and rollback operations.
• Network Basics: Basic familiarity with HTTP, RPC, and network failure modes.

3. Why This Topic Matters

When you book a vacation on a modern website, three distinct services execute changes:

• Payment Service: Charges your credit card.
• Flight Service: Reserves your seat on the plane.
• Hotel Service: Book your hotel room.

If the Payment Service charges your card, but the Hotel Service fails because the room was sold out, the system cannot leave your card charged with no hotel room booked. Either all three operations must succeed, or any successful step must be undone. Coordinating this atomic behavior across separate databases is critical for any production-grade microservice architecture.

4. Real-world Analogy

Think of a Wedding Coordinator preparing a ceremony:

Phase 1 (Prepare): The coordinator calls the Caterer, the Florist, and the Venue Manager, asking: *"Are you ready to perform your service today?"* Each vendor checks their inventory. If all say *"Yes, ready,"* the coordinator moves to the next step. If even one vendor says *"No, my driver is sick,"* the coordinator aborts.

Phase 2 (Commit): Since all vendors agreed, the coordinator calls them back and says: *"Execute the plan!"* The caterer cooks, the florist delivers, and the venue opens. If a vendor goes missing after agreeing, the coordinator has a problem (blocking state), but otherwise, the wedding proceeds atomically.

5. Core Concepts

• Two-Phase Commit (2PC): A consensus protocol where a central Coordinator coordinates a transaction across multiple Participant nodes. It splits writes into a Prepare Phase (voting) and a Commit Phase (executing).
• Three-Phase Commit (3PC): An extension of 2PC that adds a Pre-Commit Phase to remove the blocking problem by allowing timeout-based commits if communication with the coordinator is lost.
• Saga Pattern: An eventually consistent workflow pattern. Instead of lock-based transactions, a Saga splits the workflow into a sequence of local transactions. If a step fails, the Saga coordinates compensating transactions in reverse order to clean up.
• Choreography-based Saga: Sagas coordinated dynamically via event broadcasting. Services listen to events from message brokers and execute local actions.
• Orchestration-based Saga: Sagas coordinated by a central Orchestrator class/service that explicitly commands each participant service.
• Transaction Outbox Pattern: Storing event messages in a local database outbox table within the same ACID transaction as the business update, then publishing them asynchronously to prevent dual-write inconsistencies.

6. Visualization

Two-Phase Commit (2PC) Flow

Saga Orchestration Rollback (Failure Scenario)

7. How It Works

Two-Phase Commit Lifecycle

1. Prepare Phase (Voting):
   • The Coordinator writes a start record to its local Write-Ahead Log (WAL) and sends a Prepare message to all participant nodes.
   • Each participant executes the transaction locally up to the point of committing, acquires locks on relevant records, writes its own WAL, and votes Yes (ready) or No (aborted).
2. Commit Phase (Execution):
   • If all participants vote Yes, the Coordinator writes a Commit record to its log and sends a Commit message to all nodes.
   • Participants finalize their updates, release all locks, and return an acknowledgment (ACK).
   • If any participant votes No or times out, the Coordinator writes an Abort record and commands all nodes to Rollback their local changes.

8. Internal Architecture

Distributed Transaction setups coordinate separate transactional systems:

• Transaction Coordinator: Tracks the overall transaction state and writes decisions to its log to survive crashes.
• Resource Managers (Participants): The local database engines (Postgres, MySQL) that process local transactions, manage locks, and listen to coordinator commands.
• Outbox Publisher / Message Broker: In event-driven setups (Sagas), a broker (like Kafka or RabbitMQ) acts as the asynchronous communication channel between colleague services.

9. Request Lifecycle

Let's trace a Trip Booking Saga orchestrating a Flight Booking and a Hotel Booking:

1. Initiation: Client sends a POST /trips request to the Trip Orchestrator Service.
2. Step 1 (Flight): The Orchestrator calls POST /flights/book to the Flight Service. The Flight Service reserves seat 14A, records it in its DB, and returns Success.
3. Step 2 (Hotel): The Orchestrator calls POST /hotels/book to the Hotel Service. The Hotel Service database is full and returns a 507 Insufficient Inventory error.
4. Compensate Step 1 (Rollback): The Orchestrator catches the error, looks up its transaction history, and calls POST /flights/cancel to the Flight Service. The Flight Service marks seat 14A as vacant.
5. Response: The Orchestrator returns a 503 Service Unavailable (Booking Failed) message to the client. All systems have converged back to their original states.

10. Deep Dive

The Dual-Write Problem

A common design flaw is attempting to update a database and publish an event to a message queue in the same method:

If the queue publish fails, the database remains updated, but downstream services (like Shipping or Inventory) never receive the event. If you reverse the order and publish first, the database write might fail, leading to ghost orders downstream.

Solution: Transaction Outbox Pattern

Instead of publishing directly, write the business data and an event record into a local database table named outbox within the same local transaction:

An asynchronous background publisher (polling the table or using Change Data Capture tools like Debezium reading the database log) reads the outbox records and forwards them safely to the message queue. This guarantees at-least-once delivery without blocking locking mechanisms.

11. Production Example

• Uber: Uses orchestration-based Sagas to coordinate rides. When a user requests a ride, the system coordinates payment authorization, driver allocation, and GPS tracking. If no driver accepts the ride within a timeout, the payment authorization is automatically rolled back via a compensating transaction.
• Stripe: Implements strict idempotency keys across all payment transactions. If network anomalies occur during api calls, Stripe client libraries retry safely. The server checks the database ledger; if the transaction exists, it returns the cached response, preventing double charges.

12. Advantages

• 2PC guarantees Strong Consistency: All nodes commit or none do. No stale reads or partial commit states.
• Sagas unlock Horizontal Scalability: Because they don't hold global database locks, Sagas scale horizontally across microservices.
• High Resiliency: Compensations allow eventually consistent rollbacks even if participant services experience transient downtime.

13. Limitations

• 2PC Blocking State: If the Coordinator crashes during the Commit Phase, participants must hold onto their locks indefinitely to preserve consistency, blocking other operations.
• No Read Isolation in Sagas: Sagas do not support ACID "Isolation." While a Saga is running, other clients can read partially committed states (e.g. money deducted but flight not booked yet), which can cause read anomalies.
• Complex Rollbacks: Compensating transactions are difficult to write and test. What if a compensation fails midway? You must implement monitoring, alerts, and manual overrides.

14. Trade-offs

15. Performance Considerations

• Latency: 2PC requires multiple network roundtrips (coordinator $\leftrightarrow$ participants) and synchronous disk flushes. This adds significant latency.
• Lock Saturation: Holding locks during 2PC limits database connection pools. High traffic will result in queue saturation.
• Outbox Publisher Polling Overhead: Reading the outbox table via polling queries (e.g. SELECT * FROM outbox WHERE status = 'PENDING') adds disk I/O. Prefer log-based CDC (like Debezium) which parses database binary transaction logs with zero query overhead.

16. Failure Scenarios

• Coordinator Crashes Mid-Commit in 2PC: Participants have voted Yes but never receive the Commit/Abort signal. They are in a blocking state and cannot release their locks. Recovery: Coordinator must read its WAL log on startup to determine the state and broadcast the final decision.
• Saga Compensating Transaction Fails: A compensation call returns a 500 Server Error. The orchestrator cannot complete the rollback. Recovery: The orchestrator must publish the failed step to a Dead Letter Queue (DLQ), generate an alert, and flag it for manual intervention or automated retry loops.

17. Best Practices

• Enforce Idempotency: All Saga steps and compensation endpoints must be idempotent. If a network timeout occurs, retrying the call must not create duplicate actions.
• Keep Sagas Short: Limit the number of steps in a Saga. If a workflow has more than 5 steps, check if some can be grouped or handled asynchronously outside the transaction scope.
• Use Outbox with Change Data Capture (CDC): Avoid polling outbox tables. Use tools that stream database commit logs to Kafka directly.

18. Common Mistakes

• Using 2PC in Microservices: Applying 2PC across independent microservices owned by different teams. This introduces tight runtime coupling and creates a distributed monolith.
• Mutable compensations: Writing compensations that depend on current state instead of checkpoint snapshots. A compensation must restore the *specific state* at the time of the transaction.
• Forgetting Idempotency Keys: Accepting transactions without client-provided idempotency keys, risking duplicate processing on retries.

19. Implementation: Saga Orchestrator

20. Interview Questions

21. Practice Exercises

• Easy: Design a choreography-based event routing table for a user registration flow triggering verification emails and discount coupon creation.
• Medium: Extend the Java/Python/C++ Saga Orchestrator code to support step-specific timeout triggers that trigger rollbacks automatically.
• Hard: Design the database schema and worker execution code for an Outbox Publisher handling duplicate event checks using a watermark system.

22. Challenge Problem

Design a Multi-Wallet Crypto Transfer System. Users can transfer assets between wallets on separate blockchains. The transaction must deduct assets from Blockchain A and credit Blockchain B. Because blockchains are independent networks with no common locking system, you cannot use 2PC. Describe the complete Saga architecture, details of the orchestrator state machine, how you will handle smart contract failures, and how you will handle a case where the credit step on Blockchain B fails but the deduction on Blockchain A cannot be easily undone (since blockchain states are immutable).

23. Summary

• Distributed transactions guarantee consistency across multiple databases or microservices.
• 2PC provides strong consistency but is blocking and limits scalability.
• Sagas provide eventual consistency, running independent local transactions with compensating rollbacks.
• The Transaction Outbox Pattern safely decouples business writes from event broker publishing, solving the dual-write problem.

24. Cheat Sheet

25. Quiz

26. Further Reading

• Research Paper: *"Sagas"* (1987) by Hector Garcia-Molina and Kenneth Salem.
• Debezium Docs: Change Data Capture patterns for Outbox routing.
• Uber Engineering Blog: *"Cadence: Uber's Workflow Orchestration Engine"* (for Orchestrator-based Sagas).

27. Next Lesson Preview

In the next lesson, we will explore Sharding. We will learn how to horizontally partition massive databases across multiple physical machines to bypass single-server memory and throughput ceilings!