Distributed Transactions & Resilience
The Problem with Distributed Transactions — Why 2PC Fails
Analyzing performance blocks and failure modes of Two-Phase Commit protocols.
Interview: Distributed system trade-offs. Expect questions on ACID vs BASE properties, CAP theorem constraints, the execution steps of 2PC, and why XA transactions degrade performance.
Introduction
In monolithic architectures, database transactions are simple. You can execute multiple SQL queries, and if any step fails, the database engine executes a single rollback command to protect database consistency. In a microservices architecture, however, data is distributed across multiple isolated databases.
Maintaining ACID consistency across physical boundaries is a major challenge. The traditional solution, **Two-Phase Commit (2PC)** (often implemented via XA transactions), attempts to coordinate a unified commit across databases. However, 2PC is a blocking protocol that introduces latency and reduces reliability.
Why It Matters
Distributed transactions do not scale. Understanding the performance limits and failure modes of 2PC helps you design resilient, eventually consistent systems using event-driven architectures.
The Two Phases of 2PC
The protocol relies on a central coordinator to manage databases (cohorts):
• 1. Prepare Phase: The coordinator contacts all cohorts, requesting them to execute the transaction locally, lock resources, and vote: COMMIT (if successful) or ABORT (if failed).
• 2. Commit Phase: If all cohorts vote COMMIT, the coordinator writes a commit record to its log and sends a commit command to cohorts. If any cohort votes ABORT (or fails to reply), the coordinator sends a rollback command to all cohorts, freeing resources.
Why 2PC Fails to Scale
Two-Phase Commit introduces three critical problems:
- The Blocking Protocol Problem: If the coordinator crashes mid-way through the commit phase, cohorts are left in a suspended state. They must hold database locks indefinitely to maintain consistency, causing downstream request queues to back up.
- High Latency and Lock Contention: Resources remain locked across all databases throughout the entire two phases. This reduces throughput and increases the likelihood of deadlocks.
- Network Dependency: A single network partition or slow cohort forces the entire transaction to wait or rollback, degrading cluster availability.
Quick Quiz
Q1: What is the main structural vulnerability of the Two-Phase Commit (2PC) protocol during network partitions?
A) It does not support SQL databases.
B) It is a blocking protocol. If the coordinator crashes after cohorts vote, cohorts must hold locks indefinitely, blocking other transactions.
C) It deletes transaction logs automatically.
D) It forces data replication across nodes.
Answer: B — The blocking nature of 2PC makes it vulnerable to coordinator and network failures, locking up databases and degrading system availability.
Scenario-Based Challenge
Production Scenario:
Your enterprise payment system executes distributed transactions across three database instances (Order, Inventory, Billing). Latency spikes from 40ms to 5 seconds under load. You trace it to lock contention. How do you resolve this performance degradation?
View Solution
To eliminate lock contention and improve scaling:
1. **Decommission 2PC/XA Transactions:** Remove atomic distributed transaction layers from your database connections.
2. **Implement the Saga Pattern:** Replace atomic transactions with a sequence of local transactions. Each service commits its changes locally and publishes events to coordinate subsequent steps.
3. **Accept Eventual Consistency (BASE):** Accept that databases will be temporarily inconsistent during execution, and use compensating workflows to resolve failures.
Interview Questions
1. Explain the difference between ACID and BASE models in system design.
ACID (Atomicity, Consistency, Isolation, Durability) guarantees strong consistency, ensuring database records are always updated atomically (common in monoliths). BASE (Basically Available, Soft State, Eventual Consistency) prioritizes availability over instant consistency, allowing databases to be temporarily out of sync while resolving consistency eventually (common in distributed systems).
Production Considerations
Avoid XA transactions in high-throughput cloud environments. They decrease system availability, block connection pools during network drops, and create single points of failure in your architecture.