Distributed Locks

TL;DR

Avoiding concurrent transactional race states using global lock databases (Redlock, Chubby, ZooKeeper).

Visual System Topology

Distributed Locks Execution Topology

Inbound Node Ingests request

Distributed Locks Engine Processes operations

Target Replica Updates state

Concept Overview

A distributed lock is a mutual exclusion mechanism that coordinates access to shared resources across multiple independent processes in a distributed system. Unlike a single-process mutex, a distributed lock must be acquired via a network call to a shared store, making it subject to network partitions, clock skew, and process crashes.

Distributed locks are notoriously difficult to implement correctly. A naive implementation based on "check-then-set" in Redis is vulnerable to race conditions. Even "correct" implementations like Redlock are controversial because clock jumps, GC pauses, and network delays can cause the lock holder to believe it holds the lock while it has actually expired.

Common use cases: preventing duplicate cron job execution, limiting concurrent payment processing, coordinating leader election in small clusters, and ensuring exactly-once processing of expensive operations.

Key Architectural Pillars

Redis SETNX + Expiry (Basic Pattern)

The simplest distributed lock: SET key value NX PX timeout (set only if not exists, with expiry). The expiry prevents lock starvation if the lock holder crashes. The key problem: if the holder is paused longer than the TTL (GC pause, network issue), the lock expires and another process acquires it, creating two "holders."

Example: SET lock:payment:user123 $token NX PX 5000 — acquires a 5-second lock for processing user 123's payment.

Fencing Tokens

A monotonically increasing token issued by the lock service with each successful acquisition. All downstream operations (database writes, file writes) must include the token; the storage system rejects operations with stale tokens. This prevents a GC-paused old lock holder from corrupting state after its lock expired.

Example: Lock acquired with token=42. Token 43 is issued to a new holder. The old holder tries to write with token=42; the database rejects it because it has already seen token 43.

Redlock Algorithm

Martin Kleppmann's critique of Redlock: with 5 independent Redis nodes, acquire the lock from a majority (3+) within a TTL window. If acquired from majority, the lock is valid for (TTL - time_elapsed). Controversial because it doesn't use fencing tokens and can be violated by GC pauses.

Example: Acquire lock on redis-1, redis-2, redis-3 in sequence. If 3/5 succeed within 50ms (with 5000ms TTL), lock is valid for 4950ms.

Consensus-Based Locks (etcd, ZooKeeper)

Production-grade distributed locks use consensus algorithms (Raft in etcd, ZAB in ZooKeeper) to manage locks. The lock is a key in the consensus store. etcd leases provide TTL-based expiry with automatic renewal. Fencing tokens are built-in (etcd revision numbers are monotonically increasing).

Example: Kubernetes uses etcd leases for leader election of controllers. The "kube-controller-manager" leader holds an etcd lease; the lease's revision number is the fencing token.

Lock Granularity

Coarse-grained locks (single global lock) eliminate parallelism. Fine-grained locks (per-resource locks) maximize parallelism but increase lock management overhead. Sharding the lock key space achieves a middle ground.

Example: Instead of one global lock for all payments, use locks per user_id: lock:payment:user_123, lock:payment:user_456. 1M users → 1M potential parallel lock acquisitions.

Lock Lease vs. Renewal

Short TTLs reduce blast radius when a holder crashes (lock recovers quickly) but require periodic renewal for long-running operations. Long TTLs allow long operations without renewal but block progress for long periods if the holder crashes. Heartbeat-based renewal is the standard solution.

Example: An ETL job acquires a lock with a 30-second TTL and renews it every 10 seconds. If the job crashes, the lock expires within 30 seconds.

Foundations of Distributed Systems

Networking & Communication

Data Storage & Databases

Performance & Scaling

System Architecture

Data Processing Systems

Reliability & Operations

Security

Trade-offs & Interview Thinking

Real-world Case Studies

Visual System Topology