Network Partitions & Split Brain
Detecting physical routing ruptures, isolating split node sub-graphs safely, and preventing independent sub-clusters from executing conflicting master operations through mutual consensus rules.
What you'll learn
- Network Partition
- Split Brain Problem
- Quorum-Based Consensus
- Fencing Tokens
- STONITH (Shoot The Other Node In The Head)
- Partition Detection & Timeout Tuning
TL;DR
Detecting physical routing ruptures, isolating split node sub-graphs safely, and preventing independent sub-clusters from executing conflicting master operations through mutual consensus rules.
Visual System Topology
Network Partitions & Split Brain Network Handshake Flow
Concept Overview
A network partition is a failure in a distributed system where nodes can no longer communicate with each other, dividing the cluster into two or more disconnected sub-groups. Partitions occur due to cable cuts, router failures, AZ outages, or transient packet loss storms. They are not exceptional events — in large-scale systems, partitions happen several times per year.
The Split Brain Problem is the most dangerous consequence of a network partition: both partitions believe they are the authoritative primary and continue accepting writes independently. When the partition heals, the cluster discovers it has two conflicting versions of the data with no clear "correct" history. This is catastrophic for databases where writes in both partitions cannot be trivially merged.
Solving network partitions is the core challenge of distributed consensus. The key insight: a partition-tolerant system must either sacrifice consistency (allow divergence during the partition) or sacrifice availability (refuse requests from the minority partition).
Key Architectural Pillars
Network Partition
A network failure that divides a cluster into disconnected sub-groups. Nodes within each group can communicate internally but cannot reach nodes in other groups. Neither sub-group can distinguish between "the other group is down" and "the other group is unreachable but alive."
Split Brain Problem
When a network partition causes two or more nodes to simultaneously believe they are the primary/leader, they accept writes independently, creating diverging data histories. When the partition heals, the cluster must reconcile conflicting writes — and in many cases, data loss is unavoidable.
Quorum-Based Consensus
The primary defense against split brain: require a majority of nodes (quorum = N/2 + 1) to agree before executing any write. With 5 nodes, quorum is 3. During a 2+3 split, only the 3-node partition can form a quorum and accept writes. The 2-node partition is read-only or rejects writes.
Fencing Tokens
A monotonically increasing token issued to the current leader. Every write to shared resources must include the fencing token; the storage system rejects writes with stale tokens. This ensures writes from an old leader are rejected after a new leader is elected.
STONITH (Shoot The Other Node In The Head)
A fencing mechanism that physically kills a node suspected of split-brain to guarantee it cannot continue accepting writes. Used in high-availability database clusters. Extreme but reliable.
Partition Detection & Timeout Tuning
Nodes detect partitions by the absence of heartbeat signals. The timeout value is a critical trade-off: too short → false positives (transient network hiccup triggers split brain recovery); too long → real failures take too long to detect.