ReviseAlgo Logo

Kubernetes Storage

Running Databases on Kubernetes — Trade-offs and Patterns

Evaluating operators, StatefulSets, storage performance, and risks.

Interview: Be prepared to debate stateful vs stateless architecture. Detail the differences between local NVMe and cloud-network storage (latency vs availability). Explain what Database Operators do beyond StatefulSets (failovers, replication, backups). Address split-brain mitigation.

Last Updated: June 15, 2026 • 15 min read

Introduction

Kubernetes was originally designed for scaling stateless microservices. Running stateful workloads like databases (PostgreSQL, MySQL, MongoDB, Cassandra) on Kubernetes remains one of the most debated topics in infrastructure engineering.

With the maturity of StatefulSets, persistent storage interfaces, and custom Database Operators, running databases on Kubernetes is now a highly viable pattern. However, it introduces significant architectural trade-offs, storage latencies, and operational complexities that must be carefully managed.

Why It Matters

Deciding whether to run databases on Kubernetes (e.g. self-managed Postgres on StatefulSets) versus using cloud-managed database services (like AWS RDS or Google Cloud SQL) directly impacts your team's operational overhead, cloud spending, latency profile, and disaster recovery SLA.

Real-World Analogy

Running stateless apps is like renting hotel rooms (if one has plumbing issues, you simply checkout and move to another room instantly). Running databases is like owning a historic house (it has a solid foundation, specialized wiring, and valuable heirlooms. If the house catches fire, you cannot just move to a identical room next door—you must carefully extinguish the fire and restore the structures to avoid permanent historical loss).

Storage Latency Options in Production

Storage choice dictates database read/write throughput (IOPS) and failover recovery speed:

  • Local PVs (Physical NVMe SSDs): Direct physical attachment to the host node. Provides the lowest possible latency and maximum IOPS. Drawback: Bound to a single physical server. If the host hardware crashes, the database is unavailable until the node recovers. Useful for highly replicated DBs (Cassandra, Elasticsearch) where the application layer handles node failure.
  • Network-Attached Block Storage (AWS EBS gp3, GCP PD): Disk resides on a separate network storage area network. High availability and ease of detachment/reattachment to another node. Drawback: Network latency overhead. Capped at IOPS/throughput limits which can choke write-heavy transactional systems.
  • Shared Distributed Storage (Ceph, GlusterFS): High data redundancy across nodes. Drawback: Significant network bottlenecks, double-replication overhead, and high risk of filesystem locks.

StatefulSets and Headless Services

Databases require stable network identities and unique volume attachments. Standard Deployments cannot guarantee this. Kubernetes solves this via StatefulSets:

  • Stable Network Identity: Pods are named sequentially (e.g. postgres-0, postgres-1). A Headless Service (a service manifest with clusterIP: None) creates unique DNS A-records (e.g. postgres-0.postgres-db.namespace.svc.cluster.local) pointing directly to each Pod, allowing replicas to discover the primary instance.
  • Dedicated Storage Mapping: StatefulSets use a volumeClaimTemplates block. Each created Pod gets its own independent PVC (e.g., data-postgres-0 binds only to postgres-0). If postgres-0 crashes and schedules on another node, it retains its identity and mounts data-postgres-0 again.

The Need for Database Operators

StatefulSets manage pod sequencing and volume mappings, but they have zero understanding of database-specific operations. A StatefulSet doesn't know if a replica is lagging behind the primary, how to safely trigger a failover promotion, or how to perform a consistent database backup.

This gap is filled by Operators (e.g., CloudNativePG, Zalando Postgres Operator, MongoDB Community Operator). Operators extend the Kubernetes API with Custom Resource Definitions (CRDs). A custom controller runs continuously, managing backups, replica sync loops, primary elections, and scaling operations automatically.

Split-Brain Mitigation

In database clusters, a split-brain occurs when a network partition isolates nodes. Two database instances may believe they are the legitimate primary leader, concurrently accepting writes. When the network heals, their datasets conflict, resulting in catastrophic data corruption.

Kubernetes mitigates this via fencing and consensus-driven failover. Operators utilize Raft or Consul consensus loops. If a primary node loses quorum, it is automatically demoted or terminated (STONITH - "Shoot The Other Node In The Head") before a replica is promoted.

Practical Example

Here is a PostgreSQL StatefulSet manifest using dynamic volume templates and a Headless Service for clustering:

Common Mistakes

  • Using Deployments instead of StatefulSets for clustered DBs: Deployments create pods with random hashes and mount random volumes, causing database replicas to continuously lose sync and cross-mount the wrong data paths.
  • Forgetting to clean up orphan PVCs: When scaling down a StatefulSet from 3 replicas to 1, Kubernetes does not delete the unused PVCs (to prevent accidental data loss). You must manually delete them to avoid paying for unused disks.

Quick Quiz

Q1: Why is a headless service used in front of database StatefulSets?

A) To load balance SQL queries across write replicas.

B) To bypass kube-proxy and create direct DNS A-records for each sequential Pod, enabling internal discovery and replication setup.

C) To expose the database publicly on the internet.

Answer: B — Headless services return the IP addresses of individual Pods directly, which is required for clustering discovery.

Scenario-Based Challenge

Production Scenario:

Your company is migrating a high-throughput, low-latency transaction processing database to Kubernetes. The database handles 50,000 IOPS at peak. You are choosing between cloud-attached network storage (EBS gp3) and local NVMe storage on AWS i3 instance types. What should you recommend?

View Solution

To sustain 50,000 IOPS with minimal latency, you should recommend Local PVs using physical host-attached NVMe storage. Cloud block storage (EBS gp3) would require massive provisioned IOPS scaling (which is expensive and introduces network latency overhead). However, because Local PVs bind to a single host node, you must configure database-level replication (e.g. Postgres streaming replication or MongoDB Replica Sets) across at least 3 nodes with anti-affinity rules, and implement automated backup pipelines to push database snapshots to S3.

Debugging Exercise

Stuck Pod during upgrade:

You perform a rolling update on a database StatefulSet. Pod db-0 restarts but hangs in Pending. Checking logs reveals: Failed to attach volume: Volume is in use by node-1. Scheduler assigned pod to node-2.

View Solution

Root Cause: The database Pod's PVC is bound to a single AZ network block volume. Due to cluster resource allocation shifts, the scheduler assigned the new Pod instance to node-2, which is located in a different AZ, or the volume attachment was not cleanly detached from node-1 by the cloud controller manager before node-2 tried to attach it.
Fix: Annotate your StatefulSet or Pod templates to use node affinity restricting scheduling to the specific zone where the PV exists, or wait for the cloud controller attachment timeout to release the lock, or use the WaitForFirstConsumer StorageClass configuration to coordinate scheduling zones.

Interview Questions

1. Why are database Operators preferred over raw StatefulSets for running Postgres or MongoDB in production?

While StatefulSets manage the deployment of pods and the binding of persistent volumes, they do not understand application-level operations. Database Operators automate operations like configuring master-replica streaming replication, managing primary node election and failovers, scheduling consistent transaction backups, and applying schema migrations, acting as a virtual site reliability engineer.

2. What happens to the PVC when you scale down a StatefulSet?

The PVC is intentionally preserved. Kubernetes does not automatically delete PVCs or PVs when scaling down to prevent accidental data loss. You must manually delete the unused PVCs to avoid ongoing cloud storage charges.

Production Considerations

When deploying databases in production:
1. Set Resource Requests equal to Limits for database containers to ensure they receive a Guaranteed QoS class, preventing them from being evicted or OOM-killed first during host memory pressure.
2. Configure Node Anti-Affinity to guarantee that master and replica database pods are scheduled on separate physical host servers or availability zones.
3. Run database containers as non-root users and set appropriate kernel parameters (like virtual memory configuration) via Pod sysctl security contexts.