Storage

1. Learning Objectives

By the end of this lesson, you will be able to:

• Differentiate between Block, File, and Object storage paradigms in terms of APIs, underlying architectures, and access latencies.
• Analyze RAID levels (0, 1, 5, 6, and 10), calculating storage capacity efficiency and fault-tolerance thresholds.
• Compare and contrast Network-Attached Storage (NAS) and Storage Area Networks (SAN) down to the protocol level (iSCSI, Fibre Channel, SMB, NFS).
• Evaluate write amplification, IOPS vs. throughput, and silent data corruption within storage networks.
• Design highly durable, multi-tier data architectures combining block volumes, file sharing, and object stores.

2. Prerequisites

To get the most out of this lesson, you should have a baseline understanding of:

• Computer Architecture: CPU, RAM, disk caches, and solid-state drive (SSD) vs. magnetic hard disk drive (HDD) differences.
• Operating System Basics: Kernel page caches, file system mounting, virtual file systems (VFS), and metadata (inodes).
• Basic Networking: TCP/IP networking, routing, latency overhead, and client-server communication models.

3. Why This Topic Matters

Data is the gravity of system design. Computations are ephemeral, but persistence is permanent. Choosing the wrong storage abstraction or hardware architecture guarantees scaling failure, data loss, or prohibitive operational costs.

For instance, storing database table spaces on an Object Store directly would result in terrible latencies, as it does not allow local modify-in-place operations. Conversely, attempting to store petabytes of user avatars in a POSIX filesystem on Block storage will crash directories due to inode limitations. Understanding storage design is the differentiator between a system that scales to hundreds of petabytes seamlessly and one that bottlenecks on disk controllers.

4. Real-world Analogy

Imagine managing the inventory and items in a massive logistics warehouse:

• Block Storage is a Raw Cargo Vessel: Goods are loaded in standardized, raw, numbered shipping containers. The ship's cargo master does not care if a container has gold bars, wheat, or computers inside; they only care about container #1405 located at Deck A, Row 3. It is fast, highly structured, and direct.
• File Storage is a Filing Cabinet: Documents are stored in folders, which are kept in drawers, which are nested inside cabinets. To locate a tax form, you must traverse a physical path: /Finance/Tax_Forms/2026/Form_W2.pdf. If the cabinet grows to a million files, searching the directory index takes a long time.
• Object Storage is Valet Parking / Coat Check: You hand over your car or coat. You do not get to choose where it is parked or hung. Instead, you receive a claim ticket (a unique key). You can attach metadata directly to the ticket (e.g., "red jacket", "SUV"). When you return the ticket, the system retrieves your item. It can scale to millions of items because there is no hierarchical search pathway.

5. Core Concepts

To master modern system design storage, you must grasp these core primitives:

Block Storage

Exposes raw, unformatted storage sectors directly to the host operating system. The OS formats this space with a filesystem (such as ext4 or NTFS). The system communicates using low-level protocols over SCSI/SAS command sets, identifying blocks via Logical Block Addressing (LBA).

File Storage

Organizes data hierarchically in nested folders and files. It implements locking mechanisms (POSIX standard) to allow multi-user environments to read/write concurrently without race conditions. Access is achieved via network file-sharing protocols (NFS, SMB).

Object Storage

Stores data as discrete objects in a flat structure. Each object consists of the payload (binary data), a globally unique identifier (URI/Key), and customizable metadata. It is accessed directly via HTTP REST APIs (GET, PUT, DELETE).

RAID (Redundant Array of Independent Disks)

A physical virtualization technology that pools multiple physical hard drives into a single logical unit. It achieves redundancy, performance scaling, or both through striping, mirroring, and parity calculations.

NAS vs. SAN

NAS (Network-Attached Storage) is a dedicated file-level computer sharing files over standard IP networks. SAN (Storage Area Network) is a specialized, high-speed network providing block-level access directly to disk controllers, bypassing standard TCP/IP protocol overhead in favor of Fibre Channel or iSCSI.

6. Visualization

Below is the comparison of structural layouts and access mechanics for the three main storage categories:

7. How It Works

Let's walk through the physical and logical lifecycle of operations in the three models:

1. Writing a Block in Block Storage

1. The application makes a system call to write data to a database file.
2. The DB engine or OS filesystem translates the file offset to a specific logical block address (LBA).
3. The OS storage driver wraps the request into SCSI, SATA, or NVMe commands.
4. The commands travel over the host bus adapter (HBA) or SAN (iSCSI) directly to the storage controller.
5. The storage controller writes the bytes into physical sectors on the magnetic platters or flash cells. No directory paths are traversed, resulting in sub-millisecond execution.

2. Traversing and Writing in File Storage

1. The application requests to write bytes to /shared/user/data.csv.
2. The OS filesystem mounts the network share (NFS/SMB) and checks permissions.
3. The filesystem driver traverses the directory tree structure: it looks up the directory index of /shared, finds the inode for user, and then requests the block pointers for data.csv.
4. The client locks the file (if necessary) to prevent other clients from writing.
5. The file storage server allocates blocks for the new data, updates the inode's size and access times, releases locks, and confirms success.

3. Uploading an Object in Object Storage

1. The client issues an HTTP PUT /bucket-name/user-123/avatar.png request containing the image payload and headers containing metadata (e.g. Content-Type, Author).
2. The API gateway receives the HTTP packet, validates authentication, and passes it to the directory service.
3. The directory service processes the flat key user-123/avatar.png. It hashes the key to determine which storage nodes will store the object replicas or erasure-coded shards.
4. The system writes the payload to multiple storage nodes concurrently.
5. The metadata (key, object size, checksum, custom user metadata) is written to a fast KV store or database index.
6. Once quorum (e.g., write to 2 out of 3 replica nodes) is met, the system returns an HTTP 200 OK response.

8. Internal Architecture

Each storage archetype contains a specific set of hardware and software components tailored to its access API:

9. Request Lifecycle

Lifecycle A: Raw Block Write (Database Engine to SSD Volume)

1. Application Layer: The Database Engine issues an fsync() command to commit a transaction log block.
2. OS Kernel VFS Layer: The kernel checks the cache. If bypass-cache (O_DIRECT) is enabled, the buffer bypasses page cache and is sent straight to the block I/O scheduler.
3. I/O Scheduler: The kernel organizes the block write request, sorting it by disk addresses (LBA) to minimize read/write head movement (for HDDs) or optimize block wear (for SSDs).
4. HBA & Controller: The Host Bus Adapter transmits the command via Fibre Channel/SATA wires. The Storage Controller receives the write request and registers it in its NVRAM (Non-Volatile RAM) write buffer.
5. Media Level: The controller flashes the data to the flash cells (or writes to physical sectors). An acknowledgment is sent back.
6. Completion: The OS driver returns success to the DB engine. Total latency: 100 microseconds to 2 milliseconds.

Lifecycle B: S3-Style Object Upload

1. Client Layer: An application calls s3.putObject({ bucket: 'photos', key: 'pic.jpg', body: imageBuffer }).
2. Ingress: The request goes over HTTP/2.0, traversing the internet, hitting a DNS load balancer, and arriving at the Storage API Gateway.
3. Authentication & Authorization: The gateway decrypts TLS, verifies the cryptographic signature (AWS Signature V4), and queries the identity service (IAM) for upload permissions.
4. Placement Strategy: The gateway contacts the Directory Node. The directory hashing function (like Consistent Hashing or CRUSH) maps the key photos/pic.jpg to virtual storage buckets. It returns a list of target storage node IPs (e.g., Node 15, Node 42, Node 91).
5. Concurrent Writing & Replication: The API gateway acts as a coordinator, writing the payload stream concurrently to the three target Storage Nodes.
6. Validation: Each node hashes the stream as it writes, comparing it to the client-provided MD5 checksum to guarantee zero byte corruption during transit.
7. Index Update: Once nodes acknowledge the write, the coordinator inserts the metadata record: {"{ key: 'photos/pic.jpg', size: 1048576, md5: '...', node_locations: [15, 42, 91] }"} into the globally distributed index database.
8. Response: The gateway returns an HTTP 200 OK response. Total latency: 15 to 100 milliseconds.

10. Deep Dive

Let's dissect the core architectural details of RAID configurations, Networked storage protocols, and advanced erasure coding mechanisms.

1. RAID Configurations: Deep Hardware Architecture

• RAID 0 (Striping): Splits data evenly across two or more disks with no parity or mirroring. It provides maximum read/write performance because data can be accessed in parallel across disk controllers. Capacity Efficiency: 100%. Fault Tolerance: 0 disks. If a single disk fails, the entire array is destroyed.
• RAID 1 (Mirroring): Clones the exact same data onto two or more disks. It offers excellent read performance (reads can be split between disks) and high redundancy. Write performance is limited by the speed of the slowest disk. Capacity Efficiency: 1/N where N is the number of disks. Fault Tolerance: N-1 disks.
• RAID 5 (Striping with Distributed Parity): Divides data and XOR-based parity information across three or more disks. If one disk fails, the parity blocks on the remaining drives are used to calculate the missing data on-the-fly.

  The Write Hole & Rebuild Vulnerability: When writing a small block, the system must read the old data and old parity, calculate new parity, and write new data and new parity (the "Read-Modify-Write" penalty). If a power failure occurs during this sequence, the data and parity become inconsistent. Furthermore, if a disk fails, rebuilding a RAID 5 array with large HDDs (e.g., 10TB+) can take days. During this rebuild phase, the remaining disks are worked at 100% load. If another drive suffers an Unrecoverable Read Error (URE) or total failure, the entire array is lost.

  Capacity Efficiency: (N-1)/N. Fault Tolerance: 1 disk.
• RAID 6 (Striping with Double Distributed Parity): Implements dual parity schemes (often using Reed-Solomon coding) across four or more disks. This allows the array to survive up to two simultaneous disk failures. Dual parity incurs a heavier write performance penalty than RAID 5 but offers necessary safety margins for modern high-capacity drive groups. Capacity Efficiency: (N-2)/N. Fault Tolerance: 2 disks.
• RAID 10 (1+0 - Mirrored Stripes): Combines the performance of striping (RAID 0) with the redundancy of mirroring (RAID 1). It mirrors pairs of disks first, and then stripes data across the mirrored pairs. Rebuild times are fast because the system only has to copy data from the surviving mirror disk, rather than recalculating parity across the entire array. Capacity Efficiency: 50%. Fault Tolerance: 1 disk per mirror set (up to N/2 total disks if they belong to different mirror pairs).

2. SAN vs. NAS Protocol Specs

The battle between SAN and NAS is not just physical; it is a battle of protocols and networking layers:

• SAN Protocols (Block Level):
  • Fibre Channel (FC): A high-speed network technology running custom fiber optic cables. It bypasses the standard TCP/IP networking stack entirely, mapping SCSI commands directly to Fibre Channel frames for ultra-low latency and zero network packet drops.
  • iSCSI: Encapsulates SCSI commands inside standard TCP/IP packets. This allows block storage networks to run on standard copper Ethernet switches, dramatically lowering infrastructure costs at the expense of CPU overhead for packaging/unpackaging Ethernet frames.
• NAS Protocols (File Level):
  • NFS (Network File System): Used predominantly in Linux environments. Mounts remote directories. In NFSv3, locking was handled via a separate lock manager daemon, leading to stale lock states. NFSv4 integrates locking and state handling directly into the protocol over a single TCP port (2049).
  • SMB (Server Message Block) / CIFS: Predominantly used in Windows environments. Relies on stateful connections. SMB v3.x introduces encryption, multi-channel support (binding multiple network connections to increase throughput), and direct RDMA (Remote Direct Memory Access) bypassing host CPU processing.

3. Erasure Coding (EC) vs. Replication

In massive scale object storage systems, replicating every byte 3 times (3x Replication) represents a 200% storage overhead. Erasure Coding acts as a math-based alternative:

• The object is divided into k data chunks.
• The system computes m parity chunks using Reed-Solomon algorithms.
• The total k + m chunks are distributed across k + m separate servers or failure zones.
• The original object can be reconstructed from any k chunks.
• For instance, in an EC 8+4 configuration (k=8, m=4):
  • Storage overhead is only 50% (4 parity / 8 data).
  • The system can lose up to 4 arbitrary drives or nodes simultaneously without data loss.
  • The trade-off is compute overhead: writing requires calculating parity matrices, and reading a failed disk's data requires reading from 8 other nodes and solving linear equations.

11. Production Example

AWS Simple Storage Service (S3)

Amazon S3 is designed to support 99.999999999% (11 9s) of durability. Its internal architecture solves massive scale by decoupling key indexing from physical data storage:

• The Metadata Directory: S3 uses a highly distributed, strongly consistent key-value store to map buckets and keys to physical block locators. It uses Log-Structured Merge (LSM) tree databases partitioned across nodes using consistent hashing on the key string (e.g. bucket/user-id/file.txt).
• Storage Nodes (OSDs): The payload bytes are stored on storage nodes running specialized local filesystems. Instead of saving raw filesystem directories, these nodes store flat byte streams identified by system-generated UUIDs.
• Durability Engineering: When a file is uploaded, S3 stores it across multiple Availability Zones (AZs). For standard S3 tiers, it writes to at least three AZs before returning success. Under the hood, S3 uses Erasure Coding across dozens of disks to protect against localized hardware failures, scrubbing drives continuously to detect bit rot and repair corrupted chunks proactively.

Google Colossus (The Successor to GFS)

Google's Big Table, Spanner, and MapReduce run on top of Colossus, their next-generation distributed cluster filesystem. It uses a centralized control plane (Metadata managers) paired with thousands of Chunkservers:

• Colossus Metadata Managers: These nodes are active-passive clusters configured with Paxos/Raft. They map file paths to chunk identifiers. Colossus avoids the memory bottleneck of the original Google File System (GFS) by storing metadata in a distributed database rather than keeping it all in RAM.
• Chunk Size: Instead of GFS's fixed 64MB chunks, Colossus manages smaller chunk variations and uses aggressive Reed-Solomon erasure coding (like 8+3 or 8+4) dynamically to reduce storage overhead while preserving high fault tolerance.

12. Advantages

Choosing the appropriate storage paradigm brings distinct system design benefits:

Block Storage Advantages

• Minimal Latency: Direct communication over high-speed buses or SAN paths yields sub-millisecond access.
• Arbitrary Random Access: Applications can modify specific bytes in-place without rewriting the entire file (essential for database transactional log writes).
• Dedicated Allocation: Avoids resource noisy-neighbor issues on disk since space is exclusively mapped.

File Storage Advantages

• POSIX Standard Support: Applications can use standard system calls (open, read, write, seek) without refactoring to use SDKs.
• Native Multi-Writer Sharing: Built-in locking protocols allow multiple application servers to safely edit shared folders (e.g. content management systems sharing image folders).

Object Storage Advantages

• Infinite Scalability: The flat namespace allows the system to scale horizontally by simply adding more storage nodes.
• Cost Optimization: Leverages cheap commodity hardware and advanced erasure coding, reducing storage cost per gigabyte.
• Rich Metadata: Developers can tag objects with operational attributes (e.g., userId, retentionPeriod: 7y), enabling automatic tiering and data classification lifecycle policies.

13. Limitations

Block Storage Limitations

• Poor Geographical Scalability: Block access requires low-latency fiber networks. You cannot mount a raw block device over WAN/Internet without extreme latency penalties.
• Complex Shared Access: Standard block systems cannot be mounted to multiple operating systems concurrently. Doing so without a cluster-aware filesystem (like GFS2 or VMFS) corrupts data.

File Storage Limitations

• Directory Traversal Bottleneck: As the number of folders and files scales into the millions, directory traversal and metadata lock management operations become CPU-bound.
• Metadata Server Scaling: The centralized directory manager or metadata node represents a single point of congestion.

Object Storage Limitations

• Immutability (No Modify-In-Place): Objects cannot be modified. If you change 1 byte of a 5GB file, you must re-upload the entire 5GB object.
• High Initial Latency: The HTTP handshake and API routing layers introduce initial latency overhead (typically 10-30ms), making it unusable for transactional database files.

14. Trade-offs

When architecting storage backends, engineers face several fundamental trade-offs:

1. Reliability vs. Write Amplification (RAID 10 vs. RAID 6)

Choosing RAID 10 yields ultra-fast recovery and excellent write performance because there are no parity calculations. However, it wastes 50% of the raw storage capacity. RAID 6 utilizes storage space much more efficiently but imposes a major write amplification penalty (reading, calculating, and writing double parity for every small random update).

2. Performance (Latency/IOPS) vs. Cost

Storing data on NVMe Block Storage volumes delivers the highest IOPS and lowest latency, but at a premium price. Decoupling cold or semi-active data and offloading it to Object Storage reduces costs by 90% or more, but access latencies shift from microsecond scales to double-digit milliseconds.

3. API Simplicity vs. Granularity

Object storage APIs are incredibly simple (GET, PUT keys over HTTPS), making integration straightforward. However, you trade off the power of POSIX APIs, which allow file-locking, byte-range locks, and sequential seek offsets. If your application relies on raw random seek modifications, wrapping it in Object APIs requires complex custom layers.

15. Performance Considerations

Optimizing storage layers requires understanding how different operations consume drive bandwidth and controller capacity:

IOPS vs. Throughput

• IOPS (Input/Output Operations Per Second): Measures the count of small read/write operations (typically 4KB or 8KB blocks) executed per second. Crucial for transactional workloads like Relational Databases (OLTP), which make many fast, random reads/writes.
• Throughput (MB/s): Measures the total data volume read or written per second. Crucial for sequential workloads like video streaming, log collection, or big data analytics.
• *Performance Tip:* A storage volume might hit its IOPS limit while throughput remains tiny (e.g. executing 10,000 tiny 4KB writes uses 10,000 IOPS but consumes only 40MB/s of bandwidth). Matching storage choices to workload profile is essential.

SSD Write Endurance & Write Amplification

Solid-State Drives cannot overwrite data-containing flash cells directly. They must first erase a complete block (typically 2MB to 8MB) before writing to pages (typically 4KB to 16KB) inside it. If an application performs small random writes, the SSD controller is forced to read an entire block, modify the page in memory, erase the physical block, and write the consolidated block. This process is called Write Amplification. It slows write speeds and degrades SSD life expectancy (measured in DWPD - Drive Writes Per Day).

16. Failure Scenarios

In enterprise system design, failure is a guarantee. Here are the main storage failure scenarios and mitigations:

1. Silent Data Corruption (Bit Rot)

Magnetic charge decay on HDDs or electric charge leakage on flash drives can flip individual data bits on physical disks. Since the disk controller might not register a hardware fault, the filesystem reads the corrupted block, injecting invalid state into the application.

Mitigation: Implement cryptographic checksums (e.g. SHA-256 or CRC32) for every block. Run background "scrubbing" daemons to match disk sectors against checksum indices periodically, replacing corrupted segments automatically from healthy mirrors/parity blocks.

2. The Parity Rebuild Storm

When a disk fails in a RAID 5 array, the controller is put under high load. It must read all remaining disks to reconstruct the lost data on the fly. This massive I/O load causes high latency for real-time traffic and often stresses aged disks in the array, leading to a secondary drive failure during the rebuild phase, resulting in absolute data loss.

Mitigation: Use RAID 6 or RAID 10 for larger drives. Implement hot-spare disks that are pre-wired to trigger rebuilds instantly, or migrate to erasure-coded distributed stores that reconstruct data in parallel across dozens of servers rather than bottlenecking on a single controller.

3. Split-Brain in Distributed Storage

If a network partition isolates metadata manager nodes in a distributed storage cluster (e.g., Ceph Monitors or HDFS NameNodes), two different nodes might believe they are the primary master, accepting conflicting writes from client groups.

Mitigation: Require strict quorum consensus protocols (Raft, Paxos) for metadata coordination. If a sub-cluster loses majority quorum, it must immediately transition to read-only or terminate client requests.

17. Best Practices

• Decouple State from Compute: Keep application nodes stateless. Offload media assets to Object Storage (such as Amazon S3) and serve them using a Content Delivery Network (CDN) to reduce load on application servers.
• Match Volume Configuration to Workload:
  • Use high-performance Block Storage (SSD) with provisioned IOPS for transactional databases.
  • Use file shares (NFS/SMB) for shared runtime configurations and document sharing across microservices.
  • Use object stores for cold logs, video backups, and user uploads.
• Enable Storage Tiering Lifecycles: Configure automatic policies to shift objects from standard hot storage tiers to cold archive tiers (e.g., S3 Standard -> Infrequent Access -> Glacier Deep Archive) based on access age to optimize costs.
• Implement End-to-End Encryption: Enforce encryption-at-rest (using AES-256 keys managed by HSMs) and encryption-in-transit (TLS 1.3 for object storage API calls; SMB Encryption/IPsec for file storage).

18. Common Mistakes

• Using Object Storage as a Database: Attempting to update database indexes by downloading, modifying, and re-uploading JSON or Parquet files to object stores. This results in terrible latencies and consistency race conditions.
• Running Relational Databases on Shared Network Filesystems: Running active PostgreSQL or MySQL databases on NFS or SMB shares without dedicated file lock arbitration. If the network drops packets, database journal files can become corrupted due to stale lock states.
• Deploying RAID 5 on Massive Drives: Configuring RAID 5 for arrays using 16TB+ SATA disks. The probability of another disk hitting a URE during a multi-day rebuild is extremely high.
• Ignoring Inode Limits on Block Filesystems: Storing millions of small files inside a single directory on a block-mounted ext4 volume. The system will run out of inodes long before the disk runs out of physical byte capacity, leading to "Disk Full" errors.

19. Implementation

Below is a complete, working TypeScript implementation of a mock Distributed Object Storage Engine. It demonstrates consistent hashing/routing, content checksum validation (to prevent bit rot), metadata indexing, and multi-node replication:

20. Interview Questions

Easy Question

Question: What is the primary difference between Block, File, and Object storage? Give a common real-world cloud service example of each.

Answer:

• Block storage provides raw, unformatted volumes addressed by block numbers (LBAs). Example: AWS EBS. Used for virtual machine drives and databases.
• File storage exposes a hierarchical tree of files and directories accessible via network protocols (NFS/SMB). Example: AWS EFS. Used for shared configuration folders and user directories.
• Object storage manages flat namespaces of immutable objects queried over HTTP APIs. Example: AWS S3. Used for media files, data lakes, and backups.

Medium Question

Question: Describe the "Write Hole" problem in RAID 5. How does RAID 6 or a battery-backed write cache mitigate this?

Answer: The write hole occurs when a RAID 5 system is updating a small data block. It must read the old data and old parity, calculate new parity, and write the new data and new parity (the read-modify-write cycle). If a sudden power loss occurs between writing the data and writing the parity, the data stripe becomes inconsistent. When a disk subsequently fails, the controller recalculates data using the corrupted parity, resulting in silent data corruption.

Mitigations:

• RAID 6: Uses two parity checks, making it more resilient, though not entirely immune to multi-write failures.
• Non-Volatile RAM (NVRAM) / Battery-Backed Caches: The controller cache preserves incomplete write states during power failures and flushes them to disk once power returns.
• Journaling / Log-Structured File Systems: Writing parity and data as a single transactional write prevents half-committed states.

Hard Question

Question: Design a metadata scaling model for a distributed object store that contains trillions of items. How would you handle hot partitions (e.g., millions of reads to a single key)?

Answer:

1. Metadata Distribution: Do not store metadata on a single node or in RAM. Use a distributed NoSQL engine (like Cassandra or Spanner) partitioned using consistent hashing on the key (e.g., Hash(Bucket + Key)).
2. Consistent Hashing Range Splitting: Divide the hash ring into virtual nodes. When a metadata partition grows too large or hot, dynamically split the range and re-balance shards across new nodes.
3. Handling Hot Partitions (Read Bottlenecks):
   • Read Replication / Caching: Use a write-through caching layer (like Memcached or Redis) to intercept metadata requests. For highly read-heavy keys, dynamically replicate the metadata record to a temporary "Hot Cache Ring" across multiple nodes.
   • API Rate Limiting & Coalescing: Implement request coalescing (Singleflight pattern) at the API Gateway level. If 1,000 concurrent requests request the same key metadata, send only 1 request to the database and broadcast the result to all 1,000 waiting clients.

21. Practice Exercises

Easy Exercise

Calculate the usable storage capacity of a 6-disk array with 10TB disks using RAID 0, RAID 1, RAID 5, RAID 6, and RAID 10. Write down the formulas and steps you used to arrive at each answer.

Medium Exercise

Draft a design specification for a multi-part upload client in Python or Go. The client must split a 1GB file into equal 10MB chunks, upload them concurrently across a configurable number of worker threads (e.g., 8 threads), verify each chunk using MD5 checksum headers, and make a final call to finalize assembly on the server.

Hard Exercise

Implement a basic simulation of a CRUSH map algorithm (as used in Ceph) in Python. The simulator should route object keys to specific storage nodes while respecting a hierarchy of failure zones (e.g., Rack -> Host -> Disk OSD). Show that if a Rack goes offline, the data is still reachable from replicas located in other racks.

22. Challenge Problem

Scenario: You are the lead system architect at a global video-sharing startup. The platform gets 200,000 video uploads per day (averaging 500MB each). The system must transcode uploaded videos into 4 distinct resolution profiles (1080p, 720p, 480p, 360p) immediately upon upload. The videos must be delivered to users globally with low latency. Older videos (not viewed in the last 30 days) are rarely accessed but must be retained indefinitely due to compliance. Storage cost budget is restricted.

Design the complete storage system layout. Your design should address:

• The ingestion layer storage (temporary scratch space vs. permanent raw backups).
• The transcoding storage flow (read/write IOPS required during video manipulation).
• How the files are stored for global egress (CDN caching + Object store layout).
• Lifecycle tiering policy rules to keep cold storage costs at absolute minimums.
• How you protect against complete regional disaster (Multi-region replication strategy).

23. Summary

Storage is not a one-size-fits-all component. System performance, durability, and cost hinge on selecting the correct abstraction:

• Block Storage provides low-latency, raw addressable sectors ideal for transactional databases and virtual machine root drives.
• File Storage provides an intuitive, hierarchical filesystem ideal for shared folders, CMS backends, and assets shared via NFS/SMB.
• Object Storage provides flat namespaces queried via HTTP REST APIs, offering near-infinite horizontal scale, cost efficiency, and metadata-driven lifecycles.
• RAID adds a hardware-virtualized protection layer on physical hosts, while SAN and NAS define how block and file storage are shared across corporate networks.

24. Cheat Sheet

Storage Paradigm Comparison

RAID Configuration Matrix

25. Quiz

1. Which storage type is standard for storing high-frequency database transaction logs?
   • A) Object Storage
   • B) Block Storage
   • C) File Storage
   • D) Cold Archive Storage

   Answer: B
   Explanation: Database log engines require low latency random writes (modify-in-place) which is exclusively supported by Block storage volumes. Object storage is write-once (immutable) and has too high of latency.

2. What is the primary advantage of RAID 10 over RAID 5?
   • A) RAID 10 has a lower drive count requirement.
   • B) RAID 10 does not require calculating parities, leading to better write speeds and faster disk rebuilds.
   • C) RAID 10 has higher storage space efficiency.
   • D) RAID 10 has lower hardware requirements.

   Answer: B
   Explanation: RAID 10 mirrors and stripes, bypassing mathematical parity calculations. This prevents the "read-modify-write" performance penalty and allows simple drive copying during rebuilds instead of recalculating parity blocks.

3. In Ceph or Amazon S3, what is the role of the Metadata Directory?
   • A) To store the physical binary byte payloads of the objects.
   • B) To translate HTTP requests to SCSI drive commands.
   • C) To map human-readable object keys to physical UUIDs and disk locations.
   • D) To perform RAID parity computations.

   Answer: C
   Explanation: Object storage engines decouple content from key indices. The metadata directory matches the API key (e.g. /my-bucket/pic.png) to node IP addresses and content UUIDs.

4. What is the Storage Space Efficiency of a RAID 6 array containing 8 disks of 10TB each?
   • A) 100% (80TB)
   • B) 87.5% (70TB)
   • C) 75% (60TB)
   • D) 50% (40TB)

   Answer: C
   Explanation: RAID 6 usable storage space formula is (N - 2) * Capacity. For N=8, usable space is (8 - 2) * 10TB = 60TB, which is 75% of the total 80TB.

5. Which protocol is stateful, supports multi-channel connection binding, and is primarily native to Windows filesystems?
   • A) NFS
   • B) iSCSI
   • C) Fibre Channel
   • D) SMB

   Answer: D
   Explanation: Server Message Block (SMB) is Windows' native file-sharing protocol. Version 3 introduced features like SMB Multichannel, which aggregates bandwidth across network interfaces.

6. What happens when an ext4 block filesystem runs out of inodes but still has 500GB of physical space?
   • A) The filesystem shifts to object storage automatically.
   • B) Files are compressed to make room.
   • C) The system returns "No space left on device" errors when trying to create new files.
   • D) Directories are merged to consolidate inodes.

   Answer: C
   Explanation: File allocation in block filesystems requires an inode. If the maximum number of inodes is consumed (usually due to millions of tiny files), the OS cannot log new directory listings and reports the disk as full.

7. What is "Silent Data Corruption" (Bit Rot)?
   • A) A software bug that deletes files without logging.
   • B) Physical wear of SSD cells that causes write lock failures.
   • C) Minor degradation of physical magnetic media that flips bits on disk without raising drive errors.
   • D) A network partitioning state in a distributed filesystem.

   Answer: C
   Explanation: Bit Rot is a decay of storage media over time. It flips bits silently. The system only notices it if the file is read and compared against a previously computed cryptographic checksum.

8. Why is RAID 5 rebuild particularly risky when using very large SATA hard drives (e.g. 14TB)?
   • A) SATA drives do not support RAID controllers.
   • B) Rebuilding takes a very long time, and the read stress on the remaining drives increases the probability of hitting an Unrecoverable Read Error (URE).
   • C) The controller cannot format drives larger than 2TB.
   • D) Parity calculations require too much CPU cache space.

   Answer: B
   Explanation: Rebuilding 14TB of data from parity blocks on a failed RAID 5 array takes days. During this process, every sector of all surviving disks is read. The sheer number of read operations increases the chance of hitting a URE, which aborts the rebuild and results in data loss.

9. How does Erasure Coding (e.g. 8+4) compare to 3x Replication?
   • A) Erasure Coding has higher storage overhead but lower latency.
   • B) Erasure Coding has lower storage overhead but requires more CPU computations.
   • C) 3x Replication is less reliable than 8+4 Erasure Coding.
   • D) 3x Replication has lower storage overhead.

   Answer: B
   Explanation: 8+4 EC has a 50% storage overhead, whereas 3x Replication has a 200% storage overhead. However, EC requires computing Reed-Solomon parity matrices, which increases CPU utilization.

10. Which technology allows direct network memory transfer without host CPU intervention?
    • A) iSCSI
    • B) RDMA (Remote Direct Memory Access)
    • C) SCSI Command Tag Queuing
    • D) NFS Lockd

    Answer: B
    Explanation: RDMA enables network adapters to transfer data directly to or from application memory buffers on another machine without copying it through the OS kernel and CPU pipeline.

26. Further Reading

• The Google File System (2003): Read the GFS Paper — The foundational paper on distributed file systems.
• Ceph Architecture Guide: Learn about CRUSH maps and dynamic data placement on unified storage engines.
• Designing Data-Intensive Applications: Chapter 3 (Storage and Retrieval) by Martin Kleppmann.

27. Next Lesson Preview

In the next lesson, we will move from raw byte persistence blocks to structured data paradigms. We will explore Relational databases (SQL) vs. Non-Relational databases (NoSQL), examining LSM-trees, B-Trees, transaction logs, and the trade-offs of query patterns.