Docker Architecture & Internals
Docker Images and Layers — The Union File System
How overlay2 and copy-on-write compose filesystem layers.
Interview: Critical for understanding file storage footprints, overlay directory paths, and optimizing image build sizes during pipeline development.
Introduction
When you download a 500MB Docker image, it is not downloaded as a single massive file. Instead, Docker downloads multiple smaller segments called layers.
Docker images are composed of read-only layers stacked on top of each other. When you start a container, Docker adds a thin, read-write layer (the container layer) on top. This is made possible by the Union File System (UnionFS).
Why It Matters
Without image layering, every container would require its own dedicated disk space copy of the operating system, bloating storage requirements. Layering allows multiple containers running on a host to share base image layers safely without duplicating data.
How UnionFS Works (overlay2)
The modern standard storage driver for Docker is overlay2. It overlays multiple directories on a host machine and presents them as a single, unified filesystem. It divides the filesystem into four directory types:
• LowerDir (Read-Only): The underlying image layers containing OS libraries and files. These remain immutable.
• UpperDir (Read-Write): The container's local modifications layer. Any new files created or files modified are stored here.
• MergedDir (Unified View): The combined representation of LowerDir and UpperDir. This is what the container processes actually see inside the mount.
• WorkDir: An internal helper directory used for atomic transaction file writes.
Copy-on-Write (CoW) Mechanism
To keep layers immutable, Docker uses a Copy-on-Write strategy:
1. Reading Files: If a container reads a file, Docker searches the layers from top to bottom, returning the first instance found.
2. Modifying Files: If a container attempts to modify a file located in a read-only LowerDir, Docker copies the file up to the read-write UpperDir *first*, then performs the edits. The original image layer file remains untouched.
3. Deleting Files: If a container deletes a file, a special "whiteout" file is created in the UpperDir, masking the file in the unified view without deleting it from the read-only image layers.
Practical Example
Let's see how UnionFS works under the hood by creating a manual overlay mount on Linux:
What just happened? We created a manual overlay mount resembling Docker's storage strategy. Modifying the file inside merged did not touch the lower layer; the changes were written entirely to the upper layer using Copy-on-Write.
Quick Quiz
Q1: When multiple containers are created from the same Docker image, how does host memory usage scale?
A) Each container duplicates the entire image filesystem on host storage.
B) The containers share the read-only layers in memory, and each container only consumes disk space for its specific changes in the read-write layer.
C) The host kernel creates virtual partitions on disk.
D) The image layers are destroyed once the first container runs.
Answer: B — Shared read-only layers drastically reduce disk footprint and system memory usage.
Scenario-Based Challenge
Production Scenario:
You run a heavy database engine inside a container. After a week of execution, the database host disk is saturated. You notice the container storage layer is consuming 100GB. Performance has also degraded. What caused this, and how do you resolve it?
View Solution
Cause: The database engine was configured to write its data files directly to the container's filesystem. This forces all database writes through the overlay2 storage driver, triggering Copy-on-Write actions and storing massive transaction data in the UpperDir writable layer. This introduces high filesystem overhead and storage bloat.
Solution: Configure the container to write database files to a **Docker Volume** or a **Bind Mount** (e.g., -v /host/data:/var/lib/mysql). Volumes bypass the OverlayFS driver completely, writing database transactions directly to the host filesystem at native speed, and decoupling the database data from the container lifecycle.
Interview Questions
1. How does Docker calculate image layer diffs?
Each instruction in a Dockerfile (like RUN or COPY) creates a new layer. The daemon calculates a hash (Content Addressable Identifier) based on the specific filesystem differences generated by that instruction compared to the layer below it.
2. What happens to the modifications inside a container when the container is deleted?
The container's UpperDir read-write layer is deleted from the host disk. Any changes or files created inside the container are permanently lost unless they were written to a mounted volume or persistent bind directory.
Production Considerations
Keep layers clean. If you download, extract, and compile source code in one RUN step, delete the source files and package manager caches in the *same* step. If you delete them in a later RUN step, the files are still preserved in the immutable historical layers beneath, bloating the final image size.