techlist.io
KREN
netflix

Mount Mayhem at Netflix: Scaling Containers on Modern CPUs (opens in new tab)

database-designk8sawsdockerlinux-kernelcontainerdnumaoverlayfs

Netflix’s effort to modernize its container runtime exposed a hardware-level bottleneck rather than an application problem. Under heavy startup concurrency, containers with many image layers triggered massive mount and unmount activity, causing kernel lock contention, systemd stalls, and container startup failures. The issue was especially severe on older dual-socket NUMA instances, while newer single-socket systems scaled much more reliably.

Container Startup at Netflix

  • New AWS capacity is rapidly filled with pods as applications scale.
  • Some nodes became unresponsive, with:
    • Health checks timing out for more than 30 seconds
    • Kubelet requests to containerd timing out
    • systemd processing huge numbers of mount events
    • The mount table taking tens of seconds to read
  • The problem primarily affected r5.metal instances running images with more than 50 layers.

Mount Lock Contention

  • With user namespaces, containerd performs several mount operations for every image layer:
    • open_tree() references the layer.
    • mount_setattr() applies the container’s ID mapping.
    • move_mount() creates an ID-mapped bind mount.
  • These bind mounts become OverlayFS lower directories and are later unmounted.
  • The Linux VFS uses global mount-related locks, so concurrent container creation causes CPUs to contend on the same kernel locks.
  • For 100 containers with 50 layers each, containerd performs the process twice:
    • 100 × 2 × (1 + 50 + 50) = 20,200 mount operations
  • This makes startup cost depend heavily on both container concurrency and image layer count.

Why the New Runtime Exposed the Problem

  • The old Docker-based runtime shifted file ownership while unpacking images.
  • All containers shared one host user range, avoiding repeated per-container mount work.
  • The new containerd-based runtime assigns each container a unique host user range for stronger isolation.
  • Instead of rewriting file ownership during extraction, it uses Linux ID-mapped mounts to apply ownership mappings efficiently.
  • This improves security and avoids expensive image copying, but creates many additional mount operations during startup.

Differences Between AWS Instance Types

Netflix compared:

  • r5.metal: 5th-generation Intel, dual-socket, multiple NUMA domains
  • m7i.metal-24xl: 7th-generation Intel, single-socket, single NUMA domain
  • m7a.24xlarge: 7th-generation AMD, single-socket, single NUMA domain

Results showed:

  • At low concurrency—around 20 containers or fewer—all systems performed similarly.
  • r5.metal began failing at roughly 100 concurrent container launches.
  • Newer Intel instances maintained lower startup times and better success rates.
  • AMD-based m7a instances scaled most consistently and had the fewest failures.

Kernel and CPU-Level Diagnosis

  • Profiling showed that containerd spent most of its time in Linux VFS path lookup code.
  • Specifically, threads were spinning in path_init() while waiting on a sequence lock.
  • Intel Topdown Microarchitecture Analysis found:
    • 95.5% of pipeline slots stalled on contested accesses
    • 57% attributed to false sharing
  • Cache-line bouncing and global lock contention, rather than raw CPU capacity, dominated performance.

NUMA as a Contributing Factor

  • NUMA systems divide memory among processor sockets.
  • Local memory access is faster, while remote access crosses an interconnect and introduces additional latency.
  • The dual-socket layout of r5.metal amplified contention around shared mount-related data.
  • The better behavior of newer single-socket instances indicated that CPU topology and memory locality were key contributors to the container startup bottleneck.

Practical Conclusion

High-concurrency container launches can overwhelm kernel mount infrastructure, especially when using per-container ID mapping and images with many layers. Netflix’s results suggest minimizing image layers, controlling startup concurrency, and favoring newer single-socket hardware can substantially improve reliability and scaling.