cloudflare·Mar 26, 2026·2 min readCurated summary

A one-line Kubernetes fix that saved 600 hours a year

Atlantis restarts were taking about 30 minutes, blocking infrastructure changes and consuming more than 50 engineering hours monthly. The delay was caused by Kubernetes recursively changing ownership on a large Ceph-backed PersistentVolume containing millions of files. Setting `fsGroupChangePolicy: OnRootMismatch` avoided unnecessary recursive ownership changes and reduced restart time dramatically. ### The Restart Bottleneck - Atlantis runs as a singleton Kubernetes `StatefulSet`. - Its PersistentVolume stores repository and Terraform state. - Credential rotations, onboarding, and offboarding required restarting Atlantis. - With roughly 100 restarts per month, each 30-minute delay created more than 600 hours of annual lost engineering time. - The volume had grown large enough to exhaust inodes, making storage expansion and pod restarts necessary. ### Kubernetes Made the Delay Look Like a Scheduling Problem - `kubectl rollout restart statefulset atlantis` terminated the old pod and created a replacement. - The new pod was scheduled quickly but remained stuck in `Init:0/1`. - Kubernetes events showed the image pulling successfully, but revealed no obvious cause for the long gap. - Kubelet logs showed the PersistentVolume mounting successfully, followed by repeated `context deadline exceeded` errors while syncing the pod. ### The Hidden Cost of `fsGroup` - Searching logs using the PersistentVolume name exposed the relevant message: - Kubernetes was “setting volume ownership” because an `fsGroup` was configured. - Kubernetes warned that ownership changes could be slow when a volume contained many files. - The default behavior recursively changed ownership across the entire mounted volume. - As Atlantis’s volume accumulated millions of files, this initialization step became the 30-minute bottleneck. ### The One-Line Fix - The volume configuration was changed to: ```yaml fsGroupChangePolicy: OnRootMismatch ``` - With this policy, Kubernetes checks the root directory’s ownership and only performs recursive changes when necessary. - Existing volumes with the correct ownership no longer require a full filesystem traversal during every restart. The practical lesson is to inspect kubelet and volume logs when a pod appears scheduled but remains stuck before initialization. For large persistent volumes, explicitly setting `fsGroupChangePolicy: OnRootMismatch` can eliminate costly recursive ownership changes and prevent substantial operational downtime.

Read original(opens in new tab)

KubectlDatabase DesignKubernetesTerraform+4

datadog·Mar 25, 2022Original article

Using the Dirty Pipe vulnerability to break out from containers | Datadog (opens in new tab)

The Dirty Pipe vulnerability (CVE-2022-0847) is a critical Linux kernel flaw that allows unprivileged processes to write data to any file they can read, effectively bypassing standard write permissions. This primitive is particularly dangerous in containerized environments like Kubernetes, where it can be leveraged to overwrite the host’s container runtime binary. By exploiting how the kernel manages page caches, an attacker can achieve a full container breakout and gain administrative privileges on the underlying host. ## Container Runtimes and the OCI Specification * Kubernetes utilizes the Container Runtime Interface (CRI) to manage containers via high-level runtimes like containerd or CRI-O. * These high-level runtimes rely on low-level Open Container Interface (OCI) runtimes, most commonly runC, to handle the heavy lifting of namespaces and control groups. * Isolation is achieved by runC setting up a restricted environment before executing the user-supplied entrypoint via the `execve` system call. ## Evolution of runC Vulnerabilities * A historical vulnerability, CVE-2019-5736, previously allowed escapes by overwriting the host’s runC binary through the `/proc/self/exe` file descriptor. * To mitigate this, runC was updated to either clone the binary before execution or mount the host's runC binary as read-only inside the container. * While the read-only mount improved performance through kernel cache page sharing, it created a target for the Dirty Pipe vulnerability, which specifically targets the kernel page cache. ## The Dirty Pipe Exploitation Primitive * Dirty Pipe allows an attacker to overwrite any file they can read, including read-only files, by manipulating the kernel's internal pipe-buffer structures. * The exploit targets the page cache, meaning the overwrite is non-persistent and resides only in memory; the original file on disk remains unchanged. * In a container escape scenario, the attacker waits for a runC process to start (triggered by actions like `kubectl exec`) and targets the file descriptor at `/proc/<runC-pid>/exe`. ## Proof-of-Concept Escape Walkthrough * The attack begins with a standard, unprivileged pod running a malicious script that monitors the system for new runC processes. * Once a `kubectl exec` command is issued by an administrator, the script identifies the runC PID and applies the Dirty Pipe exploit to the associated executable. * The exploit overwrites the runC binary in the kernel page cache with a malicious ELF binary. * Because the host kernel is executing this hijacked binary with root privileges to manage the container, the attacker’s malicious code (e.g., a reverse shell or administrative command) runs with full host-level authority. To protect against this attack vector, it is essential to patch the Linux kernel to a version that includes the fix for CVE-2022-0847 and ensure that container nodes are running updated distributions.

KubectlKubernetesDocker & ContainersLinux+4