Container Runtime

3 posts

datadogOriginal article

Using the Dirty Pipe vulnerability to break out from containers | Datadog (opens in new tab)

The Dirty Pipe vulnerability (CVE-2022-0847) is a critical Linux kernel flaw that allows unprivileged processes to write data to any file they can read, effectively bypassing standard write permissions. This primitive is particularly dangerous in containerized environments like Kubernetes, where it can be leveraged to overwrite the host’s container runtime binary. By exploiting how the kernel manages page caches, an attacker can achieve a full container breakout and gain administrative privileges on the underlying host. ## Container Runtimes and the OCI Specification * Kubernetes utilizes the Container Runtime Interface (CRI) to manage containers via high-level runtimes like containerd or CRI-O. * These high-level runtimes rely on low-level Open Container Interface (OCI) runtimes, most commonly runC, to handle the heavy lifting of namespaces and control groups. * Isolation is achieved by runC setting up a restricted environment before executing the user-supplied entrypoint via the `execve` system call. ## Evolution of runC Vulnerabilities * A historical vulnerability, CVE-2019-5736, previously allowed escapes by overwriting the host’s runC binary through the `/proc/self/exe` file descriptor. * To mitigate this, runC was updated to either clone the binary before execution or mount the host's runC binary as read-only inside the container. * While the read-only mount improved performance through kernel cache page sharing, it created a target for the Dirty Pipe vulnerability, which specifically targets the kernel page cache. ## The Dirty Pipe Exploitation Primitive * Dirty Pipe allows an attacker to overwrite any file they can read, including read-only files, by manipulating the kernel's internal pipe-buffer structures. * The exploit targets the page cache, meaning the overwrite is non-persistent and resides only in memory; the original file on disk remains unchanged. * In a container escape scenario, the attacker waits for a runC process to start (triggered by actions like `kubectl exec`) and targets the file descriptor at `/proc/<runC-pid>/exe`. ## Proof-of-Concept Escape Walkthrough * The attack begins with a standard, unprivileged pod running a malicious script that monitors the system for new runC processes. * Once a `kubectl exec` command is issued by an administrator, the script identifies the runC PID and applies the Dirty Pipe exploit to the associated executable. * The exploit overwrites the runC binary in the kernel page cache with a malicious ELF binary. * Because the host kernel is executing this hijacked binary with root privileges to manage the container, the attacker’s malicious code (e.g., a reverse shell or administrative command) runs with full host-level authority. To protect against this attack vector, it is essential to patch the Linux kernel to a version that includes the fix for CVE-2022-0847 and ensure that container nodes are running updated distributions.

datadog3 min readCurated summary

Escaping containers using the Dirty Pipe vulnerability | Datadog Security Labs

The post demonstrates how the Linux Dirty Pipe vulnerability can enable an unprivileged process to escape a container and gain administrative privileges on the host. The exploit abuses runC’s execution model and its host binary, which is exposed read-only inside the container but can still be modified through the kernel page cache. A proof of concept shows how a compromised Kubernetes pod can overwrite runC with a malicious executable when an administrator runs `kubectl exec`. ## Container Runtimes and runC - Kubernetes commonly uses containerd or CRI-O through the Container Runtime Interface (CRI). - These runtimes rely on lower-level OCI runtimes, most notably runC, to create isolated Linux processes. - runC configures namespaces, cgroups, and the container environment before executing the supplied entrypoint with `execve`. - During execution, `/proc/self/exe` inside the container can refer to an open descriptor for the runC binary on the host. ## Earlier runC Escape Vulnerability - CVE-2019-5736 exploited this `/proc/self/exe` behavior: - A malicious container entrypoint could write to the host’s runC binary. - Overwriting runC enabled subsequent container operations to execute attacker-controlled code with host-level privileges. - runC initially mitigated the issue by cloning its binary before execution. - It later changed the design to mount the runC binary read-only inside the container, improving performance through kernel page-cache sharing. - That optimization created conditions in which Dirty Pipe could bypass the apparent read-only protection. ## Dirty Pipe as a Container Escape Primitive - Dirty Pipe allows an unprivileged process to overwrite files it can read, even without write permission. - The modification occurs in the kernel page cache rather than persistent storage: - The original file remains intact on disk. - Dropping caches or rebooting can restore the original contents. - Despite being temporary, the overwrite is sufficient to execute malicious code when the modified binary is run. - In this case, the attacker targets the host’s runC binary through `/proc/<runC-pid>/exe`. ## Kubernetes Proof of Concept - The demonstration starts an ordinary, unprivileged pod using an attacker-controlled container image. - Its entrypoint script: - Replaces `/bin/sh` with a launcher referencing `/proc/self/exe`. - Waits for a runC process to appear. - Invokes the Dirty Pipe exploit against that process’s executable. - An administrator running `kubectl exec` causes runC to execute inside the container, triggering the overwrite. - The modified runC is replaced with a malicious ELF binary that runs commands such as `id` and `hostname`, recording their output in `/tmp/hacked`. - The exploit is adapted from the original Dirty Pipe proof of concept and the earlier runC escape technique. The attack illustrates that kernel vulnerabilities can undermine container isolation even when the container is unprivileged and the target binary is mounted read-only. Systems should promptly patch vulnerable Linux kernels and container runtimes, while treating compromised containers as potential paths to host compromise.

Read original(opens in new tab)
datadog3 min readCurated summary

How we minimized the overhead of Kubernetes in our job system

Kubernetes can improve machine management and scalability, but its scheduling and runtime overhead can significantly reduce job throughput if configured poorly. Datadog found its Kubernetes-based job system used more CPU and completed jobs 40–50% more slowly than the previous VM-based system. By designing a controlled experiment, choosing better metrics, and tuning pod resource requests, the team recovered performance to roughly VM parity while investigating the overhead of running one parent process per pod. ## Designing a Comparable Experiment - The initial comparison was difficult because the Kubernetes and VM deployments differed in: - Number of nodes - Number of worker-parent clusters - Workload and enqueue rate - A controlled experiment was created using: - Identical `c5.2xlarge` machines - The same kernel version, `3.13.0-141` - Both systems repeatedly running a simple Python job - Each Kubernetes pod contained one parent process and its worker processes, making pod count equivalent to parent-process count per node. - The older kernel did not include CPU mitigations, avoiding that variable in the comparison. ## Choosing Useful Performance Metrics ### Measuring node effort - Load average initially appeared useful for measuring machine utilization. - Kubernetes background processes—such as cluster polling and pod-state checks—artificially increased load average. - Load average counts runnable processes rather than the amount of CPU time they actually consume. - The team therefore used CPU idle time instead: - It measures unused CPU capacity. - It reflects actual CPU work rather than the number of active processes. ### Measuring system performance - The job system optimized for throughput rather than latency. - Throughput was measured by the number of jobs completed within 30 seconds. - Latency remained useful for detecting queueing problems, but throughput was the primary success metric. ## Tuning Kubernetes Resource Requests - The main performance gains came from improving pod scheduling. - The target was six pods per `c5.2xlarge` node. - Initially, each pod requested: - One full CPU core - More memory than necessary - Since the node had eight cores and approximately 1.5 GiB of memory consumed by Kubernetes and system services, only four pods could be scheduled. - Requests were reduced to: - `100m` CPU, or 100 millicores - `500 MB` memory - CPU tuning generally enabled six pods per node, although some nodes still scheduled only five. - Further memory reduction was needed because system daemons consumed enough memory to prevent six pods from fitting on some nodes. - Resource requests affect scheduling minimums, while limits constrain containers after they start. - These request changes did not slow jobs because the pods still received sufficient resources to operate. ## One Parent Process per Pod - The team considered placing multiple parent processes in each pod to reduce potential pod overhead. - One parent plus its workers was a natural application unit and simplified orchestration. - The decision depended on how much overhead each pod introduced: - High overhead would favor fewer, larger pods. - Low overhead would favor one parent per pod for simpler management. - Using `pstree`, the team identified six job-system instances per node and traced their process trees through components such as: - `containerd-shim` - `tini` - The application process - They estimated that each pod included overhead associated with three containers, particularly `containerd-shim`. - CPU overhead was then investigated using `perf sched`. The practical lesson is to compare equivalent workloads, measure actual CPU consumption rather than relying blindly on load average, and tune Kubernetes requests for the desired packing density. Resource requests should be large enough for reliable operation but not so large that they unnecessarily prevent pods from being scheduled together.

Read original(opens in new tab)