The provided content does not include the blog post itself. It contains Datadog’s navigation menu and a promotional banner announcing its recognition as a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms, but no article text about “Husky.”
## Content Available
- The page links to Datadog products covering:
- Infrastructure and application monitoring
- Logs, databases, and data observability
- Security and digital experience
- Software delivery and service management
- AI and platform capabilities
- The banner promotes Datadog’s observability-platform recognition.
- No technical details, sections, architecture diagrams, implementation discussion, or conclusions from the Husky article are included.
Please provide the article text or a complete page extract for an accurate summary.
Datadog engineers recently optimized the Datadog Agent's metric processing pipeline to achieve higher throughput and lower CPU overhead. By identifying that metric context generation—the process of creating unique keys for metrics—was a primary bottleneck, they implemented a series of algorithmic changes and Go runtime optimizations. These improvements allow the Agent to process significantly more metrics using the same computational resources.
### Identifying Bottlenecks via CPU Profiling
* Developers utilized Go’s native profiling tools to capture CPU usage during high-volume metric ingestion via DogStatsD.
* Flamegraph analysis revealed that the `addSample` and `trackContext` functions were the most CPU-intensive components of the pipeline.
* The profiling data specifically pointed to tag sorting and deduplication as the underlying operations consuming the most processing time.
### The Challenges of Metric Context Generation
* The Agent must generate a unique hash (context) for every metric received to address it within a hash table in RAM.
* To ensure the same metric always generates the same key, the original algorithm required sorting all tags and ensuring their uniqueness.
* The computational cost of sorting lists repeatedly for every incoming message created a performance ceiling for the entire metrics pipeline.
### Specialization and Runtime Optimization
* **Algorithmic Specialization:** The team implemented specialized sorting logic that adjusts based on the number of tags, optimizing the "hot path" for the most common metric structures.
* **Hashing Efficiency:** Micro-benchmarks identified Murmur3 as the most efficient hash implementation for balancing speed and collision resistance in this use case.
* **Leveraging Go Runtime:** The team transitioned from 128-bit hashes to 64-bit metric contexts. This change allowed the Agent to utilize Go's internal `mapassign_fast64` and `mapaccess2_fast64` functions, which provide optimized map operations for 64-bit keys.
### Redesigning for Performance
* The original design followed a rigid "hash metric name -> sort tags -> deduplicate tags -> iterative hash" workflow.
* Recognizing that sorting was the primary architectural bottleneck, the team moved toward a new design intended to minimize or eliminate the overhead of traditional list sorting during context generation.
To achieve similar performance gains in high-throughput Go applications, developers should profile their applications under realistic load and look for opportunities to leverage runtime-specific optimizations, such as using 64-bit map keys to trigger specialized compiler paths.
The Datadog Agent needed to process more metrics without increasing CPU usage. Profiling showed that generating unique metric contexts—especially sorting and deduplicating tags—was a major bottleneck. Datadog improved throughput through specialized sorting paths, faster hashing, and a more efficient context-storage design.
## Identifying the Bottleneck
- Datadog uses Go’s CPU and memory profiling tools to optimize the Agent’s metrics pipeline.
- Profiles were captured while Agents processed large volumes of DogStatsD metrics, ensuring the results reflected real workload pressure.
- Flamegraphs showed that `addSample` and `trackContext` consumed the most CPU.
- Sorting-related functions, including `util.SortUniqInPlace` and `sort`, were significant contributors to that cost.
## How Metric Contexts Work
- Each received metric is assigned a metric context that uniquely identifies it in an in-memory hash table.
- The context must incorporate:
- The metric name
- Tags included in the DogStatsD message
- Container-generated tags
- The context is computed as a hash, so it must be fast while minimizing collisions.
- Tags must be consistently ordered so the same metric always produces the same context.
- The original implementation sorted tags and removed duplicates, making sorting a recurring CPU expense.
## Specialized Sorting
- Performance varied according to the number of tags attached to a metric.
- Datadog introduced specialized sorting paths based on tag count.
- This allowed common cases to use more efficient algorithms while retaining correct ordering and deduplication.
## Faster Hashing and Map Access
- Micro-benchmarks compared hash functions according to speed and uniqueness.
- Murmur3 performed best for Datadog’s requirements.
- Datadog also changed metric contexts from 128-bit to 64-bit hashes.
- A 64-bit hash still provided sufficient collision resistance for the use case and enabled Go runtime optimizations:
- `runtime.mapassign_fast64`
- `runtime.mapaccess2_fast64`
- These optimized map operations improved both context storage and metric sampling performance.
## Redesigning the Algorithm
- Sorting served two purposes: producing an ordered tag list and helping deduplicate tags.
- Because sorting was the largest bottleneck, Datadog began exploring a design that could address these responsibilities more efficiently rather than relying on a single general-purpose sort.
The practical lesson is to profile under realistic load, optimize the hottest paths, and combine targeted specialization, benchmark-driven implementation choices, and data-structure redesign to increase throughput without adding CPU capacity.
Datadog created DRUIDS in 2018 to provide consistent UX patterns across its rapidly expanding observability platform. The system succeeds only if designers and developers trust and use it, so its documentation and tooling are designed to be easy to understand, implement, and contribute to. Datadog’s approach connects design, source code, documentation, and development workflows into a shared source of truth.
## Easy to Understand
### Fast Navigation and Discovery
- The DRUIDS documentation supports Cmd+K (or Ctrl+K) quick navigation for finding components, patterns, icons, logos, and other resources.
- The “DRUIDS Loupe” lets Datadog employees inspect components directly on product pages.
- Hovering over an inspected component reveals links to its source code, Figma files, and documentation.
### Connected Documentation
- Component pages link to GitHub, Figma, and VS Code.
- JSDoc comments provide descriptions and links back to DRUIDS documentation inside VS Code.
- Figma also links back to relevant components and guidance.
- These connections minimize dead ends and keep supporting context close to where people are working.
## Easy to Implement
### Editable Playgrounds
- DRUIDS treats React, TypeScript, and CSS implementation as the source of truth because code defines the actual browser experience.
- Interactive playgrounds let users explore component props, test variations, and copy production-ready code.
- A Code Sandbox page allows multiple components to be combined into live prototypes.
- Stateful internal URLs make it easier to share ideas and reproduce bugs.
- The tools also help teams without dedicated frontend specialists build internal applications using familiar Datadog patterns.
### Auto-Generated API Documentation
- Each component includes an API table listing its props, allowed values, and descriptions.
- The tables are generated directly from source code rather than maintained separately.
- This prevents documentation from drifting away from the implementation across more than 150 components.
- Clear, meaningful prop names improve predictability for both designers and developers.
## Easy to Contribute Back
### Contribution Guidelines
- DRUIDS is intended to evolve as Datadog’s products and customer needs change.
- Internal contribution guidelines define expectations for core design, structure, prop naming, documentation, styling, responsiveness, accessibility, and testing.
- The guidelines aim to reinforce quality without making contributions burdensome or intimidating.
### CLI Tooling
- A command-line wizard generates the standard structure for new components, including code, tests, and documentation examples.
- For example, `yarn component button` creates the scaffolding for a button component.
- Automating boilerplate lets contributors focus on design, user experience, and performance.
Datadog’s central recommendation is to treat a design system as more than a component library: it should connect users, designers, and developers through consistent patterns, live examples, source-linked documentation, and lightweight contribution workflows.
The provided content does not include the blog post itself; it contains Datadog navigation links and a URL titled “Druids: The Design System That Powers Datadog.” As a result, the article’s arguments, implementation details, and conclusions cannot be reliably summarized without inventing information.
## Available Information
- The page appears to be a Datadog engineering blog post about **Druids**, Datadog’s design system.
- The surrounding navigation lists Datadog products across:
- Infrastructure and application monitoring
- Logs, security, and digital experience
- Software delivery and service management
- AI and platform capabilities
- No article sections, examples, technical architecture, or conclusions are included in the supplied text.
## Practical Conclusion
Please provide the article body or a working copy of the page content for an accurate section-by-section summary.
Tay Nishimura’s career shows that succeeding in tech is often less about fitting a standard engineering mold and more about finding work that matches one’s strengths. Although she initially struggled with the speed and coding demands of software development, her rigor, visual thinking, and careful approach became valuable in site reliability engineering. Her transition was enabled by self-directed learning, community education, and ToyNet, an open source networking simulator that demonstrated her practical abilities.
## Entering Tech from Mathematics
- Tay began as a mathematics major focused on real analysis, then added computer science after advice from a professor.
- Internships at Amazon and Google introduced her to the technology industry.
- She found a sharp contrast between academia and industry:
- School rewarded theoretical rigor.
- Industry emphasized practical, fast, and agile solutions.
- Tay also felt like an outsider because she had little exposure to computers growing up.
## Struggling with Traditional Software Engineering
- Coding did not come naturally to Tay’s visual way of thinking.
- She translated code into drawings to understand and modify it, then converted those ideas back into code.
- This process produced high-quality, careful work but made her slower than colleagues expected.
- A manager suggested product management and site reliability engineering as possible alternatives.
- Tay discovered that her deliberate pace was useful for SRE work, particularly when evaluating failure modes and making critical changes.
- Because her company offered no path into those roles, she eventually left rather than continue facing increasing stress.
## Discovering Networking and Technical Program Work
- Tay’s next role had a software engineer title but involved work closer to product or technical program management.
- She learned that job titles and responsibilities vary significantly between companies.
- With better work-life balance, she began studying computer networking in her free time.
- She created visual diagrams and learning modules to explain switches, routers, and packet flows.
- These efforts became Project Reclass, a nonprofit teaching technical skills to incarcerated people and military veterans.
- The program used improvised equipment, such as fake routers and switches, to teach concepts in environments where real networking hardware was unavailable.
## Building ToyNet During the Pandemic
- After her company laid off its entire office during COVID-19, Tay decided to pursue SRE directly.
- When prisons suspended in-person education, Project Reclass adapted by creating a digital networking simulator.
- Tay architected ToyNet, an open source platform built with:
- React
- A Flask backend
- Containerized Mininet instances for network emulation
- Users can connect simulated routers, switches, and hosts, configure IP addresses, and run commands such as `ping` and `arp`.
- ToyNet was designed to work for incarcerated learners with restricted internet access.
- Deploying it in the cloud also gave Tay practical experience that helped compensate for limited professional cloud experience.
- Companies interested in the project were more likely to advance her through the interview process, eventually leading to Datadog.
## Finding the Right Environment at Datadog
- At Datadog, Tay learned Kubernetes, chaos engineering, network traffic control, and Go.
- She found that her rigor and visual thinking were assets rather than liabilities.
- While learning Datadog’s Chaos Controller codebase, she mapped files and dependencies by drawing boxes and arrows.
- Her experience suggests that engineers do not need to learn or reason in a single conventional way; the right environment can turn an apparent weakness into a strength.
Tay’s path recommends experimenting broadly, studying independently, and building concrete projects that reveal how you think and solve problems. The most suitable tech role may emerge only after moving between companies and disciplines rather than forcing yourself to succeed in an ill-fitting position.
Datadog announces that it has been named a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms. The provided content primarily consists of Datadog’s navigation and product listings, so it does not include Gartner’s evaluation details or the announcement’s supporting arguments.
## Recognition as an Observability Leader
- Datadog highlights its placement as a Leader in Gartner’s 2026 Magic Quadrant for Observability Platforms.
- The announcement links to a Gartner-related resource hosted by Datadog.
- No specific Gartner strengths, cautions, scoring, or comparison with other vendors are included in the provided text.
## Datadog’s Broad Platform Coverage
The navigation presents Datadog as a unified platform spanning:
- **Infrastructure:** infrastructure, container, network, serverless, GPU, storage, and cloud-cost monitoring.
- **Applications and data:** APM, universal service monitoring, continuous profiling, database monitoring, data-stream monitoring, and jobs monitoring.
- **Logs and security:** log management, observability pipelines, sensitive-data scanning, cloud security, SIEM, workload protection, and application/API protection.
- **Digital experience:** browser and mobile RUM, session replay, synthetic monitoring, product analytics, experiments, and error tracking.
- **Software delivery and service management:** CI visibility, test optimization, feature flags, code coverage, incident response, SLOs, workflow automation, and case management.
- **AI:** agent observability, GPU monitoring, AI integrations, AI agents, investigation tools, and an MCP server.
Overall, the announcement positions Datadog’s broad, integrated observability and security platform as the basis for its Leader designation, but the supplied excerpt does not provide enough detail to assess Gartner’s underlying evaluation.
Datadog’s page announces that Gartner named it a Leader in the 2026 Magic Quadrant for Observability Platforms. The provided content does not include the report’s evaluation criteria, Gartner’s analysis, or Datadog’s supporting evidence; it primarily contains website navigation and links to Datadog products.
## Gartner Recognition
- Datadog highlights its placement as a **Leader** in Gartner’s Magic Quadrant for Observability Platforms.
- The announcement links to a downloadable Gartner resource.
- No details are provided about Datadog’s position, strengths, weaknesses, or comparison with other vendors.
## Datadog’s Product Portfolio
The page navigation presents Datadog as a broad observability and operations platform covering:
- **Infrastructure:** infrastructure, container, network, serverless, GPU, storage, and cloud-cost monitoring.
- **Applications:** APM, universal service monitoring, continuous profiling, dynamic instrumentation, and agent observability.
- **Data and logs:** database, data-streams, quality, jobs, log management, sensitive-data scanning, and observability pipelines.
- **Security:** code, cloud, workload, vulnerability, compliance, SIEM, and application/API protection.
- **Digital experience:** browser and mobile RUM, session replay, synthetic monitoring, product analytics, experiments, and error tracking.
- **Software delivery and service management:** CI visibility, test optimization, internal developer portals, incident response, SLOs, workflow automation, and case management.
- **AI capabilities:** AI integrations, GPU monitoring, Bits AI agents, investigation tools, MCP support, and agent observability.
## What the Provided Content Does Not Cover
- Gartner’s methodology or assessment criteria
- Specific reasons Datadog was named a Leader
- Customer feedback, market vision, or execution scores
- Technical architecture, pricing, implementation guidance, or product comparisons
The material supports the conclusion that Datadog is promoting broad platform coverage and third-party recognition, but the linked Gartner report would be needed for a substantive evaluation.
Datadog built Husky, a new event-storage system, after its original log architecture struggled with multi-tenant reliability, rapid platform growth, and evolving product requirements. The post explains how Datadog moved from metrics-oriented storage to event storage, introduced custom sharding and routing, and eventually recognized the need for a more flexible system. Husky emerged from these lessons about isolation, scalability, and retaining high-cardinality event data.
## From Metrics to Logs
- Metrics systems store pre-aggregated tuples such as `<timeseries_id, timestamp, float64>`.
- Aggregation makes metrics extremely efficient: millions of events in a second can become one compact datapoint, often requiring less than two bytes with delta-of-delta encoding.
- This model is poorly suited to logs because logs must preserve individual events and their full context.
- Metrics typically favor long-lived, low-cardinality dimensions such as:
- Datacenter
- Service
- Pod name
- Short-lived, high-cardinality fields such as transaction IDs and packet IDs are usually pre-aggregated or omitted.
- Logs instead need to support:
- Multi-kilobyte events
- High-cardinality values such as UUIDs and stack traces
- Arbitrary aggregations performed at query time
## Limitations of the Initial Logs System
- Datadog’s first Logs architecture initially worked well but became vulnerable in a multi-tenant environment.
- A single unhealthy or overloaded node could degrade service for every tenant in the cluster.
- Scaling overloaded clusters could worsen the situation because nodes began streaming data to one another while already handling excessive read and write workloads.
- Diagnosing and mitigating these cascading failures was difficult.
## Separating Storage from Clustering
Datadog’s second architecture retained the same single-node storage engine but moved clustering responsibilities into dedicated services.
- Storage nodes no longer knew about one another and behaved like independent one-node clusters.
- Failures were isolated to the tenants assigned to a particular shard instead of spreading across the entire cluster.
- A Shard Router:
- Read events from Kafka
- Reorganized them into shard-based Kafka partitions
- Dynamically assigned tenants to an appropriate number of shards based on their recent five-minute data volume
- Each shard was consumed by two storage-node replicas for redundancy.
- A custom query engine tracked tenant-to-shard assignments, queried the relevant replicas, merged partial aggregates, and produced final results.
## Growth of the Event Platform
- The new architecture substantially improved reliability and reduced operational burden.
- Datadog expanded the platform beyond Logs to support products including:
- Network Performance Monitoring
- Real User Monitoring
- Continuous Profiler
- These products generated structured, multi-kilobyte events with storage and indexing requirements similar to logs.
- As usage grew, new problems appeared:
- A tenant producing a sudden burst of events could degrade query performance for other tenants sharing its shard.
- Product teams requested longer retention for important but infrequently queried data, while still requiring it to remain immediately queryable.
- The existing architecture was increasingly difficult to adapt to these isolation, scalability, and retention requirements, motivating the development of Husky.
Datadog’s progression shows that event storage cannot simply reuse metrics-oriented designs. Systems must preserve event-level context, isolate tenants from one another, and support changing retention and query requirements as products and workloads evolve.
Datadog’s IT team built an automated system to identify inactive user accounts and reduce unnecessary SaaS spending. The approach replaces manual audits with data-driven workflows that detect inactivity, notify users or owners, and reclaim unused licenses while preserving access controls and accountability.
## Automating Account Inactivity Detection
- The system monitors account activity across SaaS applications.
- It identifies users who have not logged in or used assigned tools for a defined period.
- Automated notifications give users or managers an opportunity to confirm continued business need.
- Accounts can then be suspended, deprovisioned, or escalated for review.
## Managing SaaS Spend
- Inactivity data is used to find unused or underused licenses.
- IT can reclaim seats instead of continuing to pay for unused subscriptions.
- Usage information supports more accurate renewal and purchasing decisions.
- Centralized automation reduces the manual effort required to audit many applications.
## Governance and Operational Benefits
- Standardized workflows make account reviews more consistent across tools.
- Automated approvals and escalation paths provide visibility into decisions.
- The process helps balance cost reduction with security and user access requirements.
- IT teams gain a repeatable way to manage the growing complexity of SaaS environments.
Organizations with substantial SaaS usage can apply the same model: centralize activity data, define inactivity policies, automate notifications and approvals, and connect the results to license reclamation and access-management workflows.
Datadog expanded its Clarity auditing tool into Clarity License Manager (CLM), a system that tracks SaaS usage, reduces licensing costs, and improves security. CLM identifies inactive accounts, notifies employees, automatically deactivates unused access, and restores it quickly when needed. Its microservice architecture and application-specific adapters allow the system to scale across many SaaS products.
## The SaaS License Management Problem
- Datadog used many commercial SaaS tools with substantial per-user costs.
- License usage data was outdated and collected through quarterly manual audits.
- IT Support had to contact employees individually, creating administrative overhead and a poor user experience.
- Unused accounts also created security risks, including stale credentials that could be compromised.
## Goals of Clarity License Manager
- Monitor and automatically deactivate inactive accounts, especially in sensitive services such as cloud providers.
- Reduce the risk of leaked or abused stale credentials.
- Limit the potential impact of security incidents.
- Lower SaaS spending and support data-driven licensing decisions.
- Preserve employee productivity through an easy account restoration process.
## Usage Monitoring and Automated Workflows
- CLM gathers activity data through:
- Direct integrations with individual SaaS APIs.
- Google Workspace SAML audit logs for indirect integrations.
- Employee activity is stored per application in an Amazon RDS-backed PostgreSQL database.
- Employees receive email and Slack notifications after a configurable period of inactivity, with 90 days as the default.
- Notifications explain the specific login or application action required to remain active.
- If the employee does not respond after multiple reminders, CLM deactivates the account automatically.
- Employees can reopen access by submitting a Freshservice ticket.
- Accounts are restored within seconds, including their previous roles and permissions.
## Microservice Architecture
- CLM consists of Python microservices running on AWS Lambda.
- The services share a central PostgreSQL database.
- Microservices provide:
- Easier scaling as Datadog adds more SaaS applications.
- Greater resilience and flexibility.
- A modular foundation for future development.
- The architecture introduced complexity because services required different APIs and libraries with overlapping functionality.
## Application-Specific Adapters
- Each SaaS product is represented by an adapter shared across CLM microservices.
- Adapters isolate application-specific API logic from the core workflows.
- A typical adapter supports operations such as:
- Retrieving users.
- Fetching login activity.
- Activating and deactivating accounts.
- Onboarding and offboarding users.
- This design provides:
- Clear separation of responsibilities.
- Reusable and flexible integration code.
- Simpler microservices that do not need to handle each application’s unique behavior.
CLM demonstrates how automated usage monitoring can simultaneously improve SaaS security, reduce unnecessary spending, and minimize disruption for employees. A modular adapter-based architecture is particularly useful when managing a growing portfolio of third-party applications.
The supplied content does not include the blog post itself; it mainly contains Datadog navigation links and a promotional banner stating that Datadog was named a Leader in Gartner’s Magic Quadrant for Observability Platforms. The URL suggests the missing article concerns a gRPC, DNS, and load-balancing incident, but no incident details are provided.
## Available Content
- Datadog promotes its recognition as a Gartner Magic Quadrant Leader.
- The navigation lists products across:
- Infrastructure and application monitoring
- Logs, databases, and data observability
- Security
- Digital experience monitoring
- Software delivery and service management
- AI and platform capabilities
- The linked page path references an engineering post titled “gRPC, DNS, and Load Balancing Incident.”
## Missing Technical Details
- No description of the incident or its impact
- No explanation of the DNS or load-balancing failure
- No timeline, root-cause analysis, or remediation steps
- No lessons learned or recommendations
Please provide the full article text for a substantive technical summary.
A routine update to a critical metrics query service caused intermittent errors and increased latency. Although logs initially pointed to DNS failures, the investigation revealed a deeper networking problem involving dropped packets and saturated AWS VPC connection tracking. The incident highlighted how Kubernetes, Cilium, AWS networking, and DNS behavior can interact in ways that obscure the true cause.
## Initial Symptoms and Apparent DNS Failures
- Errors increased whenever the metrics query service was rolled out.
- The service retrieves data from metric stores for dashboards and monitor evaluations.
- Automatic retries reduced user-facing failures but increased latency.
- Service logs showed DNS errors when connecting to dependencies inside Kubernetes.
- The investigation therefore began with the cluster’s DNS infrastructure.
## NodeLocal DNSCache Reaches Its Limits
- NodeLocal DNSCache runs as a `node-local-dns` DaemonSet on every Kubernetes node.
- DNS pods had:
- A 64 MB memory limit
- A `max_concurrent` limit of 1,000 requests
- The pods experienced out-of-memory errors and rejected requests during rollouts.
- Increasing memory to 256 MB stopped the OOM errors, but DNS failures continued.
- Request volume was far below the expected capacity:
- Normally about 400 queries per second
- Nearly 2,000 queries per second during rollouts
- Expected capacity of at least 200,000 queries per second
- Upstream resolvers were marked unhealthy, suggesting that NodeLocal DNSCache could not establish or maintain connections.
- Because upstream requests could wait up to five seconds, connection failures consumed concurrency slots and made the cache appear overloaded.
## Evidence of a Network Problem
- The instances were below their 5-Gbps sustained throughput limits.
- TCP retransmits increased in correlation with service rollouts.
- Engineers suspected brief traffic spikes, or microbursts, that were not visible in aggregate throughput metrics.
- This shifted the investigation from DNS configuration toward lower-level AWS networking behavior.
## AWS VPC Connection Tracking
- ENA metrics revealed a significant increase in `conntrack_allowance_exceeded`.
- This metric counts packets dropped when VPC connection tracking becomes saturated.
- Connection tracking maintains state for network flows and supports features such as stateful EC2 security groups.
- The infrastructure used two tracking layers:
- VPC conntrack maintained at the hypervisor level
- Linux conntrack inside each instance
- VPC conntrack appeared saturated even though Linux conntrack contained fewer than 60,000 entries—well within the observed capacity of similar instances.
- AWS Support confirmed that conntrack capacity varies by instance type and that VPC conntrack limits could differ substantially from Linux conntrack limits.
- Scaling to larger instances resolved the symptoms, but the engineers wanted to understand the traffic pattern and find a more efficient long-term solution.
## VPC Flow Logs as the Next Investigation Tool
- The team turned to Amazon VPC Flow Logs to examine the service’s low-level network behavior.
- These logs were expected to clarify why connection tracking filled up and how rollout traffic contributed to the saturation.
- The investigation was still ongoing at the point where the provided article excerpt ends.
The Dirty Pipe vulnerability (CVE-2022-0847) is a critical Linux kernel flaw that allows unprivileged processes to write data to any file they can read, effectively bypassing standard write permissions. This primitive is particularly dangerous in containerized environments like Kubernetes, where it can be leveraged to overwrite the host’s container runtime binary. By exploiting how the kernel manages page caches, an attacker can achieve a full container breakout and gain administrative privileges on the underlying host.
## Container Runtimes and the OCI Specification
* Kubernetes utilizes the Container Runtime Interface (CRI) to manage containers via high-level runtimes like containerd or CRI-O.
* These high-level runtimes rely on low-level Open Container Interface (OCI) runtimes, most commonly runC, to handle the heavy lifting of namespaces and control groups.
* Isolation is achieved by runC setting up a restricted environment before executing the user-supplied entrypoint via the `execve` system call.
## Evolution of runC Vulnerabilities
* A historical vulnerability, CVE-2019-5736, previously allowed escapes by overwriting the host’s runC binary through the `/proc/self/exe` file descriptor.
* To mitigate this, runC was updated to either clone the binary before execution or mount the host's runC binary as read-only inside the container.
* While the read-only mount improved performance through kernel cache page sharing, it created a target for the Dirty Pipe vulnerability, which specifically targets the kernel page cache.
## The Dirty Pipe Exploitation Primitive
* Dirty Pipe allows an attacker to overwrite any file they can read, including read-only files, by manipulating the kernel's internal pipe-buffer structures.
* The exploit targets the page cache, meaning the overwrite is non-persistent and resides only in memory; the original file on disk remains unchanged.
* In a container escape scenario, the attacker waits for a runC process to start (triggered by actions like `kubectl exec`) and targets the file descriptor at `/proc/<runC-pid>/exe`.
## Proof-of-Concept Escape Walkthrough
* The attack begins with a standard, unprivileged pod running a malicious script that monitors the system for new runC processes.
* Once a `kubectl exec` command is issued by an administrator, the script identifies the runC PID and applies the Dirty Pipe exploit to the associated executable.
* The exploit overwrites the runC binary in the kernel page cache with a malicious ELF binary.
* Because the host kernel is executing this hijacked binary with root privileges to manage the container, the attacker’s malicious code (e.g., a reverse shell or administrative command) runs with full host-level authority.
To protect against this attack vector, it is essential to patch the Linux kernel to a version that includes the fix for CVE-2022-0847 and ensure that container nodes are running updated distributions.
The post demonstrates how the Linux Dirty Pipe vulnerability can enable an unprivileged process to escape a container and gain administrative privileges on the host. The exploit abuses runC’s execution model and its host binary, which is exposed read-only inside the container but can still be modified through the kernel page cache. A proof of concept shows how a compromised Kubernetes pod can overwrite runC with a malicious executable when an administrator runs `kubectl exec`.
## Container Runtimes and runC
- Kubernetes commonly uses containerd or CRI-O through the Container Runtime Interface (CRI).
- These runtimes rely on lower-level OCI runtimes, most notably runC, to create isolated Linux processes.
- runC configures namespaces, cgroups, and the container environment before executing the supplied entrypoint with `execve`.
- During execution, `/proc/self/exe` inside the container can refer to an open descriptor for the runC binary on the host.
## Earlier runC Escape Vulnerability
- CVE-2019-5736 exploited this `/proc/self/exe` behavior:
- A malicious container entrypoint could write to the host’s runC binary.
- Overwriting runC enabled subsequent container operations to execute attacker-controlled code with host-level privileges.
- runC initially mitigated the issue by cloning its binary before execution.
- It later changed the design to mount the runC binary read-only inside the container, improving performance through kernel page-cache sharing.
- That optimization created conditions in which Dirty Pipe could bypass the apparent read-only protection.
## Dirty Pipe as a Container Escape Primitive
- Dirty Pipe allows an unprivileged process to overwrite files it can read, even without write permission.
- The modification occurs in the kernel page cache rather than persistent storage:
- The original file remains intact on disk.
- Dropping caches or rebooting can restore the original contents.
- Despite being temporary, the overwrite is sufficient to execute malicious code when the modified binary is run.
- In this case, the attacker targets the host’s runC binary through `/proc/<runC-pid>/exe`.
## Kubernetes Proof of Concept
- The demonstration starts an ordinary, unprivileged pod using an attacker-controlled container image.
- Its entrypoint script:
- Replaces `/bin/sh` with a launcher referencing `/proc/self/exe`.
- Waits for a runC process to appear.
- Invokes the Dirty Pipe exploit against that process’s executable.
- An administrator running `kubectl exec` causes runC to execute inside the container, triggering the overwrite.
- The modified runC is replaced with a malicious ELF binary that runs commands such as `id` and `hostname`, recording their output in `/tmp/hacked`.
- The exploit is adapted from the original Dirty Pipe proof of concept and the earlier runC escape technique.
The attack illustrates that kernel vulnerabilities can undermine container isolation even when the container is unprivileged and the target binary is mounted read-only. Systems should promptly patch vulnerable Linux kernels and container runtimes, while treating compromised containers as potential paths to host compromise.
