datadog·Feb 13, 2024·3 min readCurated summary

.NET Continuous Profiler: CPU and wall time profiling

Datadog’s .NET profiler uses low-overhead thread sampling to collect CPU and wall-time profiles suitable for production. CPU profiling measures time spent running on a processor, while wall-time profiling captures delays caused by I/O, locks, or scheduling. The implementation tracks managed threads, walks their stacks, aggregates samples, and periodically uploads profiles, with platform-specific optimizations for Windows and Linux. ## CPU and Wall-Time Profiling - CPU profiling identifies code consuming processor cycles. - Wall-time profiling finds slow methods regardless of whether threads are running, blocked, or waiting for I/O. - ETW, Linux `perf`, and .NET enter/leave hooks provide profiling data but impose too much overhead or require elevated privileges. - Datadog instead samples threads at intervals, captures their call stacks, and assigns durations to the samples. ## Managed Thread Sampling - The profiler tracks managed threads created through: - The thread pool - `Task` and `async/await` - Explicit `Thread` instances - `ICorProfilerCallback` notifications such as `ThreadCreated` and `ThreadDestroyed` maintain the `ManagedThreadList`. - `StackSamplerLoop` captures stacks and sends raw samples to `CpuTimeProvider` and `WallTimeProvider`. - `SamplesCollector` aggregates provider data, while a shared Rust exporter uploads profiles to Datadog every minute. - An early C# implementation caused native worker threads entering managed code to appear as application threads. Removing that implementation eliminated accidental self-sampling. ## Native Runtime Threads - Server-mode garbage collection uses native threads that can compete with application threads for CPU. - Because the .NET profiling API does not expose these threads directly, the profiler identifies them by their .NET 5 names: - `.NET Server GC` - `.NET BGC` - Their CPU usage is displayed under a Garbage Collector frame in the flame graph. ## CPU Sampling and Performance Optimizations - Every 10 milliseconds, the profiler searches for runnable managed threads, skipping up to 64 non-runnable threads. - `IsRunning` determines whether a thread is executing on a CPU: - Windows uses `NtQueryInformationThread`. - Linux reads `/proc/self/task/<tid>/stat`. - CPU sample duration is calculated from the difference between the thread’s current and previously recorded CPU consumption. - The initial Linux implementation used `std::ifstream` and `std::getline`, allocating an 8 KB buffer for each sample. - Replacing them with lower-level C file-reading code eliminated allocations and reduced CPU usage. The original approach consumed nearly 500 MB of allocations and about 2% of total CPU in testing. ## Wall-Time Profiling and Code Hotspots - Wall-time profiles work with Datadog tracing to explain why requests are slow. - The tracer provides the profiler with a span ID when a thread handles a request. - To avoid repeated expensive P/Invoke calls, the profiler exposes a memory location that the tracer can update directly. - Because short requests may finish before their thread is sampled, the profiler samples an additional group of ten span-associated threads beyond the initial five. - The duration between consecutive samples of a thread is attributed to the later sample, so heavily threaded applications produce longer individual wall-time sample durations.

Read original(opens in new tab)

LinuxDatadogRustWindows+4

datadog·Mar 25, 2022Original article

Using the Dirty Pipe vulnerability to break out from containers | Datadog (opens in new tab)

The Dirty Pipe vulnerability (CVE-2022-0847) is a critical Linux kernel flaw that allows unprivileged processes to write data to any file they can read, effectively bypassing standard write permissions. This primitive is particularly dangerous in containerized environments like Kubernetes, where it can be leveraged to overwrite the host’s container runtime binary. By exploiting how the kernel manages page caches, an attacker can achieve a full container breakout and gain administrative privileges on the underlying host. ## Container Runtimes and the OCI Specification * Kubernetes utilizes the Container Runtime Interface (CRI) to manage containers via high-level runtimes like containerd or CRI-O. * These high-level runtimes rely on low-level Open Container Interface (OCI) runtimes, most commonly runC, to handle the heavy lifting of namespaces and control groups. * Isolation is achieved by runC setting up a restricted environment before executing the user-supplied entrypoint via the `execve` system call. ## Evolution of runC Vulnerabilities * A historical vulnerability, CVE-2019-5736, previously allowed escapes by overwriting the host’s runC binary through the `/proc/self/exe` file descriptor. * To mitigate this, runC was updated to either clone the binary before execution or mount the host's runC binary as read-only inside the container. * While the read-only mount improved performance through kernel cache page sharing, it created a target for the Dirty Pipe vulnerability, which specifically targets the kernel page cache. ## The Dirty Pipe Exploitation Primitive * Dirty Pipe allows an attacker to overwrite any file they can read, including read-only files, by manipulating the kernel's internal pipe-buffer structures. * The exploit targets the page cache, meaning the overwrite is non-persistent and resides only in memory; the original file on disk remains unchanged. * In a container escape scenario, the attacker waits for a runC process to start (triggered by actions like `kubectl exec`) and targets the file descriptor at `/proc/<runC-pid>/exe`. ## Proof-of-Concept Escape Walkthrough * The attack begins with a standard, unprivileged pod running a malicious script that monitors the system for new runC processes. * Once a `kubectl exec` command is issued by an administrator, the script identifies the runC PID and applies the Dirty Pipe exploit to the associated executable. * The exploit overwrites the runC binary in the kernel page cache with a malicious ELF binary. * Because the host kernel is executing this hijacked binary with root privileges to manage the container, the attacker’s malicious code (e.g., a reverse shell or administrative command) runs with full host-level authority. To protect against this attack vector, it is essential to patch the Linux kernel to a version that includes the fix for CVE-2022-0847 and ensure that container nodes are running updated distributions.

LinuxKubernetesDocker & ContainersContainer Runtime+4

datadog·Feb 28, 2022·3 min readCurated summary

Profiling improvements in Go 1.18

Go 1.18 introduced major profiling improvements alongside features such as generics and fuzzing. Its Linux CPU profiler became substantially more accurate on multicore systems by addressing dropped `SIGPROF` signals, and profiler labels received an important correctness fix. These changes strengthened Go’s ability to connect continuous profiling data with tracing systems such as Datadog. ## More Accurate Linux CPU Profiling - Earlier Go versions used `setitimer(2)` to request a `SIGPROF` signal every 10 ms of CPU time. - On busy multicore systems, Linux could generate multiple signals within a single kernel “jiffy” window, but standard POSIX signals do not queue. - As a result, many signals were dropped: - A service using 20 CPU cores might generate roughly 2,000 profiling signals per second. - Its Go profile could contain only about 240 samples per second. - Linux’s software clock, commonly operating at 250 Hz, could only measure CPU time in roughly 4 ms intervals. This caused signal bursts and undercounted CPU usage. - `setitimer(2)` also distributed process-directed signals unevenly across threads, creating additional profiling bias. ## Combining `timer_create` and `setitimer` - `timer_create(2)` provided more reliable per-thread signal accounting and avoided most of the signal-dropping and thread-bias problems. - Its drawback was that the profiler needed awareness of every thread, including threads created independently by cgo code. - The Go 1.18 fix combined both timer mechanisms: - The signal handler identifies the signal source. - Signals from inferior sources are discarded. - The implementation accounts for short-lived threads and cgo edge cases. - The work originated from investigations into Go issues GH 35057 and GH 14434 and was developed through collaboration between contributors and Go maintainers. ## Profiler Label Correctness - Profiler labels, also called pprof labels or tags, associate key/value metadata with goroutines. - Labels are inherited by child goroutines and appear in CPU and goroutine profiles, enabling profiles to be filtered by request, service, or trace metadata. - Testing at Datadog revealed that some stack samples were missing labels they should have carried. - The cause was a CPU profiler lookup using the wrong goroutine reference. - The fix changed the profiler to use `gp.m.curg`, the thread’s actual current goroutine, rather than relying on `gp`, which can differ in certain runtime situations. Go 1.18’s profiling changes made CPU measurements more trustworthy on Linux and improved the accuracy of metadata attached to profile samples. Together, they provided a stronger foundation for correlating Go profiling with distributed tracing.

Read original(opens in new tab)

LinuxGoCpu ProfilingProfiling+4

datadog·May 17, 2017·2 min readCurated summary

Being a solutions engineer at Datadog

Solutions Engineers at Datadog combine customer support, technical investigation, and product improvement. They troubleshoot issues across configuration, dashboards, metrics, and alerting while collaborating with customers and engineers. The role also offers structured opportunities to build engineering skills and shape a career toward engineering, product management, or sales engineering. ## Role and Responsibilities - Serve as the technical point of contact for Datadog customers. - Handle tickets involving: - Agent and integration configuration - Dashboard and metric visualization - Alerting issues and bugs - Investigate customer needs, inspect source code, fix bugs when necessary, and coordinate with engineering teams. - Provide short-term workarounds and identify potential product enhancements. - Communicate with customers through chat and technical calls. ## Learning and Collaboration - New Solutions Engineers rapidly develop skills in technologies such as Linux, Ruby, and SQL. - The role requires multitasking while continuously learning new technical concepts. - Team members share knowledge through Datadog documentation and collaborative projects. - Customer feedback provides direct insight into how the product could evolve. ## Engineering Experience and Career Growth - During two-week “embedding” sprints, Solutions Engineers join engineering teams and work on projects alongside developers. - Embedding provides hands-on experience with: - Datadog’s internal technologies - Engineering and design challenges - Architectural decisions - Cross-team collaboration - Once fully trained, employees can pursue side projects such as improving demo environments or automating internal processes. - Career paths can be tailored toward engineering, product management, or sales engineering. Datadog presents the Solutions Engineer role as a strong fit for people who enjoy customer interaction, technical problem-solving, collaboration, and continuous learning. The company was hiring for these positions in New York, Paris, and remote locations.

Read original(opens in new tab)

LinuxDatadogSqlRuby+3