.NET Continuous Profiler: CPU and wall time profiling
Datadog’s .NET profiler uses low-overhead thread sampling to collect CPU and wall-time profiles suitable for production. CPU profiling measures time spent running on a processor, while wall-time profiling captures delays caused by I/O, locks, or scheduling. The implementation tracks managed threads, walks their stacks, aggregates samples, and periodically uploads profiles, with platform-specific optimizations for Windows and Linux. ## CPU and Wall-Time Profiling - CPU profiling identifies code consuming processor cycles. - Wall-time profiling finds slow methods regardless of whether threads are running, blocked, or waiting for I/O. - ETW, Linux `perf`, and .NET enter/leave hooks provide profiling data but impose too much overhead or require elevated privileges. - Datadog instead samples threads at intervals, captures their call stacks, and assigns durations to the samples. ## Managed Thread Sampling - The profiler tracks managed threads created through: - The thread pool - `Task` and `async/await` - Explicit `Thread` instances - `ICorProfilerCallback` notifications such as `ThreadCreated` and `ThreadDestroyed` maintain the `ManagedThreadList`. - `StackSamplerLoop` captures stacks and sends raw samples to `CpuTimeProvider` and `WallTimeProvider`. - `SamplesCollector` aggregates provider data, while a shared Rust exporter uploads profiles to Datadog every minute. - An early C# implementation caused native worker threads entering managed code to appear as application threads. Removing that implementation eliminated accidental self-sampling. ## Native Runtime Threads - Server-mode garbage collection uses native threads that can compete with application threads for CPU. - Because the .NET profiling API does not expose these threads directly, the profiler identifies them by their .NET 5 names: - `.NET Server GC` - `.NET BGC` - Their CPU usage is displayed under a Garbage Collector frame in the flame graph. ## CPU Sampling and Performance Optimizations - Every 10 milliseconds, the profiler searches for runnable managed threads, skipping up to 64 non-runnable threads. - `IsRunning` determines whether a thread is executing on a CPU: - Windows uses `NtQueryInformationThread`. - Linux reads `/proc/self/task/<tid>/stat`. - CPU sample duration is calculated from the difference between the thread’s current and previously recorded CPU consumption. - The initial Linux implementation used `std::ifstream` and `std::getline`, allocating an 8 KB buffer for each sample. - Replacing them with lower-level C file-reading code eliminated allocations and reduced CPU usage. The original approach consumed nearly 500 MB of allocations and about 2% of total CPU in testing. ## Wall-Time Profiling and Code Hotspots - Wall-time profiles work with Datadog tracing to explain why requests are slow. - The tracer provides the profiler with a span ID when a thread handles a request. - To avoid repeated expensive P/Invoke calls, the profiler exposes a memory location that the tracer can update directly. - Because short requests may finish before their thread is sampled, the profiler samples an additional group of ten span-associated threads beyond the initial five. - The duration between consecutive samples of a thread is attributed to the later sample, so heavily threaded applications produce longer individual wall-time sample durations.
Read original(opens in new tab)