.NET Continuous Profiler: CPU and wall time profiling

Datadog’s .NET profiler uses low-overhead thread sampling to collect CPU and wall-time profiles suitable for production. CPU profiling measures time spent running on a processor, while wall-time profiling captures delays caused by I/O, locks, or scheduling. The implementation tracks managed threads, walks their stacks, aggregates samples, and periodically uploads profiles, with platform-specific optimizations for Windows and Linux.

CPU and Wall-Time Profiling

• CPU profiling identifies code consuming processor cycles.
• Wall-time profiling finds slow methods regardless of whether threads are running, blocked, or waiting for I/O.
• ETW, Linux perf, and .NET enter/leave hooks provide profiling data but impose too much overhead or require elevated privileges.
• Datadog instead samples threads at intervals, captures their call stacks, and assigns durations to the samples.

Managed Thread Sampling

• The profiler tracks managed threads created through:
  • The thread pool
  • Task and async/await
  • Explicit Thread instances
• ICorProfilerCallback notifications such as ThreadCreated and ThreadDestroyed maintain the ManagedThreadList.
• StackSamplerLoop captures stacks and sends raw samples to CpuTimeProvider and WallTimeProvider.
• SamplesCollector aggregates provider data, while a shared Rust exporter uploads profiles to Datadog every minute.
• An early C# implementation caused native worker threads entering managed code to appear as application threads. Removing that implementation eliminated accidental self-sampling.

Native Runtime Threads

• Server-mode garbage collection uses native threads that can compete with application threads for CPU.
• Because the .NET profiling API does not expose these threads directly, the profiler identifies them by their .NET 5 names:
  • .NET Server GC
  • .NET BGC
• Their CPU usage is displayed under a Garbage Collector frame in the flame graph.

CPU Sampling and Performance Optimizations

• Every 10 milliseconds, the profiler searches for runnable managed threads, skipping up to 64 non-runnable threads.
• IsRunning determines whether a thread is executing on a CPU:
  • Windows uses NtQueryInformationThread.
  • Linux reads /proc/self/task/<tid>/stat.
• CPU sample duration is calculated from the difference between the thread’s current and previously recorded CPU consumption.
• The initial Linux implementation used std::ifstream and std::getline, allocating an 8 KB buffer for each sample.
• Replacing them with lower-level C file-reading code eliminated allocations and reduced CPU usage. The original approach consumed nearly 500 MB of allocations and about 2% of total CPU in testing.

Wall-Time Profiling and Code Hotspots

• Wall-time profiles work with Datadog tracing to explain why requests are slow.
• The tracer provides the profiler with a span ID when a thread handles a request.
• To avoid repeated expensive P/Invoke calls, the profiler exposes a memory location that the tracer can update directly.
• Because short requests may finish before their thread is sampled, the profiler samples an additional group of ten span-associated threads beyond the initial five.
• The duration between consecutive samples of a thread is attributed to the later sample, so heavily threaded applications produce longer individual wall-time sample durations.