.NET Continuous Profiler: Memory usage

Datadog’s .NET memory profiler helps identify excessive garbage collection, allocation hotspots, and objects that remain in memory after collection. It combines CLR events, operating-system thread metrics, sampled allocation data, stack traces, and weak handles to provide production-friendly memory insights. The approach favors low overhead, though some capabilities depend on the .NET version.

Measuring Garbage Collector CPU Impact

• The profiler uses CLR events to monitor garbage collection phases.
• In server GC mode, the CLR creates two high-priority threads per heap/core to process collections in parallel.
• Since .NET 5, these threads are named .NET Server GC and .NET BGC.
• At each profile export, the profiler retrieves these threads’ CPU usage from the operating system.
• It records the result as a sample with a native stack containing a Garbage Collector frame.
• This uses a pull model: the exporter periodically requests the CPU measurement because no suitable event or dedicated profiler thread exists.
• Before .NET 5, GC thread CPU usage could not reliably be identified because GCCreateConcurrentThread did not include thread IDs.

Sampling Allocations

• Per-allocation callbacks such as ICorProfilerCallback::ObjectAllocated provide detailed data but significantly slow allocation fast paths.
• GCSampledObjectAllocation and ObjectsAllocatedByClass reduce some costs but do not provide call stacks for individual allocation sites.
• Datadog instead listens to AllocationTick, emitted for roughly every 100 KB allocated.
• Each event includes:
  • The object’s ClassID and type information.
  • The allocation address.
  • The object size and total allocation size since the previous tick.
  • The allocation kind: SOH (0), LOH (1), or POH (2).
• Generic type names are reconstructed through the .NET profiling API.
• Because allocation events are synchronous, the current thread is responsible for the allocation; the profiler walks that thread’s stack to capture the allocation call site.
• This produces sampled allocation data for each heap category without imposing the cost of observing every allocation.

Tracking Objects That Survive Garbage Collection

• An allocation address alone cannot track an object indefinitely because compacting garbage collections can move objects.
• Datadog uses weak handles, created through GCHandle.Alloc, which move with objects and do not keep them alive.
• The profiler added this functionality through the .NET 7 ICorProfilerInfo13 API and its LiveObjectsProvider.
• For every sampled allocation, it creates a weak handle and records the object’s creation time.
• After each garbage collection:
  • Handles for unreachable objects are removed and destroyed.
  • Handles for surviving objects remain and are included in the next profile.
• This lets users inspect representative objects that persist after collection and investigate potential memory leaks.

Practical Recommendation

Use allocated-memory profiles to find endpoints and types responsible for excessive allocation, then examine surviving-object samples for retention or leak investigations. GC CPU data is especially useful for diagnosing applications whose high CPU usage is driven by frequent or expensive garbage collections.