How we wrote a Python profiler | Datadog

The post explains how Datadog built a low-overhead statistical profiler for Python. Rather than tracing every function call, the profiler periodically samples running threads and reconstructs their Python and native call stacks. The main challenge is collecting accurate stack data without pausing applications for too long or introducing unsafe behavior inside CPython.

Why Traditional Profiling Is Expensive

• Deterministic profilers instrument every function call and return.
• This provides detailed data but can significantly slow production workloads.
• A statistical profiler reduces overhead by sampling execution at regular intervals instead of observing every event.

Sampling Python Threads

• The profiler interrupts running threads to capture their current execution state.
• Python’s signal-handling model complicates this because signals are generally processed by the main thread.
• The implementation must coordinate native threads, operating-system signals, and the Python interpreter to sample worker threads reliably.
• Sampling must avoid interfering with application locks or triggering unsafe operations in signal handlers.

Reconstructing Call Stacks

• A useful profile needs both Python-level frames and native stack information.
• The profiler walks Python frames to identify functions, files, and line numbers.
• It also handles time spent in native extensions and the interpreter itself.
• Collected samples are aggregated into call stacks, allowing Datadog to show CPU usage and hotspots across the application.

Balancing Accuracy and Overhead

• Sampling frequency affects the trade-off between detail and runtime cost.
• More frequent samples improve visibility into short-lived work but consume more resources.
• The profiler is designed to operate continuously in production, so it prioritizes low overhead, safe memory handling, and resilience across Python versions and deployment environments.

The central recommendation is to use statistical sampling for always-on production profiling. It provides actionable performance data with far less impact than call-by-call instrumentation, provided the implementation carefully accounts for CPython’s threading, signal, and native-extension behavior.