datadogJan 31, 2023

Performance improvements in the Datadog Agent metrics pipeline | Datadog (opens in new tab)

Datadog engineers recently optimized the Datadog Agent's metric processing pipeline to achieve higher throughput and lower CPU overhead. By identifying that metric context generation—the process of creating unique keys for metrics—was a primary bottleneck, they implemented a series of algorithmic changes and Go runtime optimizations. These improvements allow the Agent to process significantly more metrics using the same computational resources. ### Identifying Bottlenecks via CPU Profiling * Developers utilized Go’s native profiling tools to capture CPU usage during high-volume metric ingestion via DogStatsD. * Flamegraph analysis revealed that the `addSample` and `trackContext` functions were the most CPU-intensive components of the pipeline. * The profiling data specifically pointed to tag sorting and deduplication as the underlying operations consuming the most processing time. ### The Challenges of Metric Context Generation * The Agent must generate a unique hash (context) for every metric received to address it within a hash table in RAM. * To ensure the same metric always generates the same key, the original algorithm required sorting all tags and ensuring their uniqueness. * The computational cost of sorting lists repeatedly for every incoming message created a performance ceiling for the entire metrics pipeline. ### Specialization and Runtime Optimization * **Algorithmic Specialization:** The team implemented specialized sorting logic that adjusts based on the number of tags, optimizing the "hot path" for the most common metric structures. * **Hashing Efficiency:** Micro-benchmarks identified Murmur3 as the most efficient hash implementation for balancing speed and collision resistance in this use case. * **Leveraging Go Runtime:** The team transitioned from 128-bit hashes to 64-bit metric contexts. This change allowed the Agent to utilize Go's internal `mapassign_fast64` and `mapaccess2_fast64` functions, which provide optimized map operations for 64-bit keys. ### Redesigning for Performance * The original design followed a rigid "hash metric name -> sort tags -> deduplicate tags -> iterative hash" workflow. * Recognizing that sorting was the primary architectural bottleneck, the team moved toward a new design intended to minimize or eliminate the overhead of traditional list sorting during context generation. To achieve similar performance gains in high-throughput Go applications, developers should profile their applications under realistic load and look for opportunities to leverage runtime-specific optimizations, such as using 64-bit map keys to trigger specialized compiler paths.

datadog-agentdatabase-designgocpu-profiling+4

datadogJan 31, 2023

Performance improvements in the Datadog Agent metrics pipeline (opens in new tab)

The Datadog Agent needed to process more metrics without increasing CPU usage. Profiling showed that generating unique metric contexts—especially sorting and deduplicating tags—was a major bottleneck. Datadog improved throughput through specialized sorting paths, faster hashing, and a more efficient context-storage design. ## Identifying the Bottleneck - Datadog uses Go’s CPU and memory profiling tools to optimize the Agent’s metrics pipeline. - Profiles were captured while Agents processed large volumes of DogStatsD metrics, ensuring the results reflected real workload pressure. - Flamegraphs showed that `addSample` and `trackContext` consumed the most CPU. - Sorting-related functions, including `util.SortUniqInPlace` and `sort`, were significant contributors to that cost. ## How Metric Contexts Work - Each received metric is assigned a metric context that uniquely identifies it in an in-memory hash table. - The context must incorporate: - The metric name - Tags included in the DogStatsD message - Container-generated tags - The context is computed as a hash, so it must be fast while minimizing collisions. - Tags must be consistently ordered so the same metric always produces the same context. - The original implementation sorted tags and removed duplicates, making sorting a recurring CPU expense. ## Specialized Sorting - Performance varied according to the number of tags attached to a metric. - Datadog introduced specialized sorting paths based on tag count. - This allowed common cases to use more efficient algorithms while retaining correct ordering and deduplication. ## Faster Hashing and Map Access - Micro-benchmarks compared hash functions according to speed and uniqueness. - Murmur3 performed best for Datadog’s requirements. - Datadog also changed metric contexts from 128-bit to 64-bit hashes. - A 64-bit hash still provided sufficient collision resistance for the use case and enabled Go runtime optimizations: - `runtime.mapassign_fast64` - `runtime.mapaccess2_fast64` - These optimized map operations improved both context storage and metric sampling performance. ## Redesigning the Algorithm - Sorting served two purposes: producing an ordered tag list and helping deduplicate tags. - Because sorting was the largest bottleneck, Datadog began exploring a design that could address these responsibilities more efficiently rather than relying on a single general-purpose sort. The practical lesson is to profile under realistic load, optimize the hottest paths, and combine targeted specialization, benchmark-driven implementation choices, and data-structure redesign to increase throughput without adding CPU capacity.

datadog-agentperformance-optimizationgocpu-profiling+3

datadogJun 3, 2019

Secure publication of Datadog Agent integrations with TUF and in-toto | Datadog (opens in new tab)

Datadog describes how it secures the publication and distribution of Datadog Agent integrations using The Update Framework (TUF) and in-toto. TUF protects clients from tampered, outdated, or incorrectly signed packages, while in-toto provides verifiable evidence about how each integration was built. Together, the systems create a chain of trust from source and build processes to the integration installed by an Agent. ## Why Agent integrations need stronger supply-chain security - Integrations are distributed software components that run inside the Datadog Agent. - A compromise of the source repository, build infrastructure, signing credentials, or distribution system could lead to malicious code reaching customers. - Authenticating only the final package is insufficient if attackers can: - Replace repository metadata - Replay an older vulnerable release - Roll back clients to compromised versions - Exploit a stolen signing key - Publish artifacts that were not produced by the approved build process ## TUF for secure package distribution - TUF separates repository responsibilities across cryptographic roles rather than relying on one signing key. - Metadata roles such as root, targets, snapshot, and timestamp help clients verify: - Which keys are trusted - Which integrations and versions are authorized - Whether metadata is current - Whether files have been modified - TUF protects against common repository attacks, including: - Key compromise through key rotation and delegation - Rollback attacks using version information - Freeze attacks using metadata expiration - Mix-and-match attacks involving inconsistent metadata - The Datadog Agent can therefore reject integrations that fail authenticity, integrity, freshness, or version checks. ## in-toto for build provenance - in-toto complements TUF by describing and verifying the steps used to produce an integration. - Build metadata can show that required steps—such as source retrieval, dependency installation, testing, packaging, and signing—were performed by authorized parties. - Attestations connect each build step to its inputs and outputs, making unauthorized substitutions easier to detect. - This ensures that a validly signed package was not merely signed, but was produced through the expected supply-chain process. ## Combining distribution security and provenance - TUF answers whether an integration is authorized and safe to download. - in-toto answers whether it was built according to the approved process. - The combined design creates layered verification: - TUF validates repository metadata and package integrity. - in-toto validates build identity, steps, and provenance. - The Agent enforces the resulting trust decisions before installation or update. - This approach limits the impact of a compromise in any single part of the publication pipeline. Datadog’s approach illustrates that software supply-chain security requires more than package signatures. Using TUF for resilient distribution metadata and in-toto for build provenance provides a stronger, defense-in-depth model for safely delivering Agent integrations.

datadog-agentsoftware-supply-chain-securitytufin-toto+1

datadogApr 16, 2018

Cgo and Python (opens in new tab)

Embedding Python in Go lets applications gradually migrate from Python, reuse existing libraries, and load scripts dynamically without recompiling. Datadog uses this approach in its Go-based Agent so checks can remain in Python while the core application moves to Go. The key is combining cgo with a Go-friendly wrapper around CPython’s C API. ## Why Embed Python in Go? - Supports incremental migration from an existing Python codebase. - Reuses mature Python libraries without reimplementing them in Go. - Enables runtime loading and execution of custom or updated Python scripts. - This dynamic extensibility is especially important for Datadog checks. ## Introducing cgo - CPython exposes a C API, while Go requires a Foreign Function Interface to call C code. - cgo provides that integration while preserving the normal `go build` workflow. - A C preamble placed immediately before `import "C"` can include headers and C code. - The pseudo-package `C` exposes C constants, functions, and types to Go. - `go build -x` reveals how cgo generates intermediate C and Go files, compiles them, and links the final binary. ## Initializing the CPython Interpreter - A Go program must initialize Python with `Py_Initialize()` before executing Python code. - It should shut down the interpreter with `Py_Finalize()` when finished. - `Py_GetVersion()` demonstrates retrieving Python information through the C API. - `#cgo` directives can use `pkg-config` to locate Python development headers and libraries, such as `python-2.7`. - The examples use Python 2, but the same approach largely applies to Python 3. ## Using a Go Wrapper - Direct cgo interaction is mostly boilerplate, so Datadog uses the `go-python` library. - The wrapper exposes operations such as: - `python.Initialize()` - `python.PyRun_SimpleString(...)` - `python.Finalize()` - This hides cgo details and makes embedded Python code look more idiomatic from Go. ## Importing and Calling Python Code - A Python module can be imported with `PyImport_ImportModule`. - Go retrieves a function using `GetAttrString`. - The function is invoked through the Python API, passing empty tuple and dictionary objects even when it accepts no arguments. - The Go code must check for failures when importing modules or locating functions. - A simple `foo.py` module containing a `hello()` function can therefore be loaded and executed from disk. Embedding CPython through cgo provides a practical bridge between Go and Python. A wrapper such as `go-python` makes the integration easier to maintain, while allowing applications like the Datadog Agent to combine a Go core with dynamically executed Python components.

datadog-agentdatabase-designpythongo+4