In Toto | Techlist.io

datadogJun 3, 20193 min readCurated summary

Secure publication of Datadog Agent integrations with TUF and in-toto

Datadog built a compromise-resilient CI/CD system to publish Agent integrations independently of full Agent releases. It combines in-toto for end-to-end supply-chain verification with TUF for secure key and metadata distribution. Together, these technologies ensure that users install only integrations derived from developer-approved source code, even if parts of the infrastructure are compromised. ## Independent Integration Publishing - Agent integrations were traditionally bundled into full Agent releases. - This delayed important integration updates and prevented users from trying new integrations immediately. - Datadog wanted automation to build and publish integrations on demand without fully trusting the automation itself. ## End-to-End Verification with in-toto - TLS and GPG package signatures help prevent man-in-the-middle attacks but do not protect against compromised build or publishing infrastructure. - in-toto defines the software supply chain as a fixed sequence of signed steps. - Each step records the inputs it received and the outputs it produced, allowing the Agent to verify that only authorized parties performed the required work. - The integration pipeline includes: - Developers signing Python and YAML source files. - CI/CD packaging the source into Python wheels without modifying existing wheels. - A signing step applying TUF signatures to the wheels. - The Datadog Agent verifying that the downloaded wheel matches the developer-signed source. ## Secure Distribution with TUF - in-toto does not itself provide a secure way to distribute, revoke, or replace verification keys. - TUF supplies signed, compromise-resilient metadata for: - The root of trust for wheels and supply-chain metadata. - The in-toto-defined workflow. - Public keys used to verify the workflow. - TUF protects against tampering, rollback attacks, and indefinite replay of outdated metadata. - Offline trust bootstrapping and protected developer keys are essential to the overall security model. ## Hardware-Protected Developer Signing - Developers use Yubikeys to generate and store GPG signing keys. - Private keys cannot be exported from the device, assuming correct firmware. - Signing requires both a secret PIN and physical interaction with the Yubikey. - A command-line tool integrates in-toto and GPG, preserving a convenient developer workflow while reducing key-compromise risk. ## Transparent Verification for Users - The Datadog Agent automatically invokes TUF and in-toto when downloading or updating integrations. - Users need no workflow changes under normal conditions. - If metadata, signatures, or supply-chain steps fail verification, installation is blocked and the Agent reports the failure. Datadog’s approach demonstrates that secure automated publishing requires layered controls: in-toto verifies how software was produced, while TUF securely manages the trust and distribution mechanisms needed to validate it.

Read original

In Toto Database Design Datadog CI/CD+4

datadogJun 3, 20193 min readCurated summary

Secure publication of Datadog Agent integrations with TUF and in-toto | Datadog

Datadog describes how it secures the publication and distribution of Datadog Agent integrations using The Update Framework (TUF) and in-toto. TUF protects clients from tampered, outdated, or incorrectly signed packages, while in-toto provides verifiable evidence about how each integration was built. Together, the systems create a chain of trust from source and build processes to the integration installed by an Agent. ## Why Agent integrations need stronger supply-chain security - Integrations are distributed software components that run inside the Datadog Agent. - A compromise of the source repository, build infrastructure, signing credentials, or distribution system could lead to malicious code reaching customers. - Authenticating only the final package is insufficient if attackers can: - Replace repository metadata - Replay an older vulnerable release - Roll back clients to compromised versions - Exploit a stolen signing key - Publish artifacts that were not produced by the approved build process ## TUF for secure package distribution - TUF separates repository responsibilities across cryptographic roles rather than relying on one signing key. - Metadata roles such as root, targets, snapshot, and timestamp help clients verify: - Which keys are trusted - Which integrations and versions are authorized - Whether metadata is current - Whether files have been modified - TUF protects against common repository attacks, including: - Key compromise through key rotation and delegation - Rollback attacks using version information - Freeze attacks using metadata expiration - Mix-and-match attacks involving inconsistent metadata - The Datadog Agent can therefore reject integrations that fail authenticity, integrity, freshness, or version checks. ## in-toto for build provenance - in-toto complements TUF by describing and verifying the steps used to produce an integration. - Build metadata can show that required steps—such as source retrieval, dependency installation, testing, packaging, and signing—were performed by authorized parties. - Attestations connect each build step to its inputs and outputs, making unauthorized substitutions easier to detect. - This ensures that a validly signed package was not merely signed, but was produced through the expected supply-chain process. ## Combining distribution security and provenance - TUF answers whether an integration is authorized and safe to download. - in-toto answers whether it was built according to the approved process. - The combined design creates layered verification: - TUF validates repository metadata and package integrity. - in-toto validates build identity, steps, and provenance. - The Agent enforces the resulting trust decisions before installation or update. - This approach limits the impact of a compromise in any single part of the publication pipeline. Datadog’s approach illustrates that software supply-chain security requires more than package signatures. Using TUF for resilient distribution metadata and in-toto for build provenance provides a stronger, defense-in-depth model for safely delivering Agent integrations.

Read original

In Toto Software Supply Chain Security Datadog Agent Tuf+1