datadog

110 posts

datadog

Steganography at scale: Embedding share URLs in Datadog widget screenshots (opens in new tab)

Datadog is building a way for screenshots to preserve the context normally available through share links. The system invisibly embeds a compact widget identifier into screenshot pixels, while storing the full widget definition in Redis. This approach aims to combine screenshots’ convenience with share links’ ability to restore queries, time ranges, and dashboard state, at massive scale. ## From Share Links to Context-Aware Screenshots - Copying a Datadog widget creates a backend record and places a unique share URL on the clipboard. - Pasting the URL into a dashboard or notebook restores the widget. - Slack and Teams integrations can render a live preview linking back to Graph Explorer. - Screenshots remain popular because they are quick, intuitive, and visually consistent. - However, screenshots normally lose: - The time range - Underlying queries - Visualization type - Dashboard state - Template variables and other configuration ## Storing a Compact Snapshot Reference - A complete widget definition can be about 2 kB, including queries, display settings, legends, titles, time ranges, dimensions, and deep links. - Encoding all of that directly into an image would be impractical. - Instead, Datadog stores the full definition in Redis and embeds only a randomly generated snapshot ID in the screenshot. - Snapshot records are retained for one hour because screenshots are typically pasted within seconds or minutes. - The frontend generates IDs optimistically so watermarks appear immediately, before the backend cache operation completes. - Redis keys include the organization ID, preventing collisions between different customers. - An 8-byte identifier provides roughly 2⁶⁴ possible values; under the stated traffic assumptions, the estimated collision risk is about one in 37 million. ## Encoding Data in Widget Borders - Every dashboard widget has a uniform, 1-pixel border, making it a reliable place to add metadata without visualization-specific code. - An initial design used individual pixels with two colors to represent bits, but encoding 64 bits would require at least 64 pixels and could become visible. - The chosen approach stores multiple bits in each pixel’s RGB channels. - Each color channel is offset from the base border color by up to seven values, allowing up to nine bits per pixel. - Two sentinel pixels, encoded with maximum RGB offsets, mark the beginning and end of the watermark. - Because the encoded pixels remain close to the border’s original color, the watermark is intended to remain imperceptible while remaining recoverable by software. ## Scaling and Collision Considerations - Datadog renders more than one billion widgets per day, with peaks of roughly 35,000 widgets per second. - The watermark design therefore has to minimize payload size while supporting high throughput. - Shorter identifiers are easier to hide but increase collision risk, requiring organization-scoped keys and carefully chosen identifier sizes. Datadog’s design uses screenshots as lightweight carriers for references rather than embedding complete widget data. By combining subtle border-based pixel encoding with short-lived Redis snapshots, screenshots can potentially regain the contextual and interactive benefits of share links without changing their appearance.

datadog

Steganography at scale: Embedding share URLs in Datadog widget screenshots (opens in new tab)

Datadog developed invisible pixel-level watermarks so screenshots can retain the context normally preserved by share links. The system embeds a compact widget snapshot ID into a widget’s border, while the full metadata remains in a Redis cache. This approach preserves screenshots’ convenience while enabling recovery of queries, time ranges, settings, and deep links at Datadog’s scale. ## Share Links Versus Screenshots - Copying a Datadog widget creates a backend record and places a unique share URL in the clipboard. - Pasting the URL into a dashboard or notebook restores the widget. - Slack and Teams integrations can render a live preview and link to Graph Explorer. - Screenshots are easier to use and provide a consistent visual snapshot, but normally lose: - Time range - Underlying queries - Visualization type - Dashboard state - Configuration and context ## Encoding Only a Snapshot ID - A complete widget definition averages about 2 kB and may include queries, display settings, legends, time-frame overrides, template variables, dimensions, and deep links. - Rather than embedding all of that data in the image, Datadog stores it in Redis and embeds only a randomly generated key. - The frontend generates the snapshot ID optimistically before the cache write completes, allowing watermarking without waiting for a backend response. - Records are retained for one hour because screenshots are usually shared within seconds or minutes. - At more than 1 billion widget renders per day, IDs must be compact while avoiding cross-customer collisions. - Datadog prefixes the cache key with the organization ID. An 8-byte ID provides roughly 2⁶⁴ possible values, producing an estimated collision probability of about 1 in 37 million under the stated usage assumptions. ## Watermarking the Widget Border - Every dashboard widget has a consistent 1-pixel border, making it a reliable location for encoding data regardless of visualization type. - An initial design represented each bit with a separate colored pixel, but 64 pixels were needed for 8 bytes and could become visible. - The final design stores data in RGB color adjustments: - Each pixel encodes up to 9 bits by offsetting the red, green, and blue channels. - The base color is calculated by subtracting 3 from each channel. - Channel offsets of up to 7 represent the encoded values. - Two sentinel pixels, using a `+7/+7/+7` offset, mark the beginning and end of the watermark. - Eight pixels between the sentinels encode one byte each: - 3 bits in red - 3 bits in green - 2 highest bits in blue ## Design Constraints - The watermark must remain nearly invisible and avoid adding interface elements. - It must work across different widget sizes, color profiles, display densities, and copy-paste workflows. - The border-based method avoids visualization-specific implementations while keeping the encoded region short. Datadog’s approach combines cached metadata with subtle RGB-level encoding, allowing screenshots to function like context-preserving share links without changing their appearance or the user’s workflow.

datadog

How we built a real-world evaluation platform for autonomous SRE agents at scale (opens in new tab)

Bits AI SRE improved in isolated scenarios but lacked a way to detect regressions across the broader range of production incidents. The team found that tool-level tests and live replays could not capture failures caused by multi-step reasoning or changing telemetry. They built a replayable evaluation platform combining realistic investigation labels, scalable orchestration, and longitudinal performance tracking. ## Subtle Regressions from Well-Intentioned Features - Adding a monitor’s service name to the agent’s initial context improved some internal investigations. - Across broader scenarios, it introduced irrelevant signals that confused the agent and degraded unrelated investigations. - Because there was no representative evaluation set, the team could not measure the change’s wider impact before internal misses exposed it. - The incident demonstrated the need to evaluate every change across diverse investigation types. ## Limits of Tool Tests and Live Replay - Testing tools individually failed to capture errors caused by incorrect interactions between valid tool outputs. - Live investigation replay was difficult to scale because: - Results were not consistently aggregated. - Production environments changed. - Telemetry expired, making investigations unreplayable. - Standard evaluation frameworks assumed clean inputs and static datasets, unlike agents operating over production telemetry. - The team needed controlled, offline replay of realistic end-to-end investigations. ## Evaluation Labels and World Snapshots - Each label represents one production-style investigation. - It contains: - **Ground truth:** the issue’s actual root cause. - **World snapshot:** the queries and signals available when the issue occurred. - The agent is never shown the root cause directly; it must reason from the preserved signals. - Labels must cover varied technologies and failure modes, including: - Kubernetes pod failures - Kafka lag - Bad-code deployments - Complex multi-service business failures - A narrow or overly clean dataset would make performance appear better than it really is. ## Orchestrating Evaluations at Scale - The platform runs Bits against labels, scores the outcomes, and tracks quality over time. - It supports comparisons across: - Investigation categories - Model variants - Configuration versions - Evaluation runs - The architecture consists of a shared label set, an orchestration layer, and reporting infrastructure. - This allows teams to determine whether improvements in one domain, such as Kafka, regress another, such as Kubernetes. ## Scaling Label Creation - The team initially created labels manually from Datadog alerts. - Manual labeling provided early coverage but consumed engineering time and remained far from representative. - They embedded label generation into Bits itself: - Customer feedback and investigation data are used to derive root causes. - Relevant queries are preserved as the world snapshot. - Each user interaction becomes a potential evaluation case. - This increased label creation rates by an order of magnitude and allowed coverage to grow with product usage. ## Agent-Assisted Validation - Early labels required extensive human review, especially when feedback was ambiguous or reconstructed signals were uncertain. - As ingestion grew, manual review became a bottleneck. - Bits was then used to assist with validation by aggregating related signals, identifying relationships, and resolving ambiguous feedback before human review. ## Practical Conclusion Reliable agent improvement requires more than testing individual tools or replaying live incidents. A representative, production-derived label set combined with reproducible end-to-end evaluations makes regressions visible and enables safer iteration.

datadog

When upserts don't update but still write: Debugging Postgres performance at scale (opens in new tab)

The provided content does not include the tech blog post itself. It consists primarily of Datadog’s website navigation and a promotional link about its Gartner recognition, so there is not enough article content to summarize reliably. ## Available Information - Datadog was named a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms. - The page navigation lists Datadog products across: - Infrastructure and application monitoring - Database and log management - Security - Digital experience monitoring - Software delivery - Incident and service management - AI-powered observability tools - The URL path references `debugging-postgres-performance`, suggesting the intended article may concern PostgreSQL performance debugging, but the article text is not included. Please provide the blog post’s body or a working text extraction for a substantive summary.

datadog

When an AI agent came knocking: Catching malicious contributions in Datadog’s open source repos (opens in new tab)

Datadog announces that Gartner has named it a Leader in the 2026 Magic Quadrant for Observability Platforms. The surrounding product catalog presents Datadog as a broad platform spanning infrastructure, applications, data, logs, security, digital experience, software delivery, service management, and AI. However, the provided content does not include the blog post’s detailed analysis or Gartner’s specific evaluation criteria. ## Gartner Recognition - Datadog highlights its position as a **Leader** in the **Gartner Magic Quadrant for Observability Platforms 2026**. - The announcement links to a Gartner resource but provides no further details about the ranking, strengths, or limitations. ## Broad Observability Platform - **Infrastructure:** Infrastructure and container monitoring, metrics, Kubernetes autoscaling, network monitoring, serverless, cloud cost, storage, GPU monitoring, and Cloudcraft. - **Applications and data:** Application Performance Monitoring, service monitoring, profiling, dynamic instrumentation, database monitoring, data streams, data quality, and jobs monitoring. - **Logs and security:** Log management, sensitive-data scanning, audit trails, observability pipelines, cloud security, SIEM, code security, vulnerability management, and workload protection. - **Digital experience:** Browser and mobile RUM, product analytics, session replay, synthetic monitoring, mobile testing, error tracking, and experiments. - **Software delivery and service management:** CI visibility, test optimization, continuous testing, feature flags, code coverage, event management, SLOs, incident response, workflow automation, and service catalogs. - **AI capabilities:** Agent observability, GPU monitoring, AI integrations, AI agents, investigation tools, security analysis, MCP Server, and agent-building features. Datadog’s positioning is based on consolidating telemetry, security, developer, operations, and AI capabilities into one observability platform. To assess the Gartner recognition fully, readers would need the linked report or the complete article, which is not included here.

datadog

Designing MCP tools for agents: Lessons from building Datadog's MCP server (opens in new tab)

Datadog’s initial MCP server simply exposed existing APIs, but real-world agent use revealed major problems with context limits, inaccurate trend analysis, and tool overload. The team redesigned its tools around token efficiency, query-based analysis, and a smaller, more deliberate tool surface. These changes improved both answer quality and cost, though emerging agent features may eventually reduce the need for some optimizations. ## Context Efficiency Matters - Observability results can be extremely large: a log record may range from roughly 100 characters to 1 MB. - CSV or TSV is more token-efficient than JSON for tabular data, often using about half as many tokens per record. - YAML can reduce token usage for nested data by around 20% compared with JSON. - Removing rarely used fields from default responses, while allowing agents to request them when needed, further reduces output size. - Combined formatting and field-trimming improvements allowed some tools to return approximately five times more records within the same token budget. - Pagination by record count is unreliable when records vary greatly in size. Datadog instead paginates by token budget and returns a cursor when the limit is reached. - Tools such as Cursor and Claude Code increasingly write long results to disk, which could make response-format efficiency less important in the future. ## Let Agents Query Data - Retrieval-only tools forced agents to infer trends from incomplete samples, such as guessing which services generated the most errors. - Agents sometimes repeatedly fetched logs to compensate, wasting tokens and producing unreliable answers. - SQL lets agents aggregate and filter data directly: ```sql SELECT service, COUNT(*) AS error_count FROM logs WHERE status = 'error' GROUP BY service ORDER BY error_count DESC LIMIT 10 ``` - Agents can select only necessary fields, limit row counts, and calculate aggregates without loading raw data. - SQL improved correctness and reduced costs; some evaluation scenarios became about 40% cheaper. - Supporting SQL at Datadog’s scale required significant infrastructure work because traditional relational databases were insufficient. ## Tools Are Not Free - Exposing every API endpoint as a separate tool increases tool-selection errors and consumes context through tool descriptions. - Flexible tools can support multiple related workflows through carefully designed schemas, reducing the total tool count. - Toolsets provide a core collection by default while allowing users to opt into specialized capabilities, though users must anticipate their needs. - Layered tools can first explain how to accomplish a task and then execute it, keeping specialized functionality out of the initial context. - Layering introduces additional tool calls and therefore increases latency. - Improving agent context management, including tool search and dynamically loaded skills, may reduce the need for aggressive tool minimization over time. The practical recommendation is to design MCP tools for how agents actually reason: minimize and control output size, provide query and aggregation capabilities instead of raw retrieval alone, and expose a focused set of flexible tools rather than mirroring every API endpoint.

datadog

Designing MCP tools for agents: Lessons from building Datadog's MCP server | Datadog (opens in new tab)

Datadog is presented as a Leader in the 2026 Gartner Magic Quadrant for Observability Platforms. The provided content, however, consists almost entirely of Datadog’s website navigation rather than the blog post itself, so it does not include Gartner’s evaluation criteria, Datadog’s strengths, or any supporting analysis. ## Gartner Recognition - The page headline announces Datadog’s “Leader” position in the Gartner Magic Quadrant for Observability Platforms. - A link is provided to a Gartner-related resource page. - No ranking details, competitor comparisons, or Gartner commentary are included in the supplied text. ## Datadog’s Product Coverage The navigation indicates that Datadog offers a broad observability and operations platform spanning: - **Infrastructure:** infrastructure, container, network, serverless, GPU, storage, and cloud-cost monitoring. - **Applications:** APM, service monitoring, profiling, dynamic instrumentation, and agent observability. - **Data and logs:** database, data-stream, data-quality, job, log, and sensitive-data monitoring. - **Digital experience:** browser and mobile RUM, session replay, synthetic monitoring, product analytics, and error tracking. - **Security:** code, cloud, workload, vulnerability, compliance, SIEM, and application/API protection. - **Software delivery and service management:** CI visibility, testing, developer portals, incident response, SLOs, workflows, and case management. - **AI:** agent observability, GPU monitoring, AI integrations, Bits AI agents, and an MCP server. ## Limitations of the Provided Content - The actual article body is absent. - The text does not explain why Gartner recognized Datadog as a Leader. - It provides no technical findings, customer examples, methodology, or conclusions beyond the headline. The supplied excerpt supports only the conclusion that Datadog announced Gartner recognition and positions itself as a comprehensive observability platform. A substantive summary would require the full article text.

datadog

How we reduced the size of our Agent Go binaries by up to 77% | Datadog (opens in new tab)

The supplied text does not include the tech blog post itself. It contains Datadog navigation links and a promotional banner announcing its recognition as a Leader in the 2026 Gartner Magic Quadrant for Observability Platforms, but no article body or technical sections. ## Available content - Datadog promotes observability products covering: - Infrastructure and Kubernetes monitoring - Application performance monitoring - Logs and database monitoring - Security - Digital experience monitoring - Software delivery and CI visibility - Service management - AI-powered investigation and monitoring - The page links to an engineering article at: - `/blog/engineering/agent-go-binaries/` - No technical explanation, examples, conclusions, or section content from that article is included. Please provide the blog post’s full text or relevant excerpt for a substantive summary.

datadog

Hardening eBPF for runtime security: Lessons from Datadog Workload Protection | Datadog (opens in new tab)

The provided text does not include the blog post’s article body. It contains Datadog’s navigation menu and a link to an engineering post titled around “eBPF workload protection lessons,” so there is not enough source material to accurately summarize its technical arguments or conclusions. ## Available information - The page is hosted by Datadog’s engineering blog. - The linked topic concerns workload protection built with eBPF. - Datadog’s broader product areas include infrastructure monitoring, application performance monitoring, security, logs, and AI. - The excerpt itself does not describe: - The eBPF implementation - Design challenges or trade-offs - Performance considerations - Security detection methods - Lessons learned or recommendations Please provide the article text or a fuller extract for a substantive summary.

datadog

Scaling real-time file monitoring with eBPF: How we filtered billions of kernel events per minute | Datadog (opens in new tab)

Datadog was named a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms. The announcement positions Datadog as a provider of broad, integrated monitoring across infrastructure, applications, logs, security, digital experiences, software delivery, and AI. The supplied content does not include Gartner’s evaluation details or the blog post’s supporting arguments. ## Recognition and Positioning - Datadog highlights its designation as a Leader in the Gartner Magic Quadrant for Observability Platforms. - The announcement emphasizes Datadog’s unified observability platform rather than a single monitoring product. ## Breadth of the Platform - **Infrastructure:** infrastructure, container, network, serverless, GPU, storage, and cloud-cost monitoring. - **Applications and data:** APM, database monitoring, continuous profiling, data-stream monitoring, and job monitoring. - **Logs and observability operations:** log management, sensitive-data scanning, audit trails, and observability pipelines. - **Security:** cloud security, SIEM, workload protection, code security, vulnerability management, and application/API protection. - **Digital experience:** browser and mobile RUM, session replay, synthetic monitoring, product analytics, and error tracking. - **Software delivery and service management:** CI visibility, testing, feature flags, incident response, SLOs, workflow automation, and case management. - **AI capabilities:** agent observability, GPU monitoring, AI integrations, Bits AI agents, and investigation tools. Overall, the available material presents Datadog’s Gartner Leader recognition and extensive product coverage, but it does not provide enough article text to summarize the specific reasoning behind the designation.

datadog

Replication redefined: How we built a low-latency, multi-tenant data replication platform | Datadog (opens in new tab)

The supplied content does not include the blog post’s article text. It contains Datadog’s navigation menu and a promotional link to its Gartner recognition, while the URL suggests the post concerns CDC replication and search. ## Available Information - Datadog was named a **Leader in the 2026 Gartner Magic Quadrant for Observability Platforms**. - The page promotes Datadog products covering: - Infrastructure and application monitoring - Logs, databases, and data observability - Security and digital experience - Software delivery and service management - AI-powered observability - The referenced article URL is `engineering/cdc-replication-search`, indicating a likely focus on **change data capture (CDC), data replication, and search systems**. ## Missing Article Details - No sections, technical explanations, architecture diagrams, implementation details, or conclusions from the blog post are present in the supplied text. - A reliable summary of the CDC replication approach cannot be produced without the article body. Please provide the full post text or its main sections for a complete summary.

datadog

Detecting malicious pull requests at scale with LLMs | Datadog (opens in new tab)

Malicious pull requests can turn routine code review and CI workflows into supply-chain attack vectors. The post explains how attackers abuse automated builds—especially when workflows expose repository secrets or elevated GitHub permissions—and recommends treating all pull-request code as untrusted. Strong isolation, least privilege, careful workflow design, and monitoring are essential to prevent credential theft and unauthorized access. ## How Malicious Pull Requests Work - Attackers submit seemingly harmless changes that alter: - GitHub Actions workflows - Build or test scripts - Dependency configuration - Developer tooling - The malicious code executes automatically when CI runs the pull request. - Its goal may be to: - Exfiltrate repository or cloud credentials - Modify artifacts - Access internal systems - Establish persistence in the development pipeline ## Why CI Workflows Are Vulnerable - Pull-request jobs often execute attacker-controlled code through tests, package installation, or build commands. - Using privileged workflow events such as `pull_request_target` can expose secrets while checking out untrusted contributor code. - Broad `GITHUB_TOKEN` permissions increase the impact of a compromised job. - Secrets may leak through logs, environment variables, artifacts, or outbound network requests. ## Defensive Engineering Practices - Treat code from forks and external contributors as untrusted. - Avoid making secrets available to pull-request jobs. - Use minimal `GITHUB_TOKEN` permissions and separate privileged workflows from validation workflows. - Pin third-party GitHub Actions and dependencies to trusted commits or versions. - Require explicit approval before running workflows from untrusted contributors. - Isolate CI jobs with ephemeral runners, restricted network access, and limited filesystem permissions. - Review changes to workflow files with heightened scrutiny. ## Detection and Response - Monitor workflow behavior for unexpected network connections, credential access, or modified build outputs. - Audit repository and CI permissions regularly. - Use short-lived credentials and OIDC-based cloud access instead of long-lived static secrets. - Preserve workflow logs and artifacts to support investigation. - Revoke credentials immediately if a pull request or CI job is suspected of compromise. The practical recommendation is to design CI as though every pull request could be hostile: validate untrusted code in a restricted environment, keep secrets and write permissions out of those jobs, and require deliberate promotion into trusted workflows.

datadog

From hand-tuned Go to self-optimizing code: Building BitsEvolve | Datadog (opens in new tab)

The provided content does not include the blog post itself. It consists primarily of Datadog’s navigation menu and a promotional link announcing its 2026 Gartner Magic Quadrant recognition. As a result, there is not enough article content to produce a reliable technical summary. ### Available Information - Datadog is promoted as a “Leader” in the Gartner Magic Quadrant for Observability Platforms. - The page links to Datadog products covering: - Infrastructure and application monitoring - Logs, databases, and data observability - Security - Digital experience monitoring - Software delivery - Incident and service management - AI and automation - The referenced blog URL appears to be titled **“Self-Optimizing System,”** but its article text is not included. Please provide the blog post’s main content or a complete page extract for an accurate summary.

datadog

Scaling down to speed up: How we improved efficiency of live process metrics by 100x (opens in new tab)

Datadog redesigned its real-time Processes and Containers pipeline to avoid collecting high-frequency metrics that users never see. By limiting 2-second collection to hosts actively viewed and using standard 10-second data for sorting, the company reduced real-time traffic by over 100x, cut infrastructure costs by 98%, and lowered Agent resource usage. The approach also improved scalability without sacrificing the live investigation experience. ## Original Real-Time Collection Model - Datadog Agents normally collect process and container metrics every 10 seconds. - When a user opened a live Processes or Containers view, all hosts in that tenant switched to 2-second collection. - This supported near-real-time monitoring similar to `htop`, but across distributed infrastructure. - As tenants grew, the pipeline had to process millions of processes per second, even though users typically viewed only around 50 processes or containers. - Live sorting required keeping all tenant data in memory on a single server, limiting horizontal scaling and forcing vertical scaling. ## Refocusing on User-Visible Data - Most collected metrics were never displayed to users. - Datadog determined that real-time collection only needed to be enabled for hosts running the processes or containers currently in view—up to roughly 50 hosts per user. - Internal telemetry suggested this could reduce traffic by more than 100x. - This required tracking active host subscriptions and updating them as users navigated the product. - Because sorting occurred every 10 seconds, it did not need 2-second data. Datadog switched live views to use the existing 10-second metrics, aligning live and historical sorting logic. ## Host Subscription Filtering - A proof of concept added host subscriptions to the live data servers. - Servers filtered Kafka payloads and discarded data for hosts without active subscriptions. - This immediately reduced: - Memory usage by 85% - CPU usage by 33% - The improvement came from storing fewer live metrics and processing fewer incoming payloads. - The prototype confirmed that filtering preserved product behavior while simplifying sorting. ## Moving Filtering Earlier in the Pipeline - Late filtering improved live data servers but still left unnecessary work for the rest of the system and customer-side Datadog Agents. - Datadog therefore planned to propagate subscription state to the intake service. - Live data servers publish users’ active host sets over Kafka once per second. - The intake service consumes this information and decides which hosts should activate 2-second process and container metric collection. - This allows real-time collection to be restricted to hosts users are actively investigating while maintaining responsive live views. Datadog’s redesign demonstrates that real-time systems scale more effectively when they prioritize data users can actually see. Filtering at intake, limiting high-frequency collection to subscribed hosts, and reusing standard-resolution data for sorting provide a simpler and more economical architecture without eliminating live functionality.

datadog

Scaling down to speed up: How we improved efficiency of live process metrics by 100x | Datadog (opens in new tab)

The provided content does not include the blog post itself. It contains Datadog’s navigation menu and a promotional link announcing its recognition as a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms, so there is insufficient technical material to summarize the article. ### Content Included - A link to Datadog’s Gartner announcement. - Navigation categories covering: - Infrastructure and application monitoring - Logs, databases, and data observability - Security - Digital experience monitoring - Software delivery - Service management - AI capabilities - The URL suggests the intended article may concern scaling process or pipeline efficiency, but its body is not present. Please provide the full blog post text for a substantive summary.