datadogAug 11, 2026

How we improved APM Java startup by encoding a prefix trie as a JVM constant (opens in new tab)

Startup performance is critical for users, developers, and cloud costs, but Java APM instrumentation must balance observability against the overhead of transforming classes. Datadog reduced class-matching overhead by 30% over four years by optimizing the first filtering stage: matching class-name prefixes. Its main innovation was encoding a prefix trie as a single JVM string constant, avoiding the startup cost of constructing a conventional trie. ## Java Instrumentation and Class Matching - Java APM uses the Java Instrumentation API to intercept and transform classes as they load. - Instrumentation adds method advice that records method execution and propagates tracing context. - Instrumenting every method would be too expensive, so APM first identifies valuable classes. - Applications may load tens or hundreds of thousands of classes, making efficient filtering important. - Class-name and package-prefix checks are cheaper than structural or hierarchy-based checks because they avoid parsing class files. - Datadog therefore begins with a curated ignore list of class and package prefixes. ## Startup Constraints in `premain` - Agents register transformers in the JVM’s `premain` phase, before the application’s `main` method. - At this point: - Few classes have been loaded. - The JIT compiler is cold or unavailable, especially on Java 8. - Code runs interpreted and unoptimized. - Loading or calling certain JDK classes can have irreversible side effects. - For example, touching `java.util.logging` initializes `LogManager`, potentially preventing an application from configuring its own logging manager later. - These constraints make ordinary data loading, parsing, and object construction undesirable during startup. ## Replacing a Hand-Written Matcher with a Trie - Datadog’s earlier matcher used a complex nested code structure to represent prefixes. - Although flexible, it was difficult to maintain and required special optimizations for Java 8 startup. - A trie was a natural replacement because it shares common characters among prefixes and supports efficient lookup. - A conventional trie would require: - Locating and reading a resource. - Parsing its contents. - Constructing trie nodes. - Loading additional code or dependencies. - Those operations would impose unacceptable costs during `premain`. ## Encoding the Trie as a JVM Constant - Datadog created `ClassNameTrie`, which stores the entire prefix trie in a Java string constant. - The JVM loads the encoded data with a single `ldc` bytecode instruction. - This approach avoids resource I/O and runtime trie construction. - Embedding the data in the class also makes it resilient to repackaging. - The compact representation improves cache locality and reduces startup work. ## Compact Node Representation - Java strings contain 16-bit `char` values, allowing each character to encode one of 65,536 possible values. - Each trie node stores: - A character indicating the number of branches. - Sorted branch characters, enabling binary search. - One value character per branch. - Value characters encode different outcomes: - **Leaf:** returns a definitive result and ends the search. - **Bud:** records a possible result but permits further matching. - **Inline segment length:** indicates that additional prefix characters are stored directly. - Buds and leaves can include a **glob bit**, allowing a match to apply even when extra characters remain in the class name. - The encoding reserves the remaining value range for match results, with a maximum stored value of 8,191. The broader lesson is that startup-sensitive JVM code may benefit from moving computation into class-loading time and representing lookup structures in compact constants. For Java agents, precomputed, dependency-free data structures can deliver trie-like performance without the initialization and JIT costs of building them at runtime.

application-performance-monitoringjavajvmjit-compiler+2

datadogJul 22, 2026

Unbiased Java CPU profiling with JFR in JDK 25 (opens in new tab)

Java Flight Recorder (JFR) provides low-overhead, production-safe diagnostics, but its `ExecutionSample` event can produce biased CPU profiles because it samples JVM-observed runnable threads rather than strictly measuring CPU time. For CPU-bound workloads—particularly reactive applications—this may obscure the real hotspots. Modern profilers therefore combine JFR with JVMTI, `SIGPROF`, `AsyncGetCallTrace`, and JVM-internal techniques, while the Java ecosystem works toward a supported CPU-sampling mechanism. ## How Sampling Profilers Work - Continuous profilers repeatedly capture stack traces and aggregate them to reveal recurring behavior. - CPU profilers commonly sample at fixed intervals, such as every 20 milliseconds. - **CPU-time sampling** highlights code actively consuming processor cycles. - **Wall-clock sampling** reveals latency sources, including I/O waits, lock contention, and blocked threads. - Other profilers trigger on events such as allocations, garbage collection, thread parking, or lock contention. - Regardless of the trigger, profilers capture a stack, associate it with an event, and aggregate the results. ## Limitations of JFR’s `ExecutionSample` - JFR is integrated into the JVM and designed for low-overhead, always-on production use. - Its `ExecutionSample` event captures stacks from a rotating subset of runnable threads. - CPU-heavy threads tend to appear more often, but samples are not strictly proportional to actual CPU consumption. - This can lead to incomplete or biased results on CPU-saturated systems. - Reactive applications are a notable example: their scheduling behavior can cause thread CPU usage and hotspots to be underrepresented. ## CPU Sampling with `AsyncGetCallTrace` - JVMTI agents can use operating-system signals such as `SIGPROF` to sample threads according to CPU time. - The signal handler invokes HotSpot’s `AsyncGetCallTrace` to walk Java stacks asynchronously. - This approach avoids safepoint bias and can capture stacks during arbitrary execution states. - Tools such as `async-profiler` use this technique to produce profiles that more closely match actual CPU usage. - The drawback is that `AsyncGetCallTrace` is an unsupported internal JVM API. - Under heavy load, it can occasionally fault, requiring profilers to add extensive safeguards. - Datadog also uses **vmstructs walking**, which reads internal JVM metadata to recover stack and runtime information unavailable through standard APIs. ## The Safety–Accuracy Tradeoff - JFR offers stability, structured runtime telemetry, and low overhead. - `AsyncGetCallTrace` and vmstructs walking offer more accurate CPU sampling. - Relying on JVM internals creates maintenance and reliability risks because those interfaces are not officially stable. - Consequently, modern profilers combine JFR with unsupported sampling mechanisms rather than choosing only one approach. ## Toward a Supported CPU Profiling Event - Datadog, SAP, Amazon, and OpenJDK contributors recognized that this limitation affected the broader profiling ecosystem. - JFR was already the natural foundation for safe, continuous profiling. - The missing capability was a first-class CPU sampling event that could provide accurate CPU-based results without depending on unsupported JVM internals. - Datadog participated in OpenJDK discussions to explain why existing sampling was insufficient and to help improve the platform’s profiling foundation. Ultimately, accurate production CPU profiling requires both JFR’s safety and CPU-time-based sampling. A supported JFR CPU profiling event would remove the ecosystem’s dependence on fragile JVM internals while preserving the low-overhead behavior needed for continuous use.

javajvmcpu-profilingjava-flight-recorder+3

datadogJul 1, 2026

How we measure data completeness at scale (opens in new tab)

Datadog built a real-time data-completeness system to ensure that every customer’s telemetry is available for dashboards, alerts, queries, and AI-driven decisions. Because ingestion spans hundreds of distributed paths and customers may send delayed or retried data, global or watermark-based tracking is unreliable. The system instead tracks payloads segment by segment, using idempotent create and acknowledgment events to identify losses and calculate end-to-end completeness. ## Defining Completeness at Datadog’s Scale - Completeness means every ingested payload—metrics, logs, spans, or other telemetry—is ultimately available to customers. - The system must measure completeness: - Across hundreds of services and ingestion paths - For each individual customer - In real time - With enough detail to identify where degradation occurred - Customer traffic may take different routes because of partitioning, isolation, and traffic patterns. - Metrics and APM pipelines can each involve hundreds or tens of distinct paths, creating a large number of possible failure points. - The completeness system must remain independent of the services it monitors so it can provide trustworthy diagnostics during incidents. ## Tracking Completeness by Pipeline Segment - Datadog considered watermark-based tracking, but delayed customer data, replayed traffic, and pipeline loops made predictable watermarks impractical. - Pipelines are divided into segments representing steps within or between services. - For example, intake-in to intake-out is one segment. - Intake-out to processing-in is another. - Each segment is measured independently, allowing engineers to locate degradation within a service or between services. - Segment-level tracking also adapts to pipelines whose branches appear or disappear over time. ## Counting Creates and Acknowledgments - When a payload enters a segment, the system records a create event. - When it exits, the system records an acknowledgment using the payload’s unique identifier. - Comparing creates with acknowledgments reveals whether payloads were lost in that segment. - Events are organized into time buckets based on when the payload first entered Datadog, using a Datadog-controlled timestamp rather than the customer’s clock. - Each identifier has a state per segment: - Created - Acknowledged - Acknowledged before the create event arrived - Duplicate create or acknowledgment events are ignored, making the system idempotent despite retries and event reordering. ## Calculating End-to-End Completeness - Segment completeness is the ratio of payloads exiting a segment to those entering it. - For sequential services, overall completeness is calculated by multiplying segment ratios. - Parallel branches require a different approach: - Treating branches as one pipeline would make completeness wait for the slowest branch. - Instead, Datadog uses a weighted average, giving each branch influence proportional to the volume it processes. - In the example, one branch reaches 94% completeness by multiplying 98% and 96% across two sequential services, while another branch reaches 100%. - Combining these branch measurements produces a more accurate view of currently available data without incorrectly marking all data incomplete because one branch is slower. ## Practical Conclusion Segment-level, identifier-based tracking gives Datadog a real-time and customer-specific view of data completeness. It both supports reliable end-to-end calculations and helps humans or automated systems quickly determine where ingestion problems are occurring.

observabilitydistributed-systemskafkaapm+3

datadogJun 23, 2026

How we migrated a live routing system using AI-assisted refactoring (opens in new tab)

Stream Router evolved from a small configuration file into a critical control-plane service routing Datadog’s massive metrics workload. Its original FoundationDB key-value model eventually hit transaction-size and performance limits because relational relationships were reconstructed in application code. Datadog redesigned the system around PostgreSQL and DuckDB, using AI-assisted, test-driven refactoring to accelerate the migration without disrupting production traffic. ## Stream Router’s Role in Datadog’s Metrics Pipeline - Datadog processes more than a hundred trillion events per day. - Stream Router determines which Kafka cluster, topic, partitions, and sharding strategy should handle each datapoint. - It serves both producers and queriers but does not process Kafka messages itself. - Routing decisions change frequently as infrastructure evolves, making correctness and historical tracking essential. ## From Configuration File to Control Plane - In 2016, routing was managed through a small configuration file distributed to services. - As the platform grew, the file expanded to thousands of lines and required manual edits and rollouts. - Stream Router replaced this workflow with: - A centralized gRPC service - API-managed routes - Automated, gradual rollouts - The write path used FoundationDB, while the read path served static RocksDB snapshots restored into memory. - This eventually became a bottleneck as routing tables and operational changes grew larger. ## Why the Key-Value Model Stopped Scaling - Routes reference streams and sharding strategies, while rules reference routes. - These relationships are inherently relational and require cross-entity validation. - The KV implementation loaded tens of thousands of records into application processes and reconstructed database-like relationships in code. - Some operations exceeded FoundationDB transaction-size limits. - Moving to PostgreSQL without changing the access patterns would not solve the issue; certain operations were estimated to require 45 minutes because of thousands of sequential database round trips. - The fundamental problem was the data model and application logic, not simply the choice of database. ## Designing the New Storage Architecture - The team redesigned the schema manually before using AI tools. - The relational model introduced explicit foreign keys between: - Streams - Sharding strategies - Routes - Rules - PostgreSQL was selected for the write path because it provided the required relational semantics and transaction model. - DuckDB was selected for the read path because: - It is embeddable and suitable for snapshot-based serving - It supports array columns - Its SQL dialect is closely compatible with PostgreSQL - Shared query logic could therefore work across both storage engines. ## AI-Assisted Refactoring - Claude and Cursor were used to accelerate a systematic, test-driven migration. - For each method, developers supplied: - The old implementation - The new schema - A failing test - AI generated an initial implementation, while tests determined whether it was correct. - The models assisted with method-level refactoring rather than autonomously designing the architecture. - Human expertise remained central to schema design, migration strategy, and evaluating system-level risks. ## Foundations for a Safe Migration - The migration benefited from infrastructure already present at Datadog. - Stream Router’s storage layer was isolated behind an internal `Controller` interface. - This modularity helped contain storage changes and enabled incremental refactoring. - Existing tests and clear boundaries provided confidence in generated implementations while production traffic continued. The central lesson is that AI was most effective as an accelerator inside a disciplined engineering process. A well-designed relational schema, modular storage abstraction, and failing tests provided the safety mechanisms; AI helped implement the resulting changes faster, but did not replace human architectural judgment.

database-designpostgresqlkafkagrpc+4

datadogJun 4, 2026

When failover isn’t safe: Building high-availability PostgreSQL on Kubernetes (opens in new tab)

Datadog’s gameday testing exposed a PostgreSQL failure mode in which network latency caused replication lag to grow until no standby could be safely promoted. Although the clusters remained writable, they could not fail over without risking data loss, forcing operators to wait for connectivity and replicas to recover. Datadog’s solution was to redesign failover candidates around synchronous replication coordinated by Patroni, balancing stronger durability with acceptable write latency. ## The Zonal Failure That Exposed the Weakness - A simulated availability-zone failure introduced network latency in a staging environment. - Several Kubernetes-based PostgreSQL clusters had primary nodes in the affected zone. - Communication between primaries and replicas degraded, causing: - Rapidly increasing replication lag - Stalled writes - Applications serving stale data - No replica being current enough for safe promotion - The clusters prioritized continued writes over durability, leaving them writable but unable to fail over safely. ## Baseline PostgreSQL Architecture - Each cluster uses a single-writer design: - One active leader handles writes. - Two standby nodes are reserved for failover and do not serve application traffic. - A separate read-replica pool handles read-only traffic and scales independently. - Read replicas are intentionally excluded from failover candidates. - Patroni manages replication, leader elections, and failover. - ZooKeeper acts as Patroni’s distributed configuration store, tracking: - The current leader lock - Cluster configuration - Member replication state and latest LSN - ZooKeeper’s ephemeral leader key ensures that only one node can become primary. - During partitions, Patroni favors safety by pausing or demoting nodes that cannot verify cluster state. ## Why Failover Was Not Safe - Patroni checks replication lag before promoting a standby using `maximum_lag_on_failover`. - During the gameday, all eligible standbys exceeded that threshold. - Patroni correctly rejected promotion because each candidate could have been missing committed transactions. - The cluster therefore had no safe writable primary, even though the original leader was impaired. - The failure was a consequence of asynchronous replication and network latency, not a failure in Patroni’s safety mechanisms. ## Asynchronous Versus Synchronous Replication - **Asynchronous replication**, used originally: - Lets the leader commit and respond without waiting for replicas. - Provides low write latency and high throughput. - Can lose transactions committed on the leader but not yet copied to a standby. - **Synchronous replication**: - Requires the leader to receive acknowledgment from at least one replica before confirming a transaction. - Reduces the chance that a failover candidate is significantly behind. - Provides stronger durability, but may increase write latency when replicas experience network or availability problems. ## The Redesigned Approach - Datadog reworked its PostgreSQL deployment so failover candidates use synchronous replication. - Patroni coordinates these replicas and continues to enforce safe leader election. - The design aims to make failover both automatic and safe while limiting performance impact. - Benchmarking and failure testing were used to evaluate the trade-off between durability and latency. Datadog’s experience demonstrates that asynchronous replication can leave a system operational but unable to fail over during network disruption. For clusters where data durability and automatic recovery are critical, synchronous replication for designated failover candidates offers a safer architecture, provided its latency and availability costs are measured carefully.

k8spostgresqlhigh-availabilitydatabase-replication+4

datadogJun 2, 2026

From single pull requests to full software packages: Detecting malicious code at scale (opens in new tab)

BewAIre evolved from a pull-request malware detector into a system for scanning dependency packages and upstream registries. Its core improvement is a two-stage pipeline: a cheap LLM filter handles routine changes, while a more capable agent investigates suspicious cases using external tools and repository context. This approach raised accuracy from 97.4% to 99.86%, eliminated false positives in a 690-diff sample, and reduced latency and cost through early exits. ## Expanding Beyond Pull Requests - Software supply-chain attacks increasingly compromise trusted dependencies such as axios, LiteLLM, and Mistral. - BewAIre initially focused on detecting malicious pull requests, identifying security testing, bug-bounty activity, and real attacks such as the Hackerbot campaign. - The team aimed to apply the same LLM-based detection to complete packages and package registries without sacrificing accuracy, latency, or predictable cost. ## Limits of Single-Pass LLM Evaluation - BewAIre began as a basic “LLM-as-judge” system that analyzed diffs through an inference API. - More capable reasoning models improved detection but increased costs. - Large diffs, especially dependency upgrades, challenged context-window limits. - Two changes addressed these limitations: - A filter-then-review escalation path. - Tool-enabled investigation allowing models to gather additional evidence. ## Two-Stage Filtering and Investigation - The filter phase: - Runs on every change using a fast, inexpensive model. - Uses straightforward prompts and diff chunking for large changes. - Produces a binary suspicious/benign decision. - Ends processing immediately when a change appears benign. - The investigation phase: - Runs only when the filter raises a concern. - Uses a stronger reasoning model in an agentic loop. - Can inspect commits, files, contributor histories, dependency metadata, and commit ranges through GitHub APIs. - Checks for reverted commits, typosquatting, suspicious contributor behavior, and dependency risks using sources such as osv.dev and Datadog SCA. ## Detecting Obfuscated Attacks - In the Hackerbot Claw example, the system identified a malicious filename containing shell command substitution. - A base64-encoded payload decoded to a `curl ... | bash` command that downloaded and executed remote code. - The investigation agent added useful context: - The contributor account was newly created, had no profile information, and had no followers. - The pull request had no reviews or approvals. - `${IFS}` obfuscation was used to evade security filters. - Combining code analysis with repository and author context made the final assessment more precise. ## Combining LLMs with Static Checks - The filter model could mistakenly treat Datadog-like typosquatting domains as legitimate without access to investigative tools. - BewAIre added preprocessing that extracts domains and compares them against a static list of known typosquatting variants. - This hybrid design improves reliability while avoiding the cost and nondeterminism of performing every check through a powerful LLM. ## Measured Results - Accuracy improved from 97.4% to 99.86% across 690 representative test diffs. - False positives fell from 17 to zero. - Most benign changes exit during the inexpensive filter stage. - Suspicious changes still receive deeper analysis, preserving broad coverage while controlling latency and cost. The practical recommendation is to combine inexpensive broad screening with selective, tool-driven investigation. Static security checks should complement LLM reasoning, especially for predictable threats such as domain typosquatting.

llmagentic-aidependency-scanningsoftware-supply-chain-security+4

datadogApr 21, 2026

Steganography at scale: Embedding share URLs in Datadog widget screenshots (opens in new tab)

Datadog is building a way for screenshots to preserve the context normally available through share links. The system invisibly embeds a compact widget identifier into screenshot pixels, while storing the full widget definition in Redis. This approach aims to combine screenshots’ convenience with share links’ ability to restore queries, time ranges, and dashboard state, at massive scale. ## From Share Links to Context-Aware Screenshots - Copying a Datadog widget creates a backend record and places a unique share URL on the clipboard. - Pasting the URL into a dashboard or notebook restores the widget. - Slack and Teams integrations can render a live preview linking back to Graph Explorer. - Screenshots remain popular because they are quick, intuitive, and visually consistent. - However, screenshots normally lose: - The time range - Underlying queries - Visualization type - Dashboard state - Template variables and other configuration ## Storing a Compact Snapshot Reference - A complete widget definition can be about 2 kB, including queries, display settings, legends, titles, time ranges, dimensions, and deep links. - Encoding all of that directly into an image would be impractical. - Instead, Datadog stores the full definition in Redis and embeds only a randomly generated snapshot ID in the screenshot. - Snapshot records are retained for one hour because screenshots are typically pasted within seconds or minutes. - The frontend generates IDs optimistically so watermarks appear immediately, before the backend cache operation completes. - Redis keys include the organization ID, preventing collisions between different customers. - An 8-byte identifier provides roughly 2⁶⁴ possible values; under the stated traffic assumptions, the estimated collision risk is about one in 37 million. ## Encoding Data in Widget Borders - Every dashboard widget has a uniform, 1-pixel border, making it a reliable place to add metadata without visualization-specific code. - An initial design used individual pixels with two colors to represent bits, but encoding 64 bits would require at least 64 pixels and could become visible. - The chosen approach stores multiple bits in each pixel’s RGB channels. - Each color channel is offset from the base border color by up to seven values, allowing up to nine bits per pixel. - Two sentinel pixels, encoded with maximum RGB offsets, mark the beginning and end of the watermark. - Because the encoded pixels remain close to the border’s original color, the watermark is intended to remain imperceptible while remaining recoverable by software. ## Scaling and Collision Considerations - Datadog renders more than one billion widgets per day, with peaks of roughly 35,000 widgets per second. - The watermark design therefore has to minimize payload size while supporting high throughput. - Shorter identifiers are easier to hide but increase collision risk, requiring organization-scoped keys and carefully chosen identifier sizes. Datadog’s design uses screenshots as lightweight carriers for references rather than embedding complete widget data. By combining subtle border-based pixel encoding with short-lived Redis snapshots, screenshots can potentially regain the contextual and interactive benefits of share links without changing their appearance.

datadogperformance-optimizationrediswatermarking+3

datadogApr 21, 2026

Steganography at scale: Embedding share URLs in Datadog widget screenshots (opens in new tab)

Datadog developed invisible pixel-level watermarks so screenshots can retain the context normally preserved by share links. The system embeds a compact widget snapshot ID into a widget’s border, while the full metadata remains in a Redis cache. This approach preserves screenshots’ convenience while enabling recovery of queries, time ranges, settings, and deep links at Datadog’s scale. ## Share Links Versus Screenshots - Copying a Datadog widget creates a backend record and places a unique share URL in the clipboard. - Pasting the URL into a dashboard or notebook restores the widget. - Slack and Teams integrations can render a live preview and link to Graph Explorer. - Screenshots are easier to use and provide a consistent visual snapshot, but normally lose: - Time range - Underlying queries - Visualization type - Dashboard state - Configuration and context ## Encoding Only a Snapshot ID - A complete widget definition averages about 2 kB and may include queries, display settings, legends, time-frame overrides, template variables, dimensions, and deep links. - Rather than embedding all of that data in the image, Datadog stores it in Redis and embeds only a randomly generated key. - The frontend generates the snapshot ID optimistically before the cache write completes, allowing watermarking without waiting for a backend response. - Records are retained for one hour because screenshots are usually shared within seconds or minutes. - At more than 1 billion widget renders per day, IDs must be compact while avoiding cross-customer collisions. - Datadog prefixes the cache key with the organization ID. An 8-byte ID provides roughly 2⁶⁴ possible values, producing an estimated collision probability of about 1 in 37 million under the stated usage assumptions. ## Watermarking the Widget Border - Every dashboard widget has a consistent 1-pixel border, making it a reliable location for encoding data regardless of visualization type. - An initial design represented each bit with a separate colored pixel, but 64 pixels were needed for 8 bytes and could become visible. - The final design stores data in RGB color adjustments: - Each pixel encodes up to 9 bits by offsetting the red, green, and blue channels. - The base color is calculated by subtracting 3 from each channel. - Channel offsets of up to 7 represent the encoded values. - Two sentinel pixels, using a `+7/+7/+7` offset, mark the beginning and end of the watermark. - Eight pixels between the sentinels encode one byte each: - 3 bits in red - 3 bits in green - 2 highest bits in blue ## Design Constraints - The watermark must remain nearly invisible and avoid adding interface elements. - It must work across different widget sizes, color profiles, display densities, and copy-paste workflows. - The border-based method avoids visualization-specific implementations while keeping the encoded region short. Datadog’s approach combines cached metadata with subtle RGB-level encoding, allowing screenshots to function like context-preserving share links without changing their appearance or the user’s workflow.

datadogredisincident-managementwatermarking+3

datadogApr 7, 2026

How we built a real-world evaluation platform for autonomous SRE agents at scale (opens in new tab)

The provided content does not include the blog post itself. It contains Datadog navigation links and a page title announcing that Datadog was named a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms, but no substantive discussion of the evaluation platform or its conclusions. ## Available Information - Datadog’s page promotes its recognition as a Gartner Magic Quadrant Leader. - The navigation lists products across: - Infrastructure and application monitoring - Logs, databases, and data observability - Security - Digital experience monitoring - CI/CD and software delivery - Incident and service management - AI capabilities, including Bits AI Agents and Bits Investigation - The referenced URL path suggests the intended article may concern Datadog’s “Bits AI eval platform,” but the article text is not included. ## Conclusion Please provide the full blog post content for a meaningful section-by-section summary.

aik8sobservabilitylog-management+4

datadogApr 7, 2026

How we built a real-world evaluation platform for autonomous SRE agents at scale (opens in new tab)

Bits AI SRE improved in isolated scenarios but lacked a way to detect regressions across the broader range of production incidents. The team found that tool-level tests and live replays could not capture failures caused by multi-step reasoning or changing telemetry. They built a replayable evaluation platform combining realistic investigation labels, scalable orchestration, and longitudinal performance tracking. ## Subtle Regressions from Well-Intentioned Features - Adding a monitor’s service name to the agent’s initial context improved some internal investigations. - Across broader scenarios, it introduced irrelevant signals that confused the agent and degraded unrelated investigations. - Because there was no representative evaluation set, the team could not measure the change’s wider impact before internal misses exposed it. - The incident demonstrated the need to evaluate every change across diverse investigation types. ## Limits of Tool Tests and Live Replay - Testing tools individually failed to capture errors caused by incorrect interactions between valid tool outputs. - Live investigation replay was difficult to scale because: - Results were not consistently aggregated. - Production environments changed. - Telemetry expired, making investigations unreplayable. - Standard evaluation frameworks assumed clean inputs and static datasets, unlike agents operating over production telemetry. - The team needed controlled, offline replay of realistic end-to-end investigations. ## Evaluation Labels and World Snapshots - Each label represents one production-style investigation. - It contains: - **Ground truth:** the issue’s actual root cause. - **World snapshot:** the queries and signals available when the issue occurred. - The agent is never shown the root cause directly; it must reason from the preserved signals. - Labels must cover varied technologies and failure modes, including: - Kubernetes pod failures - Kafka lag - Bad-code deployments - Complex multi-service business failures - A narrow or overly clean dataset would make performance appear better than it really is. ## Orchestrating Evaluations at Scale - The platform runs Bits against labels, scores the outcomes, and tracks quality over time. - It supports comparisons across: - Investigation categories - Model variants - Configuration versions - Evaluation runs - The architecture consists of a shared label set, an orchestration layer, and reporting infrastructure. - This allows teams to determine whether improvements in one domain, such as Kafka, regress another, such as Kubernetes. ## Scaling Label Creation - The team initially created labels manually from Datadog alerts. - Manual labeling provided early coverage but consumed engineering time and remained far from representative. - They embedded label generation into Bits itself: - Customer feedback and investigation data are used to derive root causes. - Relevant queries are preserved as the world snapshot. - Each user interaction becomes a potential evaluation case. - This increased label creation rates by an order of magnitude and allowed coverage to grow with product usage. ## Agent-Assisted Validation - Early labels required extensive human review, especially when feedback was ambiguous or reconstructed signals were uncertain. - As ingestion grew, manual review became a bottleneck. - Bits was then used to assist with validation by aggregating related signals, identifying relationships, and resolving ambiguous feedback before human review. ## Practical Conclusion Reliable agent improvement requires more than testing individual tools or replaying live incidents. A representative, production-derived label set combined with reproducible end-to-end evaluations makes regressions visible and enables safer iteration.

k8sdatadogkafkasre+3

datadogMar 23, 2026

When upserts don't update but still write: Debugging Postgres performance at scale (opens in new tab)

The provided content does not include the tech blog post itself. It consists primarily of Datadog’s website navigation and a promotional link about its Gartner recognition, so there is not enough article content to summarize reliably. ## Available Information - Datadog was named a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms. - The page navigation lists Datadog products across: - Infrastructure and application monitoring - Database and log management - Security - Digital experience monitoring - Software delivery - Incident and service management - AI-powered observability tools - The URL path references `debugging-postgres-performance`, suggesting the intended article may concern PostgreSQL performance debugging, but the article text is not included. Please provide the blog post’s body or a working text extraction for a substantive summary.

k8sdatadogobservabilitypostgresql+2

datadogMar 23, 2026

When upserts don't update but still write: Debugging Postgres performance at scale (opens in new tab)

Datadog needed to track when ephemeral hosts were last seen so inactive hosts could be deleted after seven days. A seemingly inexpensive PostgreSQL upsert caused disk writes to double and WAL syncs to quadruple, despite most operations not changing any data. Investigating the WAL revealed that conflict-handling upserts still lock conflicting rows and generate WAL activity, consuming the database’s limited write capacity. ## Tracking Host Activity Efficiently - Hosts stop reporting telemetry when they terminate, but Datadog has no direct termination signal. - Hosts inactive for seven days can be safely removed from the metadata store. - Updating the main host table on every observation would be too expensive because: - Large data centers generate more than 25,000 observations per second. - PostgreSQL MVCC creates a new row version for every update. - Updating the main table would rewrite all host metadata. - Datadog created a separate `host_last_ingested` table containing: - `host_id` as the primary key - `last_ingested` with a default timestamp - The table used `fillfactor=80` to leave page space for future updates. - No index was created on `last_ingested`, allowing updates to use Heap-Only Tuples (HOT) and avoid additional index writes. - Because only daily freshness was required, the timestamp needed to change at most once per day. ## The Conditional Upsert The initial query inserted a host if it did not exist and otherwise updated `last_ingested` only when the previous value was more than a day old: ```sql INSERT INTO host_last_ingested AS t VALUES ($1, now()) ON CONFLICT (host_id) DO UPDATE SET last_ingested = EXCLUDED.last_ingested WHERE t.last_ingested < EXCLUDED.last_ingested - '1 day'::interval; ``` - New hosts produced an insert. - Recently seen hosts matched the conflict but were expected to be no-ops because of the `WHERE` clause. - The team therefore expected most queries to avoid meaningful writes. ## Unexpected Disk and WAL Activity - During a gradual rollout at roughly 500 upserts per second: - Insertions initially increased as expected. - Actual updates remained mostly flat. - Write IOPS more than doubled. - WAL syncs increased by approximately the same amount. - This showed that the absence of an applied update did not mean the query was free. - Since PostgreSQL must flush WAL records at transaction commit, additional WAL activity directly increased disk pressure. - A PostgreSQL cluster’s single-writer design makes this write budget particularly important. ## Inspecting PostgreSQL WAL - PostgreSQL records database changes in its Write-Ahead Log, including table changes, index modifications, and related transaction activity. - The team used the `pg_walinspect` extension, available starting in PostgreSQL 15: ```sql CREATE EXTENSION pg_walinspect; ``` - Its `pg_get_wal_records_info` function allows inspection of WAL records between two Log Sequence Numbers (LSNs). - Examining the WAL helped explain why the conditional upsert generated writes even when the `WHERE` condition prevented the row update. - The underlying issue was that `ON CONFLICT DO UPDATE` still locks the conflicting row, and that locking activity is recorded in the WAL. The key lesson is that a PostgreSQL upsert that reports zero processed rows is not necessarily a true no-op. Conditional conflict updates can still create substantial WAL and locking overhead, so WAL inspection is essential when database write metrics do not match apparent update volume.

database-designpostgresqlupsertwal+4

datadogMar 9, 2026

When an AI agent came knocking: Catching malicious contributions in Datadog’s open source repos (opens in new tab)

Datadog announces that Gartner has named it a Leader in the 2026 Magic Quadrant for Observability Platforms. The surrounding product catalog presents Datadog as a broad platform spanning infrastructure, applications, data, logs, security, digital experience, software delivery, service management, and AI. However, the provided content does not include the blog post’s detailed analysis or Gartner’s specific evaluation criteria. ## Gartner Recognition - Datadog highlights its position as a **Leader** in the **Gartner Magic Quadrant for Observability Platforms 2026**. - The announcement links to a Gartner resource but provides no further details about the ranking, strengths, or limitations. ## Broad Observability Platform - **Infrastructure:** Infrastructure and container monitoring, metrics, Kubernetes autoscaling, network monitoring, serverless, cloud cost, storage, GPU monitoring, and Cloudcraft. - **Applications and data:** Application Performance Monitoring, service monitoring, profiling, dynamic instrumentation, database monitoring, data streams, data quality, and jobs monitoring. - **Logs and security:** Log management, sensitive-data scanning, audit trails, observability pipelines, cloud security, SIEM, code security, vulnerability management, and workload protection. - **Digital experience:** Browser and mobile RUM, product analytics, session replay, synthetic monitoring, mobile testing, error tracking, and experiments. - **Software delivery and service management:** CI visibility, test optimization, continuous testing, feature flags, code coverage, event management, SLOs, incident response, workflow automation, and service catalogs. - **AI capabilities:** Agent observability, GPU monitoring, AI integrations, AI agents, investigation tools, security analysis, MCP Server, and agent-building features. Datadog’s positioning is based on consolidating telemetry, security, developer, operations, and AI capabilities into one observability platform. To assess the Gartner recognition fully, readers would need the linked report or the complete article, which is not included here.

k8sdatadogobservabilitylog-management+2

datadogMar 9, 2026

When an AI agent came knocking: Catching malicious contributions in Datadog’s open source repos (opens in new tab)

Datadog describes how AI-powered attackers targeted its open-source repositories through malicious issues, pull requests, and comments. The campaign, attributed to the “hackerbot-claw” agent, focused on weaknesses in GitHub Actions and LLM-powered workflows. Datadog’s LLM-based review system and layered CI security controls detected the activity and helped limit its impact, while prompting further hardening. ## Why Open-Source Repositories Attract Attackers - Public repositories are attractive targets because automated CI/CD pipelines often build and execute code from external contributions. - Common attack techniques include: - Injecting user-controlled values, such as PR titles, into workflow scripts. - Using indirect poisoned pipeline execution to introduce malicious dependencies or build instructions. - Abusing `pull_request_target` workflows, which may run untrusted code with elevated permissions. - Prompt-injecting LLM-powered GitHub Actions used for issue triage, labeling, or code assistance. - Attackers may also disguise malicious changes through: - Large or obfuscated diffs. - Invisible Unicode characters. - Malicious libraries. - Imposter commits that resemble legitimate dependency references. ## Datadog’s LLM-Based Contribution Detection - Datadog receives dozens of external PRs each week across projects such as the Agent, tracers, SDKs, Vector, chaos-controller, and Stratus Red Team. - Its BewAIre system monitors GitHub events and selects security-relevant activity, including PRs and pushes. - BewAIre: - Extracts, normalizes, and enriches code diffs. - Sends them through a two-stage LLM pipeline. - Classifies changes as benign or malicious. - Produces a structured explanation for each verdict. - Malicious verdicts are forwarded to Datadog Cloud SIEM, where detection rules create enriched signals for the Security Incident Response Team to investigate. ## Hardening CI and Development Workflows - Datadog reduces the potential impact of successful attacks through multiple preventive controls: - Its `dd-octo-sts-action` generates minimally scoped, short-lived GitHub credentials using OIDC. - Long-lived and overly broad personal access tokens and GitHub Apps are being replaced. - Unused GitHub Actions secrets are identified and removed across thousands of repositories. - Organization-wide controls enforce branch protection, mandatory PR approval, commit signing, and lower-privilege default `GITHUB_TOKEN` permissions. - Engineers are provided with documented best practices and secure “golden paths” for CI development. ## The Hackerbot-Claw Campaign - Modern AI models are increasingly capable of offensive security tasks, especially when given tools, feedback loops, and autonomy. - StepSecurity reported an AI agent attacking open-source CI systems on March 1. - Between February 27 and March 2, the actor: - Opened 16 pull requests. - Created two issues and eight comments. - Targeted nine repositories across six organizations. - The activity was later linked to the hackerbot-claw agent, whose GitHub account was removed. - Datadog’s investigation began after BewAIre alerted the team to a suspicious contribution in the newly public `datadog-iac-scanner` repository on February 27. ## Practical Takeaway Organizations that accept public contributions should combine automated, AI-assisted review with least-privilege credentials, strict workflow permissions, secret management, mandatory approvals, and human incident response. Detection alone is insufficient; CI pipelines should be designed so that a malicious contribution has limited access and minimal opportunity to compromise secrets or production systems.

database-designllmci-cdopen-source+4

datadogMar 4, 2026

Designing MCP tools for agents: Lessons from building Datadog's MCP server (opens in new tab)

Datadog’s initial MCP server simply exposed existing APIs, but real-world agent use revealed major problems with context limits, inaccurate trend analysis, and tool overload. The team redesigned its tools around token efficiency, query-based analysis, and a smaller, more deliberate tool surface. These changes improved both answer quality and cost, though emerging agent features may eventually reduce the need for some optimizations. ## Context Efficiency Matters - Observability results can be extremely large: a log record may range from roughly 100 characters to 1 MB. - CSV or TSV is more token-efficient than JSON for tabular data, often using about half as many tokens per record. - YAML can reduce token usage for nested data by around 20% compared with JSON. - Removing rarely used fields from default responses, while allowing agents to request them when needed, further reduces output size. - Combined formatting and field-trimming improvements allowed some tools to return approximately five times more records within the same token budget. - Pagination by record count is unreliable when records vary greatly in size. Datadog instead paginates by token budget and returns a cursor when the limit is reached. - Tools such as Cursor and Claude Code increasingly write long results to disk, which could make response-format efficiency less important in the future. ## Let Agents Query Data - Retrieval-only tools forced agents to infer trends from incomplete samples, such as guessing which services generated the most errors. - Agents sometimes repeatedly fetched logs to compensate, wasting tokens and producing unreliable answers. - SQL lets agents aggregate and filter data directly: ```sql SELECT service, COUNT(*) AS error_count FROM logs WHERE status = 'error' GROUP BY service ORDER BY error_count DESC LIMIT 10 ``` - Agents can select only necessary fields, limit row counts, and calculate aggregates without loading raw data. - SQL improved correctness and reduced costs; some evaluation scenarios became about 40% cheaper. - Supporting SQL at Datadog’s scale required significant infrastructure work because traditional relational databases were insufficient. ## Tools Are Not Free - Exposing every API endpoint as a separate tool increases tool-selection errors and consumes context through tool descriptions. - Flexible tools can support multiple related workflows through carefully designed schemas, reducing the total tool count. - Toolsets provide a core collection by default while allowing users to opt into specialized capabilities, though users must anticipate their needs. - Layered tools can first explain how to accomplish a task and then execute it, keeping specialized functionality out of the initial context. - Layering introduces additional tool calls and therefore increases latency. - Improving agent context management, including tool search and dynamically loaded skills, may reduce the need for aggressive tool minimization over time. The practical recommendation is to design MCP tools for how agents actually reason: minimize and control output size, provide query and aggregation capabilities instead of raw retrieval alone, and expose a focused set of flexible tools rather than mirroring every API endpoint.

database-designdatadogobservabilityapi-design+4