datadogJun 23, 2026

How we migrated a live routing system using AI-assisted refactoring (opens in new tab)

Stream Router evolved from a small configuration file into a critical control-plane service routing Datadog’s massive metrics workload. Its original FoundationDB key-value model eventually hit transaction-size and performance limits because relational relationships were reconstructed in application code. Datadog redesigned the system around PostgreSQL and DuckDB, using AI-assisted, test-driven refactoring to accelerate the migration without disrupting production traffic. ## Stream Router’s Role in Datadog’s Metrics Pipeline - Datadog processes more than a hundred trillion events per day. - Stream Router determines which Kafka cluster, topic, partitions, and sharding strategy should handle each datapoint. - It serves both producers and queriers but does not process Kafka messages itself. - Routing decisions change frequently as infrastructure evolves, making correctness and historical tracking essential. ## From Configuration File to Control Plane - In 2016, routing was managed through a small configuration file distributed to services. - As the platform grew, the file expanded to thousands of lines and required manual edits and rollouts. - Stream Router replaced this workflow with: - A centralized gRPC service - API-managed routes - Automated, gradual rollouts - The write path used FoundationDB, while the read path served static RocksDB snapshots restored into memory. - This eventually became a bottleneck as routing tables and operational changes grew larger. ## Why the Key-Value Model Stopped Scaling - Routes reference streams and sharding strategies, while rules reference routes. - These relationships are inherently relational and require cross-entity validation. - The KV implementation loaded tens of thousands of records into application processes and reconstructed database-like relationships in code. - Some operations exceeded FoundationDB transaction-size limits. - Moving to PostgreSQL without changing the access patterns would not solve the issue; certain operations were estimated to require 45 minutes because of thousands of sequential database round trips. - The fundamental problem was the data model and application logic, not simply the choice of database. ## Designing the New Storage Architecture - The team redesigned the schema manually before using AI tools. - The relational model introduced explicit foreign keys between: - Streams - Sharding strategies - Routes - Rules - PostgreSQL was selected for the write path because it provided the required relational semantics and transaction model. - DuckDB was selected for the read path because: - It is embeddable and suitable for snapshot-based serving - It supports array columns - Its SQL dialect is closely compatible with PostgreSQL - Shared query logic could therefore work across both storage engines. ## AI-Assisted Refactoring - Claude and Cursor were used to accelerate a systematic, test-driven migration. - For each method, developers supplied: - The old implementation - The new schema - A failing test - AI generated an initial implementation, while tests determined whether it was correct. - The models assisted with method-level refactoring rather than autonomously designing the architecture. - Human expertise remained central to schema design, migration strategy, and evaluating system-level risks. ## Foundations for a Safe Migration - The migration benefited from infrastructure already present at Datadog. - Stream Router’s storage layer was isolated behind an internal `Controller` interface. - This modularity helped contain storage changes and enabled incremental refactoring. - Existing tests and clear boundaries provided confidence in generated implementations while production traffic continued. The central lesson is that AI was most effective as an accelerator inside a disciplined engineering process. A well-designed relational schema, modular storage abstraction, and failing tests provided the safety mechanisms; AI helped implement the resulting changes faster, but did not replace human architectural judgment.

database-designpostgresqlkafkagrpc+4

datadogMar 23, 2026

When upserts don't update but still write: Debugging Postgres performance at scale (opens in new tab)

Datadog needed to track when ephemeral hosts were last seen so inactive hosts could be deleted after seven days. A seemingly inexpensive PostgreSQL upsert caused disk writes to double and WAL syncs to quadruple, despite most operations not changing any data. Investigating the WAL revealed that conflict-handling upserts still lock conflicting rows and generate WAL activity, consuming the database’s limited write capacity. ## Tracking Host Activity Efficiently - Hosts stop reporting telemetry when they terminate, but Datadog has no direct termination signal. - Hosts inactive for seven days can be safely removed from the metadata store. - Updating the main host table on every observation would be too expensive because: - Large data centers generate more than 25,000 observations per second. - PostgreSQL MVCC creates a new row version for every update. - Updating the main table would rewrite all host metadata. - Datadog created a separate `host_last_ingested` table containing: - `host_id` as the primary key - `last_ingested` with a default timestamp - The table used `fillfactor=80` to leave page space for future updates. - No index was created on `last_ingested`, allowing updates to use Heap-Only Tuples (HOT) and avoid additional index writes. - Because only daily freshness was required, the timestamp needed to change at most once per day. ## The Conditional Upsert The initial query inserted a host if it did not exist and otherwise updated `last_ingested` only when the previous value was more than a day old: ```sql INSERT INTO host_last_ingested AS t VALUES ($1, now()) ON CONFLICT (host_id) DO UPDATE SET last_ingested = EXCLUDED.last_ingested WHERE t.last_ingested < EXCLUDED.last_ingested - '1 day'::interval; ``` - New hosts produced an insert. - Recently seen hosts matched the conflict but were expected to be no-ops because of the `WHERE` clause. - The team therefore expected most queries to avoid meaningful writes. ## Unexpected Disk and WAL Activity - During a gradual rollout at roughly 500 upserts per second: - Insertions initially increased as expected. - Actual updates remained mostly flat. - Write IOPS more than doubled. - WAL syncs increased by approximately the same amount. - This showed that the absence of an applied update did not mean the query was free. - Since PostgreSQL must flush WAL records at transaction commit, additional WAL activity directly increased disk pressure. - A PostgreSQL cluster’s single-writer design makes this write budget particularly important. ## Inspecting PostgreSQL WAL - PostgreSQL records database changes in its Write-Ahead Log, including table changes, index modifications, and related transaction activity. - The team used the `pg_walinspect` extension, available starting in PostgreSQL 15: ```sql CREATE EXTENSION pg_walinspect; ``` - Its `pg_get_wal_records_info` function allows inspection of WAL records between two Log Sequence Numbers (LSNs). - Examining the WAL helped explain why the conditional upsert generated writes even when the `WHERE` condition prevented the row update. - The underlying issue was that `ON CONFLICT DO UPDATE` still locks the conflicting row, and that locking activity is recorded in the WAL. The key lesson is that a PostgreSQL upsert that reports zero processed rows is not necessarily a true no-op. Conditional conflict updates can still create substantial WAL and locking overhead, so WAL inspection is essential when database write metrics do not match apparent update volume.

database-designpostgresqlupsertwal+4

datadogMar 9, 2026

When an AI agent came knocking: Catching malicious contributions in Datadog’s open source repos (opens in new tab)

Datadog describes how AI-powered attackers targeted its open-source repositories through malicious issues, pull requests, and comments. The campaign, attributed to the “hackerbot-claw” agent, focused on weaknesses in GitHub Actions and LLM-powered workflows. Datadog’s LLM-based review system and layered CI security controls detected the activity and helped limit its impact, while prompting further hardening. ## Why Open-Source Repositories Attract Attackers - Public repositories are attractive targets because automated CI/CD pipelines often build and execute code from external contributions. - Common attack techniques include: - Injecting user-controlled values, such as PR titles, into workflow scripts. - Using indirect poisoned pipeline execution to introduce malicious dependencies or build instructions. - Abusing `pull_request_target` workflows, which may run untrusted code with elevated permissions. - Prompt-injecting LLM-powered GitHub Actions used for issue triage, labeling, or code assistance. - Attackers may also disguise malicious changes through: - Large or obfuscated diffs. - Invisible Unicode characters. - Malicious libraries. - Imposter commits that resemble legitimate dependency references. ## Datadog’s LLM-Based Contribution Detection - Datadog receives dozens of external PRs each week across projects such as the Agent, tracers, SDKs, Vector, chaos-controller, and Stratus Red Team. - Its BewAIre system monitors GitHub events and selects security-relevant activity, including PRs and pushes. - BewAIre: - Extracts, normalizes, and enriches code diffs. - Sends them through a two-stage LLM pipeline. - Classifies changes as benign or malicious. - Produces a structured explanation for each verdict. - Malicious verdicts are forwarded to Datadog Cloud SIEM, where detection rules create enriched signals for the Security Incident Response Team to investigate. ## Hardening CI and Development Workflows - Datadog reduces the potential impact of successful attacks through multiple preventive controls: - Its `dd-octo-sts-action` generates minimally scoped, short-lived GitHub credentials using OIDC. - Long-lived and overly broad personal access tokens and GitHub Apps are being replaced. - Unused GitHub Actions secrets are identified and removed across thousands of repositories. - Organization-wide controls enforce branch protection, mandatory PR approval, commit signing, and lower-privilege default `GITHUB_TOKEN` permissions. - Engineers are provided with documented best practices and secure “golden paths” for CI development. ## The Hackerbot-Claw Campaign - Modern AI models are increasingly capable of offensive security tasks, especially when given tools, feedback loops, and autonomy. - StepSecurity reported an AI agent attacking open-source CI systems on March 1. - Between February 27 and March 2, the actor: - Opened 16 pull requests. - Created two issues and eight comments. - Targeted nine repositories across six organizations. - The activity was later linked to the hackerbot-claw agent, whose GitHub account was removed. - Datadog’s investigation began after BewAIre alerted the team to a suspicious contribution in the newly public `datadog-iac-scanner` repository on February 27. ## Practical Takeaway Organizations that accept public contributions should combine automated, AI-assisted review with least-privilege credentials, strict workflow permissions, secret management, mandatory approvals, and human incident response. Detection alone is insufficient; CI pipelines should be designed so that a malicious contribution has limited access and minimal opportunity to compromise secrets or production systems.

database-designllmci-cdopen-source+4

datadogMar 4, 2026

Designing MCP tools for agents: Lessons from building Datadog's MCP server (opens in new tab)

Datadog’s initial MCP server simply exposed existing APIs, but real-world agent use revealed major problems with context limits, inaccurate trend analysis, and tool overload. The team redesigned its tools around token efficiency, query-based analysis, and a smaller, more deliberate tool surface. These changes improved both answer quality and cost, though emerging agent features may eventually reduce the need for some optimizations. ## Context Efficiency Matters - Observability results can be extremely large: a log record may range from roughly 100 characters to 1 MB. - CSV or TSV is more token-efficient than JSON for tabular data, often using about half as many tokens per record. - YAML can reduce token usage for nested data by around 20% compared with JSON. - Removing rarely used fields from default responses, while allowing agents to request them when needed, further reduces output size. - Combined formatting and field-trimming improvements allowed some tools to return approximately five times more records within the same token budget. - Pagination by record count is unreliable when records vary greatly in size. Datadog instead paginates by token budget and returns a cursor when the limit is reached. - Tools such as Cursor and Claude Code increasingly write long results to disk, which could make response-format efficiency less important in the future. ## Let Agents Query Data - Retrieval-only tools forced agents to infer trends from incomplete samples, such as guessing which services generated the most errors. - Agents sometimes repeatedly fetched logs to compensate, wasting tokens and producing unreliable answers. - SQL lets agents aggregate and filter data directly: ```sql SELECT service, COUNT(*) AS error_count FROM logs WHERE status = 'error' GROUP BY service ORDER BY error_count DESC LIMIT 10 ``` - Agents can select only necessary fields, limit row counts, and calculate aggregates without loading raw data. - SQL improved correctness and reduced costs; some evaluation scenarios became about 40% cheaper. - Supporting SQL at Datadog’s scale required significant infrastructure work because traditional relational databases were insufficient. ## Tools Are Not Free - Exposing every API endpoint as a separate tool increases tool-selection errors and consumes context through tool descriptions. - Flexible tools can support multiple related workflows through carefully designed schemas, reducing the total tool count. - Toolsets provide a core collection by default while allowing users to opt into specialized capabilities, though users must anticipate their needs. - Layered tools can first explain how to accomplish a task and then execute it, keeping specialized functionality out of the initial context. - Layering introduces additional tool calls and therefore increases latency. - Improving agent context management, including tool search and dynamically loaded skills, may reduce the need for aggressive tool minimization over time. The practical recommendation is to design MCP tools for how agents actually reason: minimize and control output size, provide query and aggregation capabilities instead of raw retrieval alone, and expose a focused set of flexible tools rather than mirroring every API endpoint.

database-designdatadogobservabilityapi-design+4

datadogJan 7, 2026

Hardening eBPF for runtime security: Lessons from Datadog Workload Protection (opens in new tab)

eBPF gives security tools broad, efficient, and relatively safe access to Linux kernel activity, making it well suited for runtime threat detection. Datadog chose it for Workload Protection after comparing kernel modules, tracing interfaces, ptrace, seccomp, Linux Audit, and other approaches. However, five years of production use across diverse kernels showed that eBPF’s safety and performance benefits are not automatic; reliability, compatibility, observability, and operational discipline are essential at scale. ## Why Runtime Workload Protection Is Needed - Static analysis and vulnerability scanning cannot catch every threat. - Zero-days and vulnerable third-party dependencies can remain active while patches are being prepared or deployed. - Workload Protection is intended to: - Monitor known-vulnerable workloads until they can be patched. - Continuously observe all workloads. - Detect and help mitigate previously unknown vulnerabilities during incident response. ## Alternatives Evaluated Datadog evaluated a broad range of Linux monitoring and instrumentation mechanisms: - **Linux kernel modules** - Offer deep access and can hook or replace almost any kernel function. - Are invasive and often considered too risky for production infrastructure. - **Traditional tracing interfaces** - Include inotify, fanotify, kprobes, tracepoints, and perf events. - Provide useful visibility but generally need to be combined for comprehensive coverage. - **ptrace and seccomp-bpf** - Can provide detailed user-space process visibility. - Are less suitable as a unified solution for monitoring the whole system. - **Linux Audit** - Produces configurable streams for process execution, file access, and network activity. - Is widely used by security tooling but has its own performance and operational tradeoffs. - **Other mechanisms** - Netlink, LD_PRELOAD, and binfmt_misc were also considered. - Each involves compromises in reliability, visibility, or system impact. ## Why eBPF Stood Out - **Safety checks** - The kernel statically verifies eBPF bytecode before loading it. - Verification detects issues such as infinite loops and unsafe memory access. - This is safer than deploying custom kernel modules, though eBPF can still cause harm or performance problems. - **Performance** - eBPF generally has lower overhead than approaches such as Linux Audit or ptrace. - Actual impact depends heavily on implementation and workload. - **Unified visibility** - A single mechanism can observe process, filesystem, and network activity. - This avoids assembling multiple specialized tracing systems. - **Container and namespace coverage** - eBPF provides consistent visibility across namespaces, cgroups, and containers. - CO-RE (Compile Once–Run Everywhere) improves portability across Linux distributions and kernel versions. - **Enforcement capabilities** - BPF LSM programs support mandatory access controls. - This gives eBPF enforcement power beyond ordinary tracing mechanisms, which is important for runtime security. ## Lessons from Operating eBPF at Scale After five years of operating an agent that hooks process scheduling, filesystem, and networking internals, Datadog emphasizes that production eBPF is more complicated than its reputation suggests. The six areas of operational experience are: - Ensuring programs load, attach, and continue firing across kernel versions. - Capturing and enriching event data accurately. - Monitoring and auditing eBPF usage to reduce the attack surface. - Coexisting with other eBPF-based tools on the same host. - Measuring and controlling performance overhead. - Shipping changes safely through disciplined rollout practices. The practical recommendation is to treat eBPF as powerful infrastructure rather than a maintenance-free kernel feature: validate behavior across kernels and workloads, monitor its own operation, measure overhead continuously, and use cautious deployment practices.

database-designobservabilityebpflinux-kernel+4

datadogNov 18, 2025

Scaling real-time file monitoring with eBPF: How we filtered billions of kernel events per minute (opens in new tab)

File integrity monitoring must provide more than proof that a file changed: security teams need to know how, why, and by whom it changed. Datadog found that filesystem scans, inotify, and auditd could not provide sufficient context, reliability, or scalability. An eBPF-based approach delivered kernel-level visibility into processes and containers, but required extensive filtering and edge processing to handle more than 10 billion events per minute. ## Why Traditional Monitoring Falls Short - Periodic scans can miss changes that are made and reverted between scans. - Scans show that a file changed, but not the process, container, or mechanism responsible. - `inotify` lacks the system-level context needed to correlate file events with processes and containers. - `auditd` offers richer information but can impose significant performance overhead and struggle under heavy load. ## eBPF for Context-Rich File Monitoring - eBPF observes file activity directly in the Linux kernel in real time. - Events can include: - The modified file - The process that triggered the change - The container in which the process ran - Additional security-relevant metadata - This context makes events more useful for investigations than simple “file changed” notifications. ## Scaling at the Agent and Backend - Datadog observed more than 10 billion file-related events per minute across its infrastructure. - Each serialized event was approximately 5 KB, making unrestricted transmission infeasible—potentially several terabytes per second. - Sending every event would also overload Agents through excessive CPU, memory, serialization, and network usage. - Agent-side rules filter events locally, discarding noise before transmission. - This reduced the stream to roughly one million events per minute while preserving detection coverage. ## Filtering Events in the Kernel - A basic architecture loads eBPF programs into the Agent, observes system activity, writes events to a ring buffer, and evaluates them in user space. - Sensitive workloads can generate up to 5,000 relevant syscalls per second. - Initial implementations risked ring-buffer backlogs and dropped events, creating security blind spots. - Datadog moved as much evaluation as possible into eBPF programs to reduce the number of events reaching user space. - The Agent could then perform a deeper second-stage evaluation before forwarding events to the backend. ## Two-Stage Evaluation: Approvers and Discarders - eBPF’s safety constraints limit computation, especially on older Linux kernels. - The system therefore separates evaluation into: - **In-kernel filtering:** Lightweight decisions that quickly approve or discard events. - **User-space evaluation:** More complex analysis using richer context, correlations, and logic unsuitable for the kernel. - This design balances kernel safety and performance with the need for detailed security detection. Datadog’s approach shows that scalable FIM requires combining eBPF’s deep visibility with aggressive filtering at the edge and in the kernel. The practical recommendation is to keep expensive analysis in user space while rejecting irrelevant events as early as possible.

database-designlinuxebpfsecurity-monitoring+4

datadogNov 4, 2025

Replication redefined: How we built a low-latency, multi-tenant data replication platform (opens in new tab)

Datadog built a managed, multi-tenant data replication platform to move data reliably across thousands of services without brittle, point-to-point integrations. The effort began by separating analytical search workloads from a shared PostgreSQL database, then evolved into automated pipeline provisioning with Temporal. The platform favors asynchronous replication to improve scalability and resilience, accepting limited replication lag in exchange for lower application latency and reduced operational coupling. ## Scaling Search Beyond PostgreSQL - A shared PostgreSQL database initially provided low-latency access, ACID guarantees, and low operational cost. - As data volumes grew, complex joins and aggregations became increasingly slow. - Datadog’s Metrics Summary page had to join: - 82,000 active metrics - 817,000 metric configurations - Page latency reached approximately 7 seconds at p90, while repeated facet changes generated additional expensive queries. - Index and disk bloat, memory pressure, VACUUM and ANALYZE overhead, and rising I/O wait further reduced throughput. - Rather than continuing to optimize PostgreSQL for analytical search, Datadog moved search and aggregation workloads to a dedicated search platform. - Data was denormalized during replication, producing document-oriented indexes better suited to faceted search. - The resulting system reduced page-load times by as much as 97%—from roughly 30 seconds to 1 second—while maintaining about 500 ms of replication lag. ## Automating Pipeline Provisioning with Temporal Provisioning a replication pipeline required coordinating multiple systems and configuration steps: - Enabling PostgreSQL logical replication with `wal_level`. - Creating users and assigning replication permissions. - Configuring publishers and replication slots. - Deploying Debezium instances to capture PostgreSQL changes. - Creating Kafka topics and mapping them to Debezium instances. - Adding heartbeat tables to monitor replication and prevent excessive WAL retention. - Configuring sink connectors to write Kafka data into the search platform. Manual management became increasingly difficult across many pipelines and data centers. Datadog used Temporal workflows to split provisioning into modular, repeatable tasks and combine them into higher-level orchestrations. This reduced errors, improved consistency, and allowed engineers to create and modify pipelines without repeating complex operational procedures. ## Choosing Asynchronous Replication - Synchronous replication provides strong consistency by waiting for replicas to acknowledge each write. - However, it increases latency and operational complexity, particularly across distributed environments. - Asynchronous replication allows the primary system to acknowledge writes immediately while replicas catch up afterward. - Datadog selected the asynchronous model because it decouples application performance from network latency and replica availability. - The trade-off is temporary replication lag during failures or periods of pressure, but the model offers better scalability and resilience for high-throughput systems. Datadog’s experience suggests that replication should be treated as a managed platform rather than a collection of custom integrations. Separating workloads, automating provisioning, and choosing asynchronous delivery can improve performance and reliability while reducing the operational burden on individual engineering teams.

database-designpostgresqlkafkaelasticsearch+4

datadogOct 15, 2025

Failure is inevitable: Learning from a large outage, and building for reliability in depth at Datadog (opens in new tab)

Datadog’s March 2023 outage exposed a fundamental weakness in its reliability strategy: although 40–50% of production Kubernetes nodes remained operational, customers experienced the platform as entirely unavailable. The incident showed that preventing every failure is impossible and that systems must instead continue delivering useful, accurate service when components fail. Datadog consequently began redesigning products around graceful degradation, prioritizing data preservation, fresh information, and partial results. ## Lessons from the March 2023 Incident - An unsupervised global update triggered a restart interaction that disconnected roughly 50–60% of production Kubernetes nodes. - The web interface recovered quickly, but logs, metrics, alerts, traces, and other core features became unavailable. - Pages loaded without displaying customer data, creating a nearly complete outage from the user’s perspective. ## Limits of Traditional Root-Cause Analysis - Datadog identified the legacy global security-update mechanism as the immediate trigger and disabled it. - Fixing that mechanism alone could not address the broader class of failures caused by certificates, configuration changes, overloads, date-handling bugs, or other unexpected events. - The company concluded that resilience requires reducing the impact of failures, not merely preventing one specific failure mode. ## Why Partial Infrastructure Became a Total User-Facing Failure - Datadog’s systems historically favored complete correctness over partial visibility. - For example, metric queries could wait until all relevant tags were processed to avoid showing misleading values or triggering false alerts. - During a large outage, this behavior created a “square-wave” failure: missing some data caused the system to show no data. - Ordered queues could stall fresh results behind stuck work, retries could overload already-strained services, and node-specific processing could make surviving capacity ineffective. - The underlying design assumption was that systems should either function fully or stop, rather than degrade while continuing to provide value. ## Prioritizing Graceful Degradation Datadog shifted from relying primarily on redundancy and “never-fail” architectures to explicitly designing for inevitable failures. - Customer data should never be lost, even if delivery is delayed. - Fresh, real-time data should take priority over stale backlog processing. - Systems should provide partial but accurate results whenever possible instead of returning nothing. ## Persistent Storage at the Start of Processing Pipelines - The outage caused a limited but non-zero amount of irreversible customer data loss. - Some pipelines acknowledged data before writing it to replicated storage, leaving unreplicated data only in memory or on a local disk. - When a node failed, that data disappeared and could not be recovered through agent retries. - After the node loss, surviving intake nodes also struggled to write to downstream replicated stores. - Their memory and local-disk buffers eventually filled, causing additional data loss as the outage continued. - Datadog therefore identified persistent intake storage as a key requirement for preserving data during large-scale failures. The broader recommendation is to design systems not only to prevent outages, but also to remain useful during them: preserve every accepted event, prioritize current information, and expose accurate partial results instead of failing completely.

database-designk8sdata-loss-preventiongraceful-degradation+4

datadogOct 1, 2025

Inside Husky’s query engine: Real-time access to 100 trillion events (opens in new tab)

Datadog’s Husky event store is designed not merely to store more than 100 trillion events, but to make them queryable interactively at massive scale. Its query engine handles schemaless data, highly variable tenant workloads, petabytes of object-store data, and both highly selective searches and broad analytics queries. The architecture combines distributed planning, metadata-driven pruning, coordinated execution, and reader-level optimizations to minimize the data and fragments that must be scanned. ## Husky’s Data and Query Workloads - Events contain a timestamp and flexible attributes. - Data shapes vary widely: - Logs, network events, and traces have different schemas. - Tenants may produce either a few large events or enormous numbers of small events. - Most queries fall into two categories: - **Needle-in-a-haystack searches**, such as finding a specific IP connection, error message, or trace. - **Analytics-style searches**, such as time series, grouped breakdowns, and distributions. - The engine must also support raw event retrieval and more complex queries such as joins. ## Distributed Query Path Husky divides query execution among four multi-tenant services deployed across regions and data centers. ### Query Planner - Serves as the main entry point for event-store queries. - Resolves query context, validates and throttles requests, and integrates with other Datadog data stores. - Applies optimizations and statistics to split queries into time-based steps. - Schedules those steps on orchestrators and merges their results into the final response. ### Query Orchestrator - Acts as the gateway to Husky’s stored data. - Fetches fragment metadata, including: - File paths and versions - Row counts - Timestamp boundaries - Zone maps for query matching - Dispatches only relevant fragments to reader nodes. - Uses zone-map pruning to reduce downstream work by up to 60% for structured events and about 30% on average. - Aggregates results after fragment processing, which can require more computation than query planning. ### Metadata Service - Provides an abstraction over FoundationDB clusters. - Preserves atomicity during operations such as compaction, preventing duplicate data from appearing in query results. - Separates FoundationDB implementation details from the rest of the query system. - Must work within FoundationDB’s five-second transaction limit. ### Reader Service - Receives a query and selected fragments, then returns results quickly. - Performs the direct scan and execution work over fragment data. - Contains multiple optimizations intended to keep queries interactive despite scanning data stored in blob storage at extreme scale. ## Minimizing Data Scans The reader service follows the principle that the fastest query is the one that avoids unnecessary work. - Scanning less data reduces both latency and storage costs. - Touching fewer fragments limits expensive object-store operations. - This is especially important because Husky stores millions of fragments daily and cannot afford multiple blob-storage GET requests for every fragment in every query. ## Row Groups and Reader Execution - Fragments can contain millions of rows, making fully in-memory processing risky and potentially causing memory pressure or failures. - To limit data retrieval, fragments are physically organized into **row groups**. - Row groups allow the reader to fetch only portions of a fragment needed for a query rather than loading the entire file. - The reader uses an iterator-based execution model inspired by the Volcano query-processing architecture. - The provided article ends as it begins explaining how this row-group layout supports efficient iterator-based query execution.

database-designdata-architecturefoundationdbhusky+3

datadogSep 18, 2025

From hand-tuned Go to self-optimizing code: Building BitsEvolve (opens in new tab)

Datadog found that small Go-level optimizations can produce substantial infrastructure savings when applied to heavily used, autoscaled services. Manual work—such as removing bounds checks and prioritizing common input paths—delivered improvements ranging from 25% to over 90% in targeted functions. These successes also revealed the need to automate expert optimization techniques through systems like Datadog’s internal BitsEvolve. ## Finding Hotspots That Matter - Micro-optimizations are worthwhile when: - Functions run millions or billions of times. - Services are aggressively autoscaled, allowing CPU savings to reduce machine counts. - Resource usage drops measurably. - Datadog focused on high-throughput services processing timeseries tags and values. - Individual hotspots sometimes represented only 0.5% of compute, but repeated savings could add up to tens of thousands of dollars annually. - The broader goal was a 5–10% reduction in CPU usage across many improvements. ## Removing Bounds Checks from `NormalizeTag` - `NormalizeTag` called `isNormalizedASCIITag`, a frequently executed validator for ASCII tag strings. - AI coding tools suggested changes that were correct but produced no measurable performance gains. - Examining Go assembly with Compiler Explorer revealed two `runtime.panicBounds` calls per loop iteration. - Restructuring the loop eliminated unnecessary bounds checks and enabled further tuning. - The function became 25% faster, reducing service CPU usage by 0.75% and producing projected annual savings of tens of thousands of dollars. ## Using Observability to Optimize for Real Inputs - `NormalizeTagArbTagValue` handled arbitrary input, including invalid UTF-8 and binary data, and consumed 4.5% of CPU in its processing service. - Production data showed: - Nearly all inputs were ASCII. - UTF-8 appeared in fewer than 3% of cases. - Invalid UTF-8 represented less than 0.01% of inputs. - A fast path optimized for common ASCII data made the function more than 90% faster without reducing correctness or safety. - The change generated projected annual savings of hundreds of thousands of dollars. - The result demonstrated that observability is essential: optimization decisions should reflect actual workloads rather than hypothetical edge cases. ## From Manual Optimization to Automation - Deep performance tuning requires specialized knowledge of profiling, compiler behavior, assembly, and workload analysis. - Although the results can be valuable, the process is time-consuming and difficult to scale across a large organization. - Datadog wanted to move beyond isolated “heroic” optimizations toward a repeatable and automated process. - The manual techniques used by performance engineers became the foundation for heuristics in BitsEvolve, an internal agentic system intended to optimize code systematically. Datadog’s experience suggests that organizations should combine production observability with compiler-level analysis, prioritize high-impact hot paths, and automate proven optimization patterns so performance gains do not depend solely on a small group of experts.

database-designobservabilitygoai-assisted-coding+4

datadogAug 12, 2025

Scaling down to speed up: How we improved efficiency of live process metrics by 100x (opens in new tab)

Datadog redesigned its real-time Processes and Containers pipeline to avoid collecting high-frequency metrics that users never see. By limiting 2-second collection to hosts actively viewed and using standard 10-second data for sorting, the company reduced real-time traffic by over 100x, cut infrastructure costs by 98%, and lowered Agent resource usage. The approach also improved scalability without sacrificing the live investigation experience. ## Original Real-Time Collection Model - Datadog Agents normally collect process and container metrics every 10 seconds. - When a user opened a live Processes or Containers view, all hosts in that tenant switched to 2-second collection. - This supported near-real-time monitoring similar to `htop`, but across distributed infrastructure. - As tenants grew, the pipeline had to process millions of processes per second, even though users typically viewed only around 50 processes or containers. - Live sorting required keeping all tenant data in memory on a single server, limiting horizontal scaling and forcing vertical scaling. ## Refocusing on User-Visible Data - Most collected metrics were never displayed to users. - Datadog determined that real-time collection only needed to be enabled for hosts running the processes or containers currently in view—up to roughly 50 hosts per user. - Internal telemetry suggested this could reduce traffic by more than 100x. - This required tracking active host subscriptions and updating them as users navigated the product. - Because sorting occurred every 10 seconds, it did not need 2-second data. Datadog switched live views to use the existing 10-second metrics, aligning live and historical sorting logic. ## Host Subscription Filtering - A proof of concept added host subscriptions to the live data servers. - Servers filtered Kafka payloads and discarded data for hosts without active subscriptions. - This immediately reduced: - Memory usage by 85% - CPU usage by 33% - The improvement came from storing fewer live metrics and processing fewer incoming payloads. - The prototype confirmed that filtering preserved product behavior while simplifying sorting. ## Moving Filtering Earlier in the Pipeline - Late filtering improved live data servers but still left unnecessary work for the rest of the system and customer-side Datadog Agents. - Datadog therefore planned to propagate subscription state to the intake service. - Live data servers publish users’ active host sets over Kafka once per second. - The intake service consumes this information and decides which hosts should activate 2-second process and container metric collection. - This allows real-time collection to be restricted to hosts users are actively investigating while maintaining responsive live views. Datadog’s redesign demonstrates that real-time systems scale more effectively when they prioritize data users can actually see. Filtering at intake, limiting high-frequency collection to subscribed hosts, and reusing standard-resolution data for sorting provide a simpler and more economical architecture without eliminating live functionality.

database-designdatadogdockerkafka+4

datadogAug 4, 2025

Evolving our real-time timeseries storage again: Built in Rust for performance at scale (opens in new tab)

Datadog built a sixth-generation real-time timeseries database in Rust to keep pace with rapidly growing metric volume, cardinality, and query complexity. The new engine is designed for high throughput and low latency, reportedly achieving 60× higher ingestion performance and 5× faster peak-scale queries. Its development reflects a long evolution from general-purpose databases toward a purpose-built system with tighter control over storage, I/O, and execution. ## Datadog’s Metrics Storage Architecture - The metrics platform includes ingestion, enrichment, real-time and long-term storage, querying, and alerting. - This post focuses on real-time storage, which is split into two independently deployed services: - **RTDB:** Stores raw metric tuples of `<timeseries_id, timestamp, value>`, performs aggregations, and serves recent data. - **Index database:** Stores metric identifiers and their tags as `<timeseries_id, tags>`. - A storage router distributes incoming metrics across RTDB nodes based on load. - The query service contacts the relevant RTDB and index nodes, retrieves results, and combines them. - Each RTDB node includes: - An ingestion subsystem - A storage engine - A durability snapshot module - A gRPC query layer - Throttlers for resource management - A shared control plane coordinating these components ## Generation 1: Cassandra - Cassandra provided strong write scalability and a familiar operational model. - It was influenced by systems such as OpenTSDB and HBase. - Its main weaknesses were: - Limited flexibility for real-time queries - Difficulty supporting complex alerting and analytical workloads - Inefficient retrieval of large datasets - These limitations prompted Datadog to move to Redis. ## Generation 2: Redis - Redis improved read performance and offered a flexible, easy-to-understand storage model. - Datadog avoided Redis’s built-in clustering for reliability reasons, requiring the team to operate many independent instances. - Important drawbacks included: - Single-threaded execution limiting snapshotting during live traffic - Severe but uncommon memory-management and threading failures - Serialization and cross-process communication overhead - Inefficient memory layout, disk I/O, and CPU usage at scale - Redis nevertheless provided valuable operational insight and clarified the need for a purpose-built engine with direct control over I/O and system resources. ## Generation 3: MDBM and Memory-Mapped I/O - MDBM provided a memory-mapped key-value store based on `mmap`. - The operating system’s page cache loaded database pages on demand, making disk-backed data behave similarly to in-memory structures. - This simplified storage interactions initially, but performance degraded as workloads intensified. - Memory-mapped I/O introduced subtle performance and correctness concerns, leading Datadog to conclude that explicit I/O management would scale better. ## Generation 4: A Go-Based B+ Tree - Datadog replaced MDBM with a custom B+ tree written in Go. - The engine supported a thread-per-core-oriented design, with Go’s scheduler providing a useful foundation. - This change significantly improved throughput and latency. - It also created a platform that could be optimized more aggressively for Datadog’s workload. ## Generation 5: DDSketch and RocksDB - Datadog introduced DDSketch to support distribution metrics and accurate percentile estimation. - The existing Go engine was optimized for scalar floating-point values and was difficult to extend for sketches. - RocksDB was therefore integrated to store DDSketch data, offering flexibility and strong performance. - Over time, maintaining separate storage technologies created pressure to build a unified engine capable of handling multiple metric types efficiently. ## The Move Toward a New Engine - The progression from Cassandra to Redis, MDBM, a custom Go B+ tree, and RocksDB shows a pattern of replacing general-purpose components as scale and workload diversity increased. - Each generation solved important problems but introduced new operational or architectural trade-offs. - Datadog ultimately needed a unified, purpose-built storage system with: - High-throughput ingestion - Low-latency queries - Better support for high-cardinality data - Efficient handling of different metric types - More direct control over concurrency, memory, and I/O - The sixth generation addresses these requirements through a Rust-based real-time timeseries database. Datadog’s experience suggests that general-purpose storage systems can be effective early on, but sustained growth eventually favors a specialized engine. The practical lesson is to optimize existing infrastructure first while developing a purpose-built replacement before scale and workload complexity make incremental fixes insufficient.

database-designrustredisgrpc+4

datadogJul 17, 2025

How we tracked down a Go 1.24 memory regression across hundreds of pods (opens in new tab)

Go 1.24 initially caused an unexpected ~20% increase in memory usage across several services, despite its Swiss Tables implementation being expected to reduce memory consumption. The increase appeared in system-level RSS metrics but not in Go’s runtime metrics or heap profiles. Investigation showed that a runtime allocator refactor likely caused more of the Go heap’s virtual memory to be committed to physical RAM. ## The Unexpected Go 1.24 Memory Increase - The issue emerged during an internal rollout of Go 1.24. - Multiple environments showed approximately 20% higher memory usage. - A staging bisect directly linked the increase to the Go 1.24 upgrade. - The behavior was surprising because Go 1.24’s headline Swiss Tables feature promised lower CPU and memory overhead. ## Ruling Out Swiss Tables and Mutex Changes - Swiss Tables were disabled with: ```bash GOEXPERIMENT=noswissmap ``` - Memory usage did not improve, ruling out the new map implementation as the cause. - The new spin-bit mutex implementation was disabled with: ```bash GOEXPERIMENT=nospinbitmutex ``` - The memory increase remained, eliminating this runtime change as the likely culprit. ## System Metrics vs. Go Runtime Metrics - Go runtime metrics showed almost no change after the upgrade. - System metrics reported a significant increase in resident set size (RSS). - RSS measures physical memory currently used in RAM, while Go’s runtime accounting primarily reflects allocated virtual memory. - This discrepancy matters operationally because systems such as Kubernetes and the Linux OOM Killer rely on physical-memory metrics. ## Examining the Go Heap with `/proc/[pid]/smaps` - Linux’s `/proc/[pid]/smaps` exposed memory usage for individual mappings. - In Go 1.24, the main Go heap mapping had roughly: - 1.28 GiB of virtual memory allocated - 1.26 GiB resident in physical RAM - In Go 1.23, a similarly sized heap mapping had about 300 MiB less RSS than its virtual size. - Other memory regions were not significantly affected, indicating that the increased RSS was isolated to the Go heap. - Upstream changes to label Go-allocated memory regions should make future `maps` and `smaps` investigations easier. ## The Suspected Allocator Regression - The evidence suggested Go 1.24 was not requesting substantially more virtual memory. - Instead, previously uncommitted virtual memory was being committed to physical RAM, increasing RSS without changing Go’s internal memory totals. - A major refactoring of the runtime’s `mallocgc` function stood out in the Go 1.24 changelog. - The investigation therefore focused on this allocator change as the likely source of the regression. Go 1.24’s memory increase was caused not by Swiss Tables or mutex changes, but likely by altered heap allocation behavior in the runtime. Comparing RSS with Go’s runtime metrics—and inspecting `/proc/[pid]/smaps`—was essential for identifying the allocator-related discrepancy.

database-designk8sgolinux+4

datadogJul 17, 2025

How Go 1.24's Swiss Tables saved us hundreds of gigabytes (opens in new tab)

Go 1.24 initially caused a Go runtime regression that increased RSS across Datadog services, but some high-traffic workloads ultimately used substantially less memory. The reduction came from Go 1.24’s Swiss Tables map implementation, which made a large, mostly read-only routing cache more compact. Profiling also revealed opportunities to reduce memory further by removing redundant data from the cached values. ## The Unexpected Memory Reduction - Datadog observed roughly **500 MiB less live heap** in the `shardRoutingCache` map after upgrading to Go 1.24. - With `GOGC=100`, that translated to approximately **1 GiB less total memory usage**. - Even after accounting for an expected **400 MiB RSS increase** from the `mallocgc` regression, the service achieved a net reduction of about **600 MiB**. - The improvement was most visible in high-traffic environments because they contained larger routing caches. ## The `shardRoutingCache` Data Structure - The cache maps routing keys to shard information: ```go map[string]Response ``` - Each `Response` contains: - `ShardID int32` - `ShardType` - `RoutingKey string` - `LastModified *time.Time` - The map is populated mainly during service startup by querying a database. - It is rarely modified afterward, making its memory layout and initial allocation particularly important. - The routing key is stored both as the map key and again inside the value, creating potential redundancy. ## Estimating Memory per Entry - On a 64-bit system, a map key’s string header occupies **16 bytes**. - The value requires approximately: - 4 bytes for `ShardID` - 8 bytes for `ShardType` - 16 bytes for the `RoutingKey` string header - 8 bytes for the `LastModified` pointer - The value totals 36 bytes before alignment, or roughly **40 bytes with padding**. - Including the key header, each key-value pair requires about **56 bytes**, excluding the separately allocated string and `time.Time` data. ## Go 1.23 Bucket-Based Maps - Go 1.23 maps used hash tables organized into an array of buckets. - The number of buckets was always a power of two, and each bucket contained **eight slots**. - Reads and writes required scanning the slots in the selected bucket to find a matching key or an empty position. - When a bucket filled, Go added linked overflow buckets, which increased memory usage and made lookups more expensive. - Map growth occurred when the average load factor exceeded **13/16, or 6.5 of 8 slots**. - The map then allocated twice as many buckets. - To avoid a large latency spike, growth was incremental: old and new bucket arrays coexisted while entries were gradually moved during subsequent writes. ## Why Workload Shape Matters - The routing cache is populated in a startup-heavy phase and then primarily read. - Such a workload benefits from a compact map representation because it does not need frequent insertions or growth. - Differences in cache size and traffic patterns explain why the memory improvement was significant in some environments but not uniform across the fleet. Go 1.24’s Swiss Tables implementation can substantially reduce memory usage for large, stable maps, even when another runtime change causes RSS growth. Teams should profile real production heaps after Go upgrades and inspect large structs for duplicated strings, unnecessary pointers, and other avoidable per-entry overhead.

database-designgogarbage-collectionmemory-optimization+4

datadogJun 24, 2025

How we built a real-time, client-side noise suppression library without server dependencies (opens in new tab)

Datadog’s CoScreen team needed high-quality noise suppression that could run in real time on client devices and integrate with WebRTC. Since existing solutions were either too slow, server-dependent, expensive, or difficult to embed, they built and open-sourced **dtln-rs**, a portable Rust library based on the DTLN model. It processes one second of audio in about 33 ms on an M1 MacBook Pro and supports WebAssembly, Node.js, and native clients. ## Introducing dtln-rs - dtln-rs is a lightweight, open-source noise reduction library based on the Dual-Signal Transformation LSTM Network (DTLN). - It can produce: - A WebAssembly module - A native Rust library - A Node.js native module - The library is designed to integrate with WebRTC-based applications. - Datadog also released a demo showing how to embed the filter in an application or webpage. ## Demonstrating Real-World Noise Suppression - The project was motivated by common remote-work disruptions, including lawn mowers and other background noise. - In one test, the filter removed a neighbor’s lawn mower so effectively that a colleague could not tell it was running. - The team used this result as evidence that the embedded library could provide meaningful value to CoScreen users. ## How DTLN Enables Real-Time Processing - AI noise suppression learns to distinguish desired speech from unwanted background sounds. - DTLN uses a short-time Fourier transform (STFT) to divide audio into smaller segments and analyze the magnitude of different frequencies. - It also uses phase information, which describes the starting position of each frequency in the sound wave. - A model analyzes magnitude and phase data to determine which parts are speech and which are noise. - Its LSTM-based architecture can adapt to different environments, such as: - Air-conditioner hum - Cafe conversations - Paper rustling - The combination of deep learning and efficient signal processing allows DTLN to operate with near-instantaneous latency. ## Why Existing Noise Suppression Solutions Were Insufficient - Many advanced machine-learning models require powerful backend servers, with processed audio sent back over the network. - This approach adds latency, infrastructure complexity, and operating costs. - WebRTC remains widely adopted but generally relies on older, built-in noise reduction techniques. - Earlier solutions such as RNNoise can reduce noise but often do not match the quality of newer commercial systems. - Although Web Audio and WebAssembly make custom client-side processing possible, implementation still requires substantial engineering effort. - Large companies can deploy specialized servers and models trained on enormous speech datasets, but smaller teams may not have the resources to do so. - CoScreen’s search for an alternative led to DTLN, which could run in real time on standard hardware and be embedded directly into client applications. ## Practical Recommendation For WebRTC applications needing client-side, real-time noise suppression, dtln-rs offers a portable alternative to expensive server-based services. Its Rust foundation and support for WebAssembly, Node.js, and native targets make it suitable for web, desktop, and embedded clients.

database-designrustdeep-learningwebassembly+4