datadogJun 23, 2026

How we migrated a live routing system using AI-assisted refactoring (opens in new tab)

Stream Router evolved from a small configuration file into a critical control-plane service routing Datadog’s massive metrics workload. Its original FoundationDB key-value model eventually hit transaction-size and performance limits because relational relationships were reconstructed in application code. Datadog redesigned the system around PostgreSQL and DuckDB, using AI-assisted, test-driven refactoring to accelerate the migration without disrupting production traffic. ## Stream Router’s Role in Datadog’s Metrics Pipeline - Datadog processes more than a hundred trillion events per day. - Stream Router determines which Kafka cluster, topic, partitions, and sharding strategy should handle each datapoint. - It serves both producers and queriers but does not process Kafka messages itself. - Routing decisions change frequently as infrastructure evolves, making correctness and historical tracking essential. ## From Configuration File to Control Plane - In 2016, routing was managed through a small configuration file distributed to services. - As the platform grew, the file expanded to thousands of lines and required manual edits and rollouts. - Stream Router replaced this workflow with: - A centralized gRPC service - API-managed routes - Automated, gradual rollouts - The write path used FoundationDB, while the read path served static RocksDB snapshots restored into memory. - This eventually became a bottleneck as routing tables and operational changes grew larger. ## Why the Key-Value Model Stopped Scaling - Routes reference streams and sharding strategies, while rules reference routes. - These relationships are inherently relational and require cross-entity validation. - The KV implementation loaded tens of thousands of records into application processes and reconstructed database-like relationships in code. - Some operations exceeded FoundationDB transaction-size limits. - Moving to PostgreSQL without changing the access patterns would not solve the issue; certain operations were estimated to require 45 minutes because of thousands of sequential database round trips. - The fundamental problem was the data model and application logic, not simply the choice of database. ## Designing the New Storage Architecture - The team redesigned the schema manually before using AI tools. - The relational model introduced explicit foreign keys between: - Streams - Sharding strategies - Routes - Rules - PostgreSQL was selected for the write path because it provided the required relational semantics and transaction model. - DuckDB was selected for the read path because: - It is embeddable and suitable for snapshot-based serving - It supports array columns - Its SQL dialect is closely compatible with PostgreSQL - Shared query logic could therefore work across both storage engines. ## AI-Assisted Refactoring - Claude and Cursor were used to accelerate a systematic, test-driven migration. - For each method, developers supplied: - The old implementation - The new schema - A failing test - AI generated an initial implementation, while tests determined whether it was correct. - The models assisted with method-level refactoring rather than autonomously designing the architecture. - Human expertise remained central to schema design, migration strategy, and evaluating system-level risks. ## Foundations for a Safe Migration - The migration benefited from infrastructure already present at Datadog. - Stream Router’s storage layer was isolated behind an internal `Controller` interface. - This modularity helped contain storage changes and enabled incremental refactoring. - Existing tests and clear boundaries provided confidence in generated implementations while production traffic continued. The central lesson is that AI was most effective as an accelerator inside a disciplined engineering process. A well-designed relational schema, modular storage abstraction, and failing tests provided the safety mechanisms; AI helped implement the resulting changes faster, but did not replace human architectural judgment.

postgresqldatabase-designkafkagrpc+4

datadogJun 4, 2026

When failover isn’t safe: Building high-availability PostgreSQL on Kubernetes (opens in new tab)

Datadog’s gameday testing exposed a PostgreSQL failure mode in which network latency caused replication lag to grow until no standby could be safely promoted. Although the clusters remained writable, they could not fail over without risking data loss, forcing operators to wait for connectivity and replicas to recover. Datadog’s solution was to redesign failover candidates around synchronous replication coordinated by Patroni, balancing stronger durability with acceptable write latency. ## The Zonal Failure That Exposed the Weakness - A simulated availability-zone failure introduced network latency in a staging environment. - Several Kubernetes-based PostgreSQL clusters had primary nodes in the affected zone. - Communication between primaries and replicas degraded, causing: - Rapidly increasing replication lag - Stalled writes - Applications serving stale data - No replica being current enough for safe promotion - The clusters prioritized continued writes over durability, leaving them writable but unable to fail over safely. ## Baseline PostgreSQL Architecture - Each cluster uses a single-writer design: - One active leader handles writes. - Two standby nodes are reserved for failover and do not serve application traffic. - A separate read-replica pool handles read-only traffic and scales independently. - Read replicas are intentionally excluded from failover candidates. - Patroni manages replication, leader elections, and failover. - ZooKeeper acts as Patroni’s distributed configuration store, tracking: - The current leader lock - Cluster configuration - Member replication state and latest LSN - ZooKeeper’s ephemeral leader key ensures that only one node can become primary. - During partitions, Patroni favors safety by pausing or demoting nodes that cannot verify cluster state. ## Why Failover Was Not Safe - Patroni checks replication lag before promoting a standby using `maximum_lag_on_failover`. - During the gameday, all eligible standbys exceeded that threshold. - Patroni correctly rejected promotion because each candidate could have been missing committed transactions. - The cluster therefore had no safe writable primary, even though the original leader was impaired. - The failure was a consequence of asynchronous replication and network latency, not a failure in Patroni’s safety mechanisms. ## Asynchronous Versus Synchronous Replication - **Asynchronous replication**, used originally: - Lets the leader commit and respond without waiting for replicas. - Provides low write latency and high throughput. - Can lose transactions committed on the leader but not yet copied to a standby. - **Synchronous replication**: - Requires the leader to receive acknowledgment from at least one replica before confirming a transaction. - Reduces the chance that a failover candidate is significantly behind. - Provides stronger durability, but may increase write latency when replicas experience network or availability problems. ## The Redesigned Approach - Datadog reworked its PostgreSQL deployment so failover candidates use synchronous replication. - Patroni coordinates these replicas and continues to enforce safe leader election. - The design aims to make failover both automatic and safe while limiting performance impact. - Benchmarking and failure testing were used to evaluate the trade-off between durability and latency. Datadog’s experience demonstrates that asynchronous replication can leave a system operational but unable to fail over during network disruption. For clusters where data durability and automatic recovery are critical, synchronous replication for designated failover candidates offers a safer architecture, provided its latency and availability costs are measured carefully.

postgresqlk8shigh-availabilitydatabase-replication+4

datadogMar 23, 2026

When upserts don't update but still write: Debugging Postgres performance at scale (opens in new tab)

The provided content does not include the tech blog post itself. It consists primarily of Datadog’s website navigation and a promotional link about its Gartner recognition, so there is not enough article content to summarize reliably. ## Available Information - Datadog was named a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms. - The page navigation lists Datadog products across: - Infrastructure and application monitoring - Database and log management - Security - Digital experience monitoring - Software delivery - Incident and service management - AI-powered observability tools - The URL path references `debugging-postgres-performance`, suggesting the intended article may concern PostgreSQL performance debugging, but the article text is not included. Please provide the blog post’s body or a working text extraction for a substantive summary.

postgresqlk8sdatadogobservability+2

datadogMar 23, 2026

When upserts don't update but still write: Debugging Postgres performance at scale (opens in new tab)

Datadog needed to track when ephemeral hosts were last seen so inactive hosts could be deleted after seven days. A seemingly inexpensive PostgreSQL upsert caused disk writes to double and WAL syncs to quadruple, despite most operations not changing any data. Investigating the WAL revealed that conflict-handling upserts still lock conflicting rows and generate WAL activity, consuming the database’s limited write capacity. ## Tracking Host Activity Efficiently - Hosts stop reporting telemetry when they terminate, but Datadog has no direct termination signal. - Hosts inactive for seven days can be safely removed from the metadata store. - Updating the main host table on every observation would be too expensive because: - Large data centers generate more than 25,000 observations per second. - PostgreSQL MVCC creates a new row version for every update. - Updating the main table would rewrite all host metadata. - Datadog created a separate `host_last_ingested` table containing: - `host_id` as the primary key - `last_ingested` with a default timestamp - The table used `fillfactor=80` to leave page space for future updates. - No index was created on `last_ingested`, allowing updates to use Heap-Only Tuples (HOT) and avoid additional index writes. - Because only daily freshness was required, the timestamp needed to change at most once per day. ## The Conditional Upsert The initial query inserted a host if it did not exist and otherwise updated `last_ingested` only when the previous value was more than a day old: ```sql INSERT INTO host_last_ingested AS t VALUES ($1, now()) ON CONFLICT (host_id) DO UPDATE SET last_ingested = EXCLUDED.last_ingested WHERE t.last_ingested < EXCLUDED.last_ingested - '1 day'::interval; ``` - New hosts produced an insert. - Recently seen hosts matched the conflict but were expected to be no-ops because of the `WHERE` clause. - The team therefore expected most queries to avoid meaningful writes. ## Unexpected Disk and WAL Activity - During a gradual rollout at roughly 500 upserts per second: - Insertions initially increased as expected. - Actual updates remained mostly flat. - Write IOPS more than doubled. - WAL syncs increased by approximately the same amount. - This showed that the absence of an applied update did not mean the query was free. - Since PostgreSQL must flush WAL records at transaction commit, additional WAL activity directly increased disk pressure. - A PostgreSQL cluster’s single-writer design makes this write budget particularly important. ## Inspecting PostgreSQL WAL - PostgreSQL records database changes in its Write-Ahead Log, including table changes, index modifications, and related transaction activity. - The team used the `pg_walinspect` extension, available starting in PostgreSQL 15: ```sql CREATE EXTENSION pg_walinspect; ``` - Its `pg_get_wal_records_info` function allows inspection of WAL records between two Log Sequence Numbers (LSNs). - Examining the WAL helped explain why the conditional upsert generated writes even when the `WHERE` condition prevented the row update. - The underlying issue was that `ON CONFLICT DO UPDATE` still locks the conflicting row, and that locking activity is recorded in the WAL. The key lesson is that a PostgreSQL upsert that reports zero processed rows is not necessarily a true no-op. Conditional conflict updates can still create substantial WAL and locking overhead, so WAL inspection is essential when database write metrics do not match apparent update volume.

postgresqldatabase-designupsertwal+4

datadogMar 9, 2026

When an AI agent came knocking: Catching malicious contributions in Datadog’s open source repos (opens in new tab)

Datadog describes how AI-powered attackers targeted its open-source repositories through malicious issues, pull requests, and comments. The campaign, attributed to the “hackerbot-claw” agent, focused on weaknesses in GitHub Actions and LLM-powered workflows. Datadog’s LLM-based review system and layered CI security controls detected the activity and helped limit its impact, while prompting further hardening. ## Why Open-Source Repositories Attract Attackers - Public repositories are attractive targets because automated CI/CD pipelines often build and execute code from external contributions. - Common attack techniques include: - Injecting user-controlled values, such as PR titles, into workflow scripts. - Using indirect poisoned pipeline execution to introduce malicious dependencies or build instructions. - Abusing `pull_request_target` workflows, which may run untrusted code with elevated permissions. - Prompt-injecting LLM-powered GitHub Actions used for issue triage, labeling, or code assistance. - Attackers may also disguise malicious changes through: - Large or obfuscated diffs. - Invisible Unicode characters. - Malicious libraries. - Imposter commits that resemble legitimate dependency references. ## Datadog’s LLM-Based Contribution Detection - Datadog receives dozens of external PRs each week across projects such as the Agent, tracers, SDKs, Vector, chaos-controller, and Stratus Red Team. - Its BewAIre system monitors GitHub events and selects security-relevant activity, including PRs and pushes. - BewAIre: - Extracts, normalizes, and enriches code diffs. - Sends them through a two-stage LLM pipeline. - Classifies changes as benign or malicious. - Produces a structured explanation for each verdict. - Malicious verdicts are forwarded to Datadog Cloud SIEM, where detection rules create enriched signals for the Security Incident Response Team to investigate. ## Hardening CI and Development Workflows - Datadog reduces the potential impact of successful attacks through multiple preventive controls: - Its `dd-octo-sts-action` generates minimally scoped, short-lived GitHub credentials using OIDC. - Long-lived and overly broad personal access tokens and GitHub Apps are being replaced. - Unused GitHub Actions secrets are identified and removed across thousands of repositories. - Organization-wide controls enforce branch protection, mandatory PR approval, commit signing, and lower-privilege default `GITHUB_TOKEN` permissions. - Engineers are provided with documented best practices and secure “golden paths” for CI development. ## The Hackerbot-Claw Campaign - Modern AI models are increasingly capable of offensive security tasks, especially when given tools, feedback loops, and autonomy. - StepSecurity reported an AI agent attacking open-source CI systems on March 1. - Between February 27 and March 2, the actor: - Opened 16 pull requests. - Created two issues and eight comments. - Targeted nine repositories across six organizations. - The activity was later linked to the hackerbot-claw agent, whose GitHub account was removed. - Datadog’s investigation began after BewAIre alerted the team to a suspicious contribution in the newly public `datadog-iac-scanner` repository on February 27. ## Practical Takeaway Organizations that accept public contributions should combine automated, AI-assisted review with least-privilege credentials, strict workflow permissions, secret management, mandatory approvals, and human incident response. Detection alone is insufficient; CI pipelines should be designed so that a malicious contribution has limited access and minimal opportunity to compromise secrets or production systems.

postgresqldatabase-designllmci-cd+4

datadogNov 4, 2025

Replication redefined: How we built a low-latency, multi-tenant data replication platform (opens in new tab)

Datadog built a managed, multi-tenant data replication platform to move data reliably across thousands of services without brittle, point-to-point integrations. The effort began by separating analytical search workloads from a shared PostgreSQL database, then evolved into automated pipeline provisioning with Temporal. The platform favors asynchronous replication to improve scalability and resilience, accepting limited replication lag in exchange for lower application latency and reduced operational coupling. ## Scaling Search Beyond PostgreSQL - A shared PostgreSQL database initially provided low-latency access, ACID guarantees, and low operational cost. - As data volumes grew, complex joins and aggregations became increasingly slow. - Datadog’s Metrics Summary page had to join: - 82,000 active metrics - 817,000 metric configurations - Page latency reached approximately 7 seconds at p90, while repeated facet changes generated additional expensive queries. - Index and disk bloat, memory pressure, VACUUM and ANALYZE overhead, and rising I/O wait further reduced throughput. - Rather than continuing to optimize PostgreSQL for analytical search, Datadog moved search and aggregation workloads to a dedicated search platform. - Data was denormalized during replication, producing document-oriented indexes better suited to faceted search. - The resulting system reduced page-load times by as much as 97%—from roughly 30 seconds to 1 second—while maintaining about 500 ms of replication lag. ## Automating Pipeline Provisioning with Temporal Provisioning a replication pipeline required coordinating multiple systems and configuration steps: - Enabling PostgreSQL logical replication with `wal_level`. - Creating users and assigning replication permissions. - Configuring publishers and replication slots. - Deploying Debezium instances to capture PostgreSQL changes. - Creating Kafka topics and mapping them to Debezium instances. - Adding heartbeat tables to monitor replication and prevent excessive WAL retention. - Configuring sink connectors to write Kafka data into the search platform. Manual management became increasingly difficult across many pipelines and data centers. Datadog used Temporal workflows to split provisioning into modular, repeatable tasks and combine them into higher-level orchestrations. This reduced errors, improved consistency, and allowed engineers to create and modify pipelines without repeating complex operational procedures. ## Choosing Asynchronous Replication - Synchronous replication provides strong consistency by waiting for replicas to acknowledge each write. - However, it increases latency and operational complexity, particularly across distributed environments. - Asynchronous replication allows the primary system to acknowledge writes immediately while replicas catch up afterward. - Datadog selected the asynchronous model because it decouples application performance from network latency and replica availability. - The trade-off is temporary replication lag during failures or periods of pressure, but the model offers better scalability and resilience for high-throughput systems. Datadog’s experience suggests that replication should be treated as a managed platform rather than a collection of custom integrations. Separating workloads, automating provisioning, and choosing asynchronous delivery can improve performance and reliability while reducing the operational burden on individual engineering teams.

postgresqldatabase-designkafkaelasticsearch+4

datadogJun 17, 2025

Breaking up a monolith: How we’re unwinding a shared database at scale (opens in new tab)

Datadog is moving away from a large shared relational database because its benefits eventually give way to coordination costs, schema fragility, noisy-neighbor problems, and scaling limits. Splitting the database is difficult and expensive, but platform investments in service development and managed Postgres can make independently owned databases practical. The key is to establish functional boundaries, provide safe cross-domain access, and automate migrations. ## Why Shared Databases Persist - Shared databases reduce operational overhead for small or fast-moving organizations. - A single database enables simple, low-latency joins across all data. - Workload isolation and access management often matter less when systems are small. - Because the cost of splitting a database is high, organizations commonly keep the shared model longer than they should. ## Signs It Is Time to Split the Database - Data grows beyond the capacity of one machine, or replication becomes too slow. - Noisy-neighbor effects make performance unpredictable. - Schema changes by one team unexpectedly affect others. - Security requirements such as access-control lists are difficult to enforce. - These issues create engineering costs, incidents, and degraded user experiences across teams. ## What Database Decomposition Requires - Identify functional ownership boundaries. - Build services for cross-domain queries where necessary. - Require consumers to use those services instead of querying another domain’s tables directly. - Provision new database instances. - Migrate data and traffic carefully from the shared database to the new instances. Datadog had previously split off large portions of its database into only a few separate databases. The experience showed that finding boundaries, enforcing them, and migrating without incidents is difficult and highly manual. ## Why Teams Resist Leaving Shared Infrastructure - Building a service may jeopardize existing product goals. - Operating a service can introduce significant maintenance and on-call work. - Cross-domain data access may be unclear or cause unacceptable latency or user impact. - Owning a database creates additional operational responsibility. - Migrations are often handcrafted, risky, and difficult to repeat. - Forcing the transition can cost more than tolerating the existing problems and create organizational resistance. ## Platform Investments That Enable Change Datadog addressed these obstacles through two major initiatives: - **Rapid:** An opinionated framework for building and operating API and gRPC services. - **OrgStore:** A managed platform for Postgres databases. Rapid reduces the cost of creating and maintaining services by providing shared configuration, common data-access patterns, and operational support. OrgStore reduces the burden of owning separate database instances. Together, these platforms make it more attractive for new projects to avoid the legacy shared database and allow existing domains to migrate incrementally. The broader lesson is that database decomposition becomes realistic when platform engineering makes service ownership, database operations, cross-domain access, and migrations safe enough to fit into normal product development.

postgresqldatabase-designobservabilitygrpc+4

datadogJan 15, 2025

Unraveling a Postgres segfault that uncovered an Arm64 JIT compiler bug | Datadog (opens in new tab)

The provided content does not include the blog post itself; it contains Datadog’s navigation menu and a link titled “Unraveling a Postgres Segfault.” The only substantive claim shown is that Datadog was named a Leader in the 2026 Gartner® Magic Quadrant™ for Observability Platforms. ### Datadog’s Observability Offering - The navigation lists products for: - Infrastructure and Kubernetes monitoring - Application performance monitoring and profiling - Database, log, and data observability - Security and cloud protection - Digital experience monitoring - CI/CD and software delivery - Incident response and service management - AI-agent and GPU observability - It also highlights platform capabilities such as dashboards, alerts, workflow automation, access control, and governance. ### Missing Post Content - No discussion of the PostgreSQL segmentation fault is included. - The supplied text does not describe the failure’s cause, investigation process, technical diagnosis, or resolution. Please provide the article body or a complete excerpt for an accurate technical summary.

postgresqlcdebuggingsegmentation-fault+1

datadogJan 15, 2025

Unraveling a Postgres segfault that uncovered an Arm64 JIT compiler bug (opens in new tab)

Postgres was crashing with segmentation faults when executing certain expensive queries on an Arm64 Kubernetes cluster. Investigators reduced the failure to a simple table scan and discovered that disabling JIT compilation prevented the crash. Assembly-level debugging ultimately traced the problem to a bug in LLVM’s Arm64 JIT support. ## Isolating the Crash - The failures occurred across multiple EC2 nodes, ruling out faulty hardware. - Query logs showed that the crashes consistently followed a small number of query patterns. - The simplest reproducer was: ```sql SELECT repo_id FROM repository; ``` - Core dumps had badly corrupted stacks, but surviving frames pointed to `ExecRunCompiledExpr`, suggesting a failure during JIT execution. - The unusually short backtraces reinforced the suspicion that the stack itself had been corrupted. ## How PostgreSQL JIT Works - PostgreSQL normally evaluates SQL expressions through a general-purpose interpreter. - JIT compilation converts expressions such as `1+1` into native machine code, reducing interpreter overhead for large workloads. - JIT can also optimize tuple deforming by converting disk tuples into in-memory values more efficiently. - PostgreSQL uses LLVM to generate the compiled code. - Because compilation adds overhead and compiled functions are not reused between queries, PostgreSQL enables JIT primarily for expensive queries based on cost thresholds. ## The Query of Death - The affected query scanned a partitioned `repository` table with: - 64 partitions - More than 1.6 million rows - 128 JIT-generated functions - Its query plan enabled expression compilation and tuple deforming: ```text JIT: Functions: 128 Expressions: true Deforming: true Inlining: false Optimization: false ``` - Running the query with: ```sql SET jit = off; ``` completed successfully. - Disabling JIT cluster-wide immediately stopped the crashes without noticeable query-latency effects. ## Root Cause Direction - The release build of PostgreSQL offered limited debugging flexibility, so the team planned to reproduce the failure in a dedicated test environment. - Further investigation eventually isolated the defect to JIT compilation on Arm64 systems. - The underlying issue was identified as an LLVM bug rather than a PostgreSQL query or hardware problem. - The investigation continued down to generated assembly and resulted in an upstream fix. The immediate mitigation was to disable PostgreSQL JIT, while the durable solution was to adopt the LLVM fix addressing the Arm64 code-generation bug.

postgresqldatabase-designk8scloud-computing+3

datadogMay 21, 2024

Engineering VP spotlight: Ivo Dimitrov (opens in new tab)

Ivo Dimitrov’s career evolved from low-level systems programming into engineering leadership focused on large-scale distributed storage. His experience at Microsoft and LinkedIn shaped his approach to building scalable data platforms, while Datadog attracted him with its talented people, modern technology, and culture of experimentation. Today, he leads Datadog’s Distributed Data Systems organization, supporting the company’s metrics, events, query, alerting, and analytics infrastructure. ## From Electrical Engineering to Systems Programming - Dimitrov initially studied electrical engineering and became interested in software while working on digital control systems. - His early work included contributing to a real-time operating system kernel. - He spent roughly a decade as an individual contributor, specializing in: - High-performance systems - Low-level programming - C and C++ - System software ## Transition from Individual Contributor to Manager - At Microsoft, Dimitrov worked on an early version of Azure Blob Storage. - Following a reorganization, he accepted an opportunity to lead his team despite having no prior management experience. - Microsoft supported the transition through: - Leadership mentorship - Formal management training - Guidance on communication, conflict resolution, and interpersonal leadership - He discovered that management allowed him to expand his ownership beyond individual projects and influence broader organizational outcomes. - The role combined his technical background with responsibilities such as cross-functional coordination, team development, and engineering strategy. ## Building Internet-Scale Storage at Microsoft and LinkedIn - At Microsoft, Dimitrov worked on storage systems supporting Hotmail. - After joining LinkedIn in 2014, he adapted to a technology environment centered on open source tools such as MySQL and Java. - He led development of Espresso, LinkedIn’s proprietary key-value storage platform. - The platform matured into a core system supporting approximately 95 percent of LinkedIn’s data sets. - He also helped oversee several other large-scale storage projects: - **Venice**, an open source platform for serving derived data - **Ambry**, an open source blob storage system - **Helix**, an open source cluster manager - These systems supported critical parts of LinkedIn’s internet-scale infrastructure. ## Why Datadog Was Appealing - Dimitrov was drawn to Datadog by three main factors: - Highly capable engineers and leaders - Interesting, modern technology - The opportunity to contribute to a rapidly growing company - Compared with the legacy systems and processes that had accumulated at LinkedIn, Datadog offered less bureaucracy and more freedom to: - Take thoughtful risks - Experiment - Deliver quickly - Fail fast and learn - Iterate and innovate - He was particularly interested in Datadog’s Kubernetes-based Metrics and Events platforms and the challenge of building best-in-class infrastructure during the company’s growth. ## Distributed Data Systems at Datadog - Dimitrov leads the Distributed Data Systems organization, which owns a portfolio of storage and data technologies. - Its responsibilities include: - **Metrics**, supporting metrics and time-series data - **Events**, handling semi-structured data such as logs, profiles, and traces - **Driveline**, a main-memory database optimized for online analytics - The **Cross-Platform Queries** team provides a unified query interface across systems that historically exposed separate, domain-specific APIs. - This reduces the learning curve for engineers and customers. - It abstracts the underlying data stores behind a common API. - The organization also operates Datadog’s Alerts platform, which generates a large share of the queries sent to the Metrics and Events systems. Dimitrov’s experience demonstrates how deep systems expertise can translate into effective engineering leadership. His recommendation by example is to remain technically engaged while expanding one’s scope—from writing individual components to shaping teams, platforms, and long-term engineering direction.

postgresqldatabase-designk8sdistributed-systems+4

datadogDec 21, 2023

Scaling self-serve analytics: The tools empowering 5,000 employees (opens in new tab)

Datadog scaled self-serve analytics from 200 to 5,000 employees by building an open-source-based platform around three pillars: trusted data, accessible tools, and organizational knowledge. The goal is to let employees answer routine questions and make informed decisions without relying on a centralized Data & Analytics team. This approach combines a single source of truth, self-service data pipelines and transformations, data discovery, quality monitoring, and training. ## The Purpose of Self-Serve Analytics - Datadog’s mission is to “empower everyone at Datadog to make data-informed decisions on their own.” - Self-service allows Data & Analytics teams to focus on higher-value initiatives instead of handling every request. - The organization identified three primary user profiles: - **Analytics Explorers:** Need discoverable data and ready-made reports. - **Analytics Builders:** Create reports and run advanced queries. - **Analytics Experts:** Expose new data, maintain business logic, and manage quality. ## Data as a Single Source of Truth - Datadog centralizes product, operational, and business data so consumers work from the same version of reality. - Its “Bring Your Own Data” (BYOD) tool lets teams expose their own data for analytics. - The shared data layer supports BI tools, notebooks, data discovery, programmatic access, and machine-learning models. - Trust depends on: - Consistent naming and modeling conventions. - Comprehensive documentation. - Continuous data-quality monitoring. ## Self-Serve Data Intake - Teams can connect internal and third-party data sources through integrations and BYOD. - The platform provides scheduling and a user interface for exposing or requesting datasets. - Pipeline observability covers: - Pipeline execution. - Data quality. - Actionable alerts when failures occur. ## Self-Serve Transformation - Analysts manage their departments’ business logic using SQL and dbt. - The development environment integrates with workflow management, metadata, and pipeline-run systems. - Enforced conventions keep the shared modeling layer consistent and understandable as more analysts contribute. - Analysts can inspect lineage, pipeline runs, quality checks, and alerts. ## Data Discovery and Metadata - Every employee can browse datasets and fields in the central data platform. - Search capabilities help users identify which data can answer a particular question. - Metadata explains: - The dataset’s origin and owner. - Definitions and intended meaning. - Where the data is used. - Sensitivity and reliability. - This context helps employees determine whether data is both relevant and trustworthy. ## Supporting Adoption - Tools alone are insufficient; Datadog also provides data knowledge, support, and training. - The Data & Analytics organization acknowledges that self-service has limits and works to mitigate risks such as misunderstanding data or applying incorrect business logic. - Success is tracked through adoption and the effectiveness of the overall self-service strategy. Datadog’s experience suggests that self-serve analytics scales best when data is treated as a product: centralized, documented, observable, and accessible through tools designed for users with different levels of expertise.

postgresqlsqldbtopen-source-tools+4

datadogMay 12, 2022

How Datadog's IT team automated account inactivity and SaaS spend management (opens in new tab)

Datadog expanded its Clarity auditing tool into Clarity License Manager (CLM), a system that tracks SaaS usage, reduces licensing costs, and improves security. CLM identifies inactive accounts, notifies employees, automatically deactivates unused access, and restores it quickly when needed. Its microservice architecture and application-specific adapters allow the system to scale across many SaaS products. ## The SaaS License Management Problem - Datadog used many commercial SaaS tools with substantial per-user costs. - License usage data was outdated and collected through quarterly manual audits. - IT Support had to contact employees individually, creating administrative overhead and a poor user experience. - Unused accounts also created security risks, including stale credentials that could be compromised. ## Goals of Clarity License Manager - Monitor and automatically deactivate inactive accounts, especially in sensitive services such as cloud providers. - Reduce the risk of leaked or abused stale credentials. - Limit the potential impact of security incidents. - Lower SaaS spending and support data-driven licensing decisions. - Preserve employee productivity through an easy account restoration process. ## Usage Monitoring and Automated Workflows - CLM gathers activity data through: - Direct integrations with individual SaaS APIs. - Google Workspace SAML audit logs for indirect integrations. - Employee activity is stored per application in an Amazon RDS-backed PostgreSQL database. - Employees receive email and Slack notifications after a configurable period of inactivity, with 90 days as the default. - Notifications explain the specific login or application action required to remain active. - If the employee does not respond after multiple reminders, CLM deactivates the account automatically. - Employees can reopen access by submitting a Freshservice ticket. - Accounts are restored within seconds, including their previous roles and permissions. ## Microservice Architecture - CLM consists of Python microservices running on AWS Lambda. - The services share a central PostgreSQL database. - Microservices provide: - Easier scaling as Datadog adds more SaaS applications. - Greater resilience and flexibility. - A modular foundation for future development. - The architecture introduced complexity because services required different APIs and libraries with overlapping functionality. ## Application-Specific Adapters - Each SaaS product is represented by an adapter shared across CLM microservices. - Adapters isolate application-specific API logic from the core workflows. - A typical adapter supports operations such as: - Retrieving users. - Fetching login activity. - Activating and deactivating accounts. - Onboarding and offboarding users. - This design provides: - Clear separation of responsibilities. - Reusable and flexible integration code. - Simpler microservices that do not need to handle each application’s unique behavior. CLM demonstrates how automated usage monitoring can simultaneously improve SaaS security, reduce unnecessary spending, and minimize disruption for employees. A modular adapter-based architecture is particularly useful when managing a growing portfolio of third-party applications.

postgresqldatabase-designpythonmicroservices+3

datadogApr 13, 2022

It's always DNS . . . except when it's not: A deep dive through gRPC, Kubernetes, and AWS networking (opens in new tab)

A routine update to a critical metrics query service caused intermittent errors and increased latency. Although logs initially pointed to DNS failures, the investigation revealed a deeper networking problem involving dropped packets and saturated AWS VPC connection tracking. The incident highlighted how Kubernetes, Cilium, AWS networking, and DNS behavior can interact in ways that obscure the true cause. ## Initial Symptoms and Apparent DNS Failures - Errors increased whenever the metrics query service was rolled out. - The service retrieves data from metric stores for dashboards and monitor evaluations. - Automatic retries reduced user-facing failures but increased latency. - Service logs showed DNS errors when connecting to dependencies inside Kubernetes. - The investigation therefore began with the cluster’s DNS infrastructure. ## NodeLocal DNSCache Reaches Its Limits - NodeLocal DNSCache runs as a `node-local-dns` DaemonSet on every Kubernetes node. - DNS pods had: - A 64 MB memory limit - A `max_concurrent` limit of 1,000 requests - The pods experienced out-of-memory errors and rejected requests during rollouts. - Increasing memory to 256 MB stopped the OOM errors, but DNS failures continued. - Request volume was far below the expected capacity: - Normally about 400 queries per second - Nearly 2,000 queries per second during rollouts - Expected capacity of at least 200,000 queries per second - Upstream resolvers were marked unhealthy, suggesting that NodeLocal DNSCache could not establish or maintain connections. - Because upstream requests could wait up to five seconds, connection failures consumed concurrency slots and made the cache appear overloaded. ## Evidence of a Network Problem - The instances were below their 5-Gbps sustained throughput limits. - TCP retransmits increased in correlation with service rollouts. - Engineers suspected brief traffic spikes, or microbursts, that were not visible in aggregate throughput metrics. - This shifted the investigation from DNS configuration toward lower-level AWS networking behavior. ## AWS VPC Connection Tracking - ENA metrics revealed a significant increase in `conntrack_allowance_exceeded`. - This metric counts packets dropped when VPC connection tracking becomes saturated. - Connection tracking maintains state for network flows and supports features such as stateful EC2 security groups. - The infrastructure used two tracking layers: - VPC conntrack maintained at the hypervisor level - Linux conntrack inside each instance - VPC conntrack appeared saturated even though Linux conntrack contained fewer than 60,000 entries—well within the observed capacity of similar instances. - AWS Support confirmed that conntrack capacity varies by instance type and that VPC conntrack limits could differ substantially from Linux conntrack limits. - Scaling to larger instances resolved the symptoms, but the engineers wanted to understand the traffic pattern and find a more efficient long-term solution. ## VPC Flow Logs as the Next Investigation Tool - The team turned to Amazon VPC Flow Logs to examine the service’s low-level network behavior. - These logs were expected to clarify why connection tracking filled up and how rollout traffic contributed to the saturation. - The investigation was still ongoing at the point where the provided article excerpt ends.

postgresqldatabase-designk8saws+4

datadogApr 2, 2019

Building highly reliable data pipelines at Datadog (opens in new tab)

Datadog’s approach to reliable data pipelines focuses on delivering correct data on time, even when individual jobs fail. Reliability therefore requires fault tolerance, monitoring, and fast recovery rather than eliminating every failure. The company achieves this through isolated, short-lived clusters and pipelines designed to limit the impact of failures. ## Reliability Means Timely, Correct Results - A reliable pipeline is one that consistently produces correct outputs within the required time window. - Occasional crashes do not necessarily make a pipeline unreliable if automatic recovery still delivers the data on schedule. - Pipelines should be designed with the expectation that failures will eventually occur. - Monitoring must detect unexpected failures early, while operational processes should support rapid recovery. ## Architecture for Batch Pipelines - Datadog streams and analyzes live data in real time but uses batch pipelines for features such as optimized long-term storage. - Historical data is stored in object storage. - Cloud Hadoop/Spark services launch and configure processing clusters. - Luigi workers manage tasks and workflows, while Spark workers compile code and submit jobs. - Jobs can be launched through a web interface, command line, or scheduler. ## One Cluster per Pipeline Instead of placing all workloads on one large Hadoop cluster, Datadog gives each pipeline its own cluster. - **Isolation:** Jobs do not compete for resources or interfere with one another, simplifying monitoring and diagnosis. - **Workload-specific hardware:** Clusters can use CPU-optimized or memory-optimized instances depending on the job. - **Elastic scaling:** Clusters can be expanded to catch up with delays or handle growing data volumes without waiting for a shared cluster. - **Safer upgrades:** Hadoop and Spark versions can be upgraded gradually across separate clusters. - Clusters are typically short-lived, averaging about three hours, although dozens may run simultaneously. ## Using Spot Instances to Encourage Fault Tolerance - AWS spot instances can reduce infrastructure costs by as much as 80%, but their nodes may be terminated whenever capacity or demand changes. - Rather than avoiding this failure mode, Datadog designs pipelines to tolerate disappearing clusters. - Long-running jobs are risky because failures discard more work and make recovery slower. - Pipelines are split into smaller jobs: - **Vertically:** Separate transformations into multiple stages, persisting intermediate results in S3. - **Horizontally:** Partition input data so multiple jobs process different portions concurrently. ## Breaking Up the Rollup Pipeline - Datadog’s rollup pipeline generates aggregated time-series data for historical metrics queries. - A single job would take more than 14 hours, making failures costly and difficult to recover from. - The pipeline is divided into two stages: - Aggregate high-resolution data and checkpoint it to S3 as Parquet files. - Convert the intermediate data into a custom format optimized for queries. - As these jobs grew, they were partitioned further using Kafka’s partitioning scheme. - Kafka partitions are grouped into shards, allowing Datadog to: - Adjust how much data each job processes. - Run more or fewer jobs as needed. - Isolate unusually large or sensitive shards. - This decomposition adds overhead because launching jobs and checkpointing to S3 take extra time, but it substantially limits the work lost during failures. ## Practical Recommendation Design pipelines around failure rather than assuming uninterrupted execution. Use isolated, scalable clusters, short jobs, intermediate checkpoints, and partitioned processing so that failures affect only a small portion of the workload and recovery remains fast.

postgresqldatabase-designawsmonitoring+4

datadogAug 23, 2017

Scaling support with Vagrant and Terraform (opens in new tab)

Datadog’s Solutions Team uses reproducible virtual environments to investigate customer issues across diverse operating systems, kernels, and integrations. Vagrant simplifies local VM creation, while provisioning scripts eliminate repeated installation and configuration work. Terraform extends the same approach to shared AWS environments, enabling teams to provision, preserve, and collaborate on sandboxes quickly. ## Reproducing Customer Environments with Vagrant - Containers are useful, but virtual machines are better when reproducing specific operating systems, kernels, orchestrators, or complex infrastructure. - Vagrant provides a simple workflow: - `vagrant init` - `vagrant up` - `vagrant ssh` - The main challenge is not creating a VM, but installing and configuring the technologies needed to match a customer’s environment. - With more than 200 integrations, engineers cannot be experts in every technology they may need to troubleshoot. ## Standardizing Setup with Provisioning Scripts - Vagrant provisioning supports tools such as Chef, Puppet, Ansible, and ordinary shell scripts. - Datadog stores reusable reproduction environments in a shared GitHub repository. - Each sandbox includes: - A `Vagrantfile` - A `setup.sh` provisioning script - A `data` directory for configuration files and supporting scripts - A `README.md` with usage information - Engineer-specific values, such as hostnames and tags, are kept in a local `.sandbox.conf.sh` file. - Once a sandbox exists, an engineer can run `vagrant up` and begin reproducing the customer issue within minutes. - The directory hierarchy organizes sandboxes by operating system, version or provider, and technology—for example, Ubuntu Xenial with Kafka. ## Sharing Remote Environments with Terraform - Terraform provides similar infrastructure management for remote cloud instances, including AWS EC2. - The team reuses the same `setup.sh` and `data` files for both Vagrant and Terraform, avoiding duplicate configuration work. - Each sandbox adds a `.tf` file that: - Creates an EC2 instance - Copies required data files - Executes the provisioning script remotely - A shared Terraform module handles common infrastructure tasks, while a `tf.example` file helps engineers create new configurations. - This preserves the same repository structure and workflow while extending sandboxes from local VMs to remote environments. ## Benefits for Team Collaboration - Remote sandboxes can remain available without consuming engineers’ local RAM. - Proper network security allows teammates to access and share environments. - Engineers can reproduce previously configured integrations during live customer interactions. - Investigations can continue across time zones, allowing teams to hand off urgent issues without rebuilding the environment. The overall recommendation is to treat reproduction environments as reusable infrastructure: encode installation and configuration steps once, store them in version control, and use Vagrant for local testing and Terraform for persistent, shared cloud sandboxes.

postgresqldockerterraformvirtual-machines+4