Foundationdb | Techlist.io

datadogJun 23, 20263 min readCurated summary

How we migrated a live routing system using AI-assisted refactoring

Stream Router evolved from a small configuration file into a critical control-plane service routing Datadog’s massive metrics workload. Its original FoundationDB key-value model eventually hit transaction-size and performance limits because relational relationships were reconstructed in application code. Datadog redesigned the system around PostgreSQL and DuckDB, using AI-assisted, test-driven refactoring to accelerate the migration without disrupting production traffic. ## Stream Router’s Role in Datadog’s Metrics Pipeline - Datadog processes more than a hundred trillion events per day. - Stream Router determines which Kafka cluster, topic, partitions, and sharding strategy should handle each datapoint. - It serves both producers and queriers but does not process Kafka messages itself. - Routing decisions change frequently as infrastructure evolves, making correctness and historical tracking essential. ## From Configuration File to Control Plane - In 2016, routing was managed through a small configuration file distributed to services. - As the platform grew, the file expanded to thousands of lines and required manual edits and rollouts. - Stream Router replaced this workflow with: - A centralized gRPC service - API-managed routes - Automated, gradual rollouts - The write path used FoundationDB, while the read path served static RocksDB snapshots restored into memory. - This eventually became a bottleneck as routing tables and operational changes grew larger. ## Why the Key-Value Model Stopped Scaling - Routes reference streams and sharding strategies, while rules reference routes. - These relationships are inherently relational and require cross-entity validation. - The KV implementation loaded tens of thousands of records into application processes and reconstructed database-like relationships in code. - Some operations exceeded FoundationDB transaction-size limits. - Moving to PostgreSQL without changing the access patterns would not solve the issue; certain operations were estimated to require 45 minutes because of thousands of sequential database round trips. - The fundamental problem was the data model and application logic, not simply the choice of database. ## Designing the New Storage Architecture - The team redesigned the schema manually before using AI tools. - The relational model introduced explicit foreign keys between: - Streams - Sharding strategies - Routes - Rules - PostgreSQL was selected for the write path because it provided the required relational semantics and transaction model. - DuckDB was selected for the read path because: - It is embeddable and suitable for snapshot-based serving - It supports array columns - Its SQL dialect is closely compatible with PostgreSQL - Shared query logic could therefore work across both storage engines. ## AI-Assisted Refactoring - Claude and Cursor were used to accelerate a systematic, test-driven migration. - For each method, developers supplied: - The old implementation - The new schema - A failing test - AI generated an initial implementation, while tests determined whether it was correct. - The models assisted with method-level refactoring rather than autonomously designing the architecture. - Human expertise remained central to schema design, migration strategy, and evaluating system-level risks. ## Foundations for a Safe Migration - The migration benefited from infrastructure already present at Datadog. - Stream Router’s storage layer was isolated behind an internal `Controller` interface. - This modularity helped contain storage changes and enabled incremental refactoring. - Existing tests and clear boundaries provided confidence in generated implementations while production traffic continued. The central lesson is that AI was most effective as an accelerator inside a disciplined engineering process. A well-designed relational schema, modular storage abstraction, and failing tests provided the safety mechanisms; AI helped implement the resulting changes faster, but did not replace human architectural judgment.

Read original

Foundationdb Database Design PostgreSQL Apache Kafka+4

datadogOct 1, 20253 min readCurated summary

Inside Husky’s query engine: Real-time access to 100 trillion events

Datadog’s Husky event store is designed not merely to store more than 100 trillion events, but to make them queryable interactively at massive scale. Its query engine handles schemaless data, highly variable tenant workloads, petabytes of object-store data, and both highly selective searches and broad analytics queries. The architecture combines distributed planning, metadata-driven pruning, coordinated execution, and reader-level optimizations to minimize the data and fragments that must be scanned. ## Husky’s Data and Query Workloads - Events contain a timestamp and flexible attributes. - Data shapes vary widely: - Logs, network events, and traces have different schemas. - Tenants may produce either a few large events or enormous numbers of small events. - Most queries fall into two categories: - **Needle-in-a-haystack searches**, such as finding a specific IP connection, error message, or trace. - **Analytics-style searches**, such as time series, grouped breakdowns, and distributions. - The engine must also support raw event retrieval and more complex queries such as joins. ## Distributed Query Path Husky divides query execution among four multi-tenant services deployed across regions and data centers. ### Query Planner - Serves as the main entry point for event-store queries. - Resolves query context, validates and throttles requests, and integrates with other Datadog data stores. - Applies optimizations and statistics to split queries into time-based steps. - Schedules those steps on orchestrators and merges their results into the final response. ### Query Orchestrator - Acts as the gateway to Husky’s stored data. - Fetches fragment metadata, including: - File paths and versions - Row counts - Timestamp boundaries - Zone maps for query matching - Dispatches only relevant fragments to reader nodes. - Uses zone-map pruning to reduce downstream work by up to 60% for structured events and about 30% on average. - Aggregates results after fragment processing, which can require more computation than query planning. ### Metadata Service - Provides an abstraction over FoundationDB clusters. - Preserves atomicity during operations such as compaction, preventing duplicate data from appearing in query results. - Separates FoundationDB implementation details from the rest of the query system. - Must work within FoundationDB’s five-second transaction limit. ### Reader Service - Receives a query and selected fragments, then returns results quickly. - Performs the direct scan and execution work over fragment data. - Contains multiple optimizations intended to keep queries interactive despite scanning data stored in blob storage at extreme scale. ## Minimizing Data Scans The reader service follows the principle that the fastest query is the one that avoids unnecessary work. - Scanning less data reduces both latency and storage costs. - Touching fewer fragments limits expensive object-store operations. - This is especially important because Husky stores millions of fragments daily and cannot afford multiple blob-storage GET requests for every fragment in every query. ## Row Groups and Reader Execution - Fragments can contain millions of rows, making fully in-memory processing risky and potentially causing memory pressure or failures. - To limit data retrieval, fragments are physically organized into **row groups**. - Row groups allow the reader to fetch only portions of a fragment needed for a query rather than loading the entire file. - The reader uses an iterator-based execution model inspired by the Volcano query-processing architecture. - The provided article ends as it begins explaining how this row-group layout supports efficient iterator-based query execution.

Read original

Foundationdb Database Design Data Architecture Husky+3

datadogJan 29, 20253 min readCurated summary

Husky: Efficient compaction at Datadog scale

Husky is a distributed event store built on object storage for observability workloads reaching trillions of events per day. Because data is written continuously, rarely updated, and queried both recently and historically, its storage layer must minimize object-store fetches while still supporting high query parallelism. The central design challenge is choosing a compaction and layout strategy that keeps fragments manageable without sacrificing query speed. ## Husky’s Query Execution Model - Ingested events are grouped into files called **fragments** and stored in systems such as Amazon S3, Google Cloud Storage, or Azure Blob Storage. - Metadata for each fragment is stored separately in FoundationDB. - For each query: - Metadata is scanned to identify relevant fragments. - Fragments are distributed among query workers. - Workers scan their assigned data. - Results are merged. - Query cost depends mainly on: - The number of fragments fetched from object storage. - The number of events scanned within those fragments. - Husky therefore focuses on: - Reducing the total number of files through efficient compaction. - Organizing data so queries scan as few irrelevant events as possible. ## The Compaction “Goldilocks” Problem - Compaction combines many small fragments into a larger fragment containing the same data. - FoundationDB transactions atomically replace the old fragments with the compacted one, ensuring queries see a consistent state either before or after compaction. - Ingestion writers buffer events per tenant to avoid producing extremely small files, but they flush periodically to keep newly ingested data queryable quickly. - These flushed fragments may contain only a few thousand events, making queries inefficient when they must fetch thousands of objects and metadata records. ## Balancing Fragment Size Husky must find a fragment size that balances several competing concerns: - **Object storage and metadata overhead** - Fewer, larger fragments reduce the number of fetches and metadata entries. - **Compaction cost** - Larger or more aggressively reorganized fragments require more CPU and more object-storage GET and PUT operations. - **Query parallelism** - Smaller fragments allow more workers to operate concurrently. - Larger fragments reduce distribution overhead but can limit parallelism for large analytical queries. - **Scan efficiency** - Query workers use vectorized execution, which is most effective when scanning sufficiently large batches of rows. - **Data locality and compression** - Compaction can place events with similar timestamps or tags near one another. - This improves compression and allows queries to skip irrelevant data, but requires additional processing and query-pattern analysis. Fragments that are too small create excessive fetch and scheduling overhead. Fragments that are too large reduce parallelism and can make broad queries slower. The goal is a “just right” size suited to typical query patterns. ## Storage Layout and Query Selectivity - Husky organizes events along both: - The time dimension. - Spatial dimensions such as tags. - Keeping commonly queried events close together reduces the amount of data that must be scanned. - Similar data also compresses more effectively. - Achieving this layout increases compaction work, creating a tradeoff between lower query cost and lower maintenance cost. ## Scalable Compaction - Husky’s storage system depends on compaction being efficient enough to run continuously at very large scale. - The design must account not only for the final fragment size, but also for the CPU and object-storage costs required to produce it. - Atomic metadata updates ensure that compaction can occur without exposing partial results or inconsistent table states. Husky’s approach treats compaction as a core part of query performance rather than simple file maintenance. A practical design must tune fragment sizes, merge frequency, and data layout together to minimize total system cost while preserving fast access to both recent events and large historical datasets.

Read original

Foundationdb Database Design Query Optimization Husky+4

datadogNov 20, 20243 min readCurated summary

How we use formal modeling, lightweight simulations, and chaos testing to design reliable distributed systems

Courier’s design illustrates why distributed systems need more than unit, integration, and chaos testing. Datadog combined formal modeling, lightweight simulation, and conventional testing to uncover system-level risks before implementation. The approach was especially important after the March 8, 2023 outage, which showed how locally reasonable decisions can produce severe global failures. ## Why Distributed Systems Require Additional Analysis - Distributed systems provide greater scale and availability, but introduce concurrency, coordination, and failure modes that are difficult to reason about intuitively. - Traditional tests operate at relatively low levels of detail and may miss high-level design flaws. - Formal models and simulations allow teams to evaluate system behavior during the design phase, before implementation choices become expensive to change. - Model checking exhaustively explores all states permitted by a design and verifies defined correctness properties. ## Formal Modeling and Lightweight Simulation - Formal modeling uses a high-level specification language to describe: - System components and their interactions - Allowed system states - Properties the system must satisfy - Lightweight simulation builds a replica that runs under controlled conditions to study statistical characteristics such as: - Latency - Cost - Scalability - Behavior under realistic workloads - Modeling verifies correctness but cannot fully assess performance-related concerns. - Neither technique validates the final implementation directly. - Keeping models and simulations synchronized with the production design adds maintenance overhead. - Datadog considered the additional effort worthwhile because Courier was foundational, needed strong reliability guarantees, and incorporated lessons from the 2023 outage. ## Courier’s Requirements Courier was created to replace a decade-old Redis-backed queuing system that had begun to face throughput, scaling, and durability limitations. Its main requirements were: - **Multi-tenancy:** Isolate teams and products so one tenant cannot significantly disrupt others. - **At-least-once delivery:** Messages must not be lost; they must be delivered and acknowledged or sent to a dead-letter queue. - **Graceful degradation and high availability:** Throughput should decline roughly linearly as compute capacity is lost, rather than collapsing entirely. - **Horizontal scalability:** Throughput should increase linearly as compute capacity is added. The graceful-degradation requirement directly addressed the March 8 outage, when lost compute capacity caused a disproportionate throughput failure. ## FoundationDB Sharding for Tenant Isolation - Courier uses multiple FoundationDB clusters. - Each tenant is assigned to a subset of clusters. - No two tenants share the exact same cluster subset. - The initial design used: - Eight FoundationDB clusters - Four clusters per tenant - A theoretical maximum of `8 choose 4 = 70` tenant assignments - If one tenant saturated or disabled its four clusters, other tenants would still retain access to at least 25% of the total cluster capacity. - This arrangement provided sufficient isolation for the intended workloads. ## Broker Layer and High Availability - A broker layer exposes gRPC APIs for: - Sending messages - Receiving messages - Deleting messages - Clients connect only to the brokers, which apply the tenant-sharding logic. - Brokers health-check FoundationDB clusters and remove unhealthy clusters from consideration. - Both brokers and FoundationDB clusters are deployed across three availability zones to improve resilience. Courier demonstrates that formal verification and simulation are valuable complements to implementation testing. For mission-critical distributed services, teams should validate both correctness and operational behavior early, while also using unit, integration, and chaos testing to verify the final system.

Read original

Foundationdb Distributed Systems Grpc High Availability+4