Evolving our real-time timeseries storage again: Built in Rust for performance at scale
Datadog built a sixth-generation real-time timeseries database in Rust to keep pace with rapidly growing metric volume, cardinality, and query complexity. The new engine is designed for high throughput and low latency, reportedly achieving 60× higher ingestion performance and 5× faster peak-scale queries. Its development reflects a long evolution from general-purpose databases toward a purpose-built system with tighter control over storage, I/O, and execution. ## Datadog’s Metrics Storage Architecture - The metrics platform includes ingestion, enrichment, real-time and long-term storage, querying, and alerting. - This post focuses on real-time storage, which is split into two independently deployed services: - **RTDB:** Stores raw metric tuples of `<timeseries_id, timestamp, value>`, performs aggregations, and serves recent data. - **Index database:** Stores metric identifiers and their tags as `<timeseries_id, tags>`. - A storage router distributes incoming metrics across RTDB nodes based on load. - The query service contacts the relevant RTDB and index nodes, retrieves results, and combines them. - Each RTDB node includes: - An ingestion subsystem - A storage engine - A durability snapshot module - A gRPC query layer - Throttlers for resource management - A shared control plane coordinating these components ## Generation 1: Cassandra - Cassandra provided strong write scalability and a familiar operational model. - It was influenced by systems such as OpenTSDB and HBase. - Its main weaknesses were: - Limited flexibility for real-time queries - Difficulty supporting complex alerting and analytical workloads - Inefficient retrieval of large datasets - These limitations prompted Datadog to move to Redis. ## Generation 2: Redis - Redis improved read performance and offered a flexible, easy-to-understand storage model. - Datadog avoided Redis’s built-in clustering for reliability reasons, requiring the team to operate many independent instances. - Important drawbacks included: - Single-threaded execution limiting snapshotting during live traffic - Severe but uncommon memory-management and threading failures - Serialization and cross-process communication overhead - Inefficient memory layout, disk I/O, and CPU usage at scale - Redis nevertheless provided valuable operational insight and clarified the need for a purpose-built engine with direct control over I/O and system resources. ## Generation 3: MDBM and Memory-Mapped I/O - MDBM provided a memory-mapped key-value store based on `mmap`. - The operating system’s page cache loaded database pages on demand, making disk-backed data behave similarly to in-memory structures. - This simplified storage interactions initially, but performance degraded as workloads intensified. - Memory-mapped I/O introduced subtle performance and correctness concerns, leading Datadog to conclude that explicit I/O management would scale better. ## Generation 4: A Go-Based B+ Tree - Datadog replaced MDBM with a custom B+ tree written in Go. - The engine supported a thread-per-core-oriented design, with Go’s scheduler providing a useful foundation. - This change significantly improved throughput and latency. - It also created a platform that could be optimized more aggressively for Datadog’s workload. ## Generation 5: DDSketch and RocksDB - Datadog introduced DDSketch to support distribution metrics and accurate percentile estimation. - The existing Go engine was optimized for scalar floating-point values and was difficult to extend for sketches. - RocksDB was therefore integrated to store DDSketch data, offering flexibility and strong performance. - Over time, maintaining separate storage technologies created pressure to build a unified engine capable of handling multiple metric types efficiently. ## The Move Toward a New Engine - The progression from Cassandra to Redis, MDBM, a custom Go B+ tree, and RocksDB shows a pattern of replacing general-purpose components as scale and workload diversity increased. - Each generation solved important problems but introduced new operational or architectural trade-offs. - Datadog ultimately needed a unified, purpose-built storage system with: - High-throughput ingestion - Low-latency queries - Better support for high-cardinality data - Efficient handling of different metric types - More direct control over concurrency, memory, and I/O - The sixth generation addresses these requirements through a Rust-based real-time timeseries database. Datadog’s experience suggests that general-purpose storage systems can be effective early on, but sustained growth eventually favors a specialized engine. The practical lesson is to optimize existing infrastructure first while developing a purpose-built replacement before scale and workload complexity make incremental fixes insufficient.
Read original(opens in new tab)