Timeseries indexing at scale
Datadog’s metrics volume grew 30× from 2017 to 2022, while customers began running increasingly complex queries. This growth exposed limitations in the Timeseries Index service, whose original indexing approach became a performance and maintenance bottleneck. The post introduces Datadog’s metrics architecture and explains how its indexing strategy evolved to handle large-scale workloads more reliably. ## Metrics Platform Architecture - **Intake** - Datadog Agents send data points through a load balancer to metrics intake. - Each point contains a metric name, timestamp, numerical value, and optional tags. - Tags such as `env`, `host`, and `service` provide dimensions for filtering, aggregation, and comparison. - Data is written to Kafka, allowing multiple consumers to process it for storage, indexing, analysis, and archiving. - **Storage** - The short-term storage layer has two services: - The Timeseries Database stores tuples of `<timeseries_id, timestamp, float64>`. - The Timeseries Index stores `<timeseries_id, tags>` mappings. - The custom Timeseries Index database is built on RocksDB and supports filtering and grouping during queries. - **Query Processing** - The distributed query layer contacts index nodes, retrieves intermediate results from the timeseries database, and combines them. - Filters such as `env:prod AND service:event-consumer` restrict results to matching data points. - Grouping by tags, such as `service`, produces separate timeseries for each group. - Aggregators such as `avg` combine values within each group. ## Why Timeseries Indexing Matters - Indexes prevent queries from scanning every timeseries associated with a metric, much like database indexes avoid full table scans. - Poorly designed or insufficient indexes can make queries slow and consume excessive CPU and memory. - As Datadog’s data volume and query complexity increased, the indexing system became a critical scalability concern. ## Automatically Generated Indexes - The original system generated indexes from live query behavior. - Slow or resource-intensive queries were recorded in a query log and analyzed periodically. - Index selection considered: - Query frequency - Execution time - Number of input timeseries identifiers scanned - Number of output identifiers returned - Highly selective queries—with a high input-to-output ratio—received indexes. - Obsolete indexes that no longer received queries were removed. - These indexes acted as materialized views, replacing expensive scans with efficient key-value lookups. ## Original Indexing Service Design - The service was written in Go and used embedded SQLite and RocksDB databases. - SQLite stored metadata, including: - Index definitions - Query logs - Query counts and timestamps - Input and output cardinalities - Query durations - Index definitions were read frequently, updated rarely, and cached entirely in memory. - Query logs were bulk-written in the background, keeping them out of the ingestion and query paths. - SQLite’s SQL interface made the metadata easy to inspect and modify manually. - RocksDB handled the high-volume write workload required to index trillions of events per day. Datadog’s experience shows that indexing strategies that work at smaller scale can become bottlenecks as data volume and query sophistication grow. Effective timeseries systems therefore need adaptive indexing, careful separation of query and ingestion workloads, and storage technologies suited to extremely high write rates.
Read original(opens in new tab)