Inside Husky’s query engine: Real-time access to 100 trillion events
Datadog’s Husky event store is designed not merely to store more than 100 trillion events, but to make them queryable interactively at massive scale. Its query engine handles schemaless data, highly variable tenant workloads, petabytes of object-store data, and both highly selective searches and broad analytics queries. The architecture combines distributed planning, metadata-driven pruning, coordinated execution, and reader-level optimizations to minimize the data and fragments that must be scanned. ## Husky’s Data and Query Workloads - Events contain a timestamp and flexible attributes. - Data shapes vary widely: - Logs, network events, and traces have different schemas. - Tenants may produce either a few large events or enormous numbers of small events. - Most queries fall into two categories: - **Needle-in-a-haystack searches**, such as finding a specific IP connection, error message, or trace. - **Analytics-style searches**, such as time series, grouped breakdowns, and distributions. - The engine must also support raw event retrieval and more complex queries such as joins. ## Distributed Query Path Husky divides query execution among four multi-tenant services deployed across regions and data centers. ### Query Planner - Serves as the main entry point for event-store queries. - Resolves query context, validates and throttles requests, and integrates with other Datadog data stores. - Applies optimizations and statistics to split queries into time-based steps. - Schedules those steps on orchestrators and merges their results into the final response. ### Query Orchestrator - Acts as the gateway to Husky’s stored data. - Fetches fragment metadata, including: - File paths and versions - Row counts - Timestamp boundaries - Zone maps for query matching - Dispatches only relevant fragments to reader nodes. - Uses zone-map pruning to reduce downstream work by up to 60% for structured events and about 30% on average. - Aggregates results after fragment processing, which can require more computation than query planning. ### Metadata Service - Provides an abstraction over FoundationDB clusters. - Preserves atomicity during operations such as compaction, preventing duplicate data from appearing in query results. - Separates FoundationDB implementation details from the rest of the query system. - Must work within FoundationDB’s five-second transaction limit. ### Reader Service - Receives a query and selected fragments, then returns results quickly. - Performs the direct scan and execution work over fragment data. - Contains multiple optimizations intended to keep queries interactive despite scanning data stored in blob storage at extreme scale. ## Minimizing Data Scans The reader service follows the principle that the fastest query is the one that avoids unnecessary work. - Scanning less data reduces both latency and storage costs. - Touching fewer fragments limits expensive object-store operations. - This is especially important because Husky stores millions of fragments daily and cannot afford multiple blob-storage GET requests for every fragment in every query. ## Row Groups and Reader Execution - Fragments can contain millions of rows, making fully in-memory processing risky and potentially causing memory pressure or failures. - To limit data retrieval, fragments are physically organized into **row groups**. - Row groups allow the reader to fetch only portions of a fragment needed for a query rather than loading the entire file. - The reader uses an iterator-based execution model inspired by the Volcano query-processing architecture. - The provided article ends as it begins explaining how this row-group layout supports efficient iterator-based query execution.
Read original(opens in new tab)