Inside the feature store powering real-time AI in Dropbox Dash (opens in new tab)
Dropbox Dash’s ranking system depends on a hybrid feature store that can combine real-time user behavior with large-scale historical data. Because Dropbox operates across on-premises and cloud environments, and because each query can trigger thousands of feature lookups, off-the-shelf systems could not meet its latency, scale, and integration requirements. The resulting architecture uses Feast for orchestration, Spark for computation, Dynovault for low-latency storage, and a custom Go serving layer, achieving roughly 25–35 ms p95 latency while keeping features fresh. ## Goals and Requirements - Dash ranks documents, images, and conversations using behavioral, contextual, and real-time signals. - A single query can fan out into thousands of feature lookups across many candidate files. - The feature store needed to: - Support sub-100 ms search latency. - Reflect user actions within seconds or minutes. - Bridge Dropbox’s on-premises services and Spark-based cloud infrastructure. - Handle both streaming-style updates and batch computations. - Let engineers develop features without managing serving and orchestration details. ## Choosing a Hybrid Architecture - Dropbox evaluated Feast, Hopsworks, Featureform, Feathr, Databricks, and Tecton. - Feast was selected because: - It separates feature definitions from infrastructure concerns. - Engineers can focus on PySpark transformations. - Its modular adapter system supports existing Dropbox infrastructure. - Feast’s DynamoDB adapter enabled integration with Dynovault, Dropbox’s DynamoDB-compatible storage system. - The architecture combines: - Feast for orchestration and serving APIs. - Spark jobs for feature computation and ingestion. - Cloud storage for offline indexing and data management. - Dynovault for online, low-latency lookups. - A custom Go service replacing Feast’s Python online serving path. - Dynovault is colocated with inference workloads and provides approximately 20 ms client-side latency. - Monitoring covers job failures, feature freshness, and data lineage. ## Replacing Python with Go for Low Latency - The initial Feast-based Python service struggled under heavy concurrency. - CPU-bound JSON parsing and Python’s Global Interpreter Lock became bottlenecks. - A multi-process design helped temporarily but introduced coordination overhead. - The serving layer was rewritten in Go using: - Lightweight goroutines. - Shared memory. - Faster JSON parsing. - The Go service now handles thousands of requests per second. - It adds only about 5–10 ms beyond Dynovault latency and achieves roughly 25–35 ms p95 latency. ## Keeping Features Fresh - Fresh signals are essential for ranking quality; actions such as opening a document should influence subsequent searches quickly. - Fully real-time computation is impractical for features requiring large joins, aggregations, and historical context. - Dropbox therefore built a three-part ingestion strategy. - Batch ingestion handles complex, high-volume transformations using a medallion architecture. - Intelligent change detection updates only modified records rather than rewriting every feature. - This reduced online-store writes from hundreds of millions to fewer than one million per run and significantly shortened update time. ## Practical Takeaway The system demonstrates that a feature store does not need to be entirely off-the-shelf or entirely real-time. Combining a modular framework with custom serving, colocated storage, batch optimization, and freshness monitoring allowed Dropbox to meet demanding latency and scale requirements while keeping feature development manageable.