Meta’s AI Storage Blueprint at Scale (opens in new tab)
Meta argues that AI progress increasingly depends on storage that can deliver massive datasets with predictable, low latency. Traditional BLOB-storage designs optimized for durable, cost-efficient HDD storage create metadata and proxying bottlenecks that stall GPUs and slow research. Meta is therefore rebuilding its storage foundation around unified metadata, direct client-to-storage access, and regional deployments colocated with GPUs. ## Storage Architecture and AI’s Growing Demands - Meta operates hundreds of exabyte-scale storage clusters supporting products such as Facebook, Instagram, Meta AI, Ads, and internal databases. - Its storage APIs are built on Tectonic, a horizontally scalable block layer providing: - High durability and availability through erasure coding - HDD and flash tiering - Placement of hot, warm, and cold data - Multi-tenant regional storage - BLOB-storage layers built on Tectonic provide globally scalable object storage and configurable durability/availability policies. - Meta’s training systems historically used an NFS-like filesystem interface over Tectonic, but are increasingly moving to BLOB storage for unified access to massive data lakes and higher performance. ## Why Storage Latency Limits GPU Utilization - AI workloads require bursty and sustained high throughput with predictable worst-case latency. - Training runs use hundreds of thousands of GPUs processing data in batches and periodically synchronizing state. - A single slow GPU can delay synchronization and extend the completion time for every GPU. - Data loaders prefetch future batches while GPUs process current ones, but high-latency storage reads can still create GPU stalls. - These stalls directly increase training costs and extend time to market. ## Problems with the Legacy BLOB Architecture - The older service-oriented design accumulated multiple stateful layers, each with its own metadata store. - A single `getObject("/bucket/path")` request could require lookups across the namelayer, volumeslayer, and containerlayer. - Cross-region metadata requests could add hundreds of milliseconds, and one slow lookup could delay the entire operation. - The architecture’s original assumptions no longer matched AI requirements: - **Latency:** AI needs bounded pMax latency, not merely acceptable average performance. - **Reliability:** AI requires high availability, but does not always need global replication by default. - **Cost:** Flash is necessary for AI-level IOPS, making storage cost-per-byte less important. - **Power:** Power used by storage competes directly with power available for GPUs. ## Rebuilding the Storage Foundation Meta redesigned the system around three major changes: - **Unified metadata schema** - Metadata from separate layers was consolidated into a flat schema backed by ZippyDB. - Path resolution can now use O(1) lookups to map objects to `(blockId, offset, size)` locations. - **Direct data access** - The dataplane proxy was removed. - A “fat client” SDK streams data directly from Tectonic storage servers. - This reduces latency, increases throughput, and lowers storage power consumption. - **Regional deployment** - The BLOB stack can operate regionally or globally. - Regional instances are colocated with GPUs in AI regions, reducing cross-region access. With the new flow, the SDK requests a read plan from the API server, which performs the metadata lookup and returns storage locations. The SDK’s embedded Tectonic BlockClient then reads directly from the underlying blocks, adding essentially no extra dataplane overhead. The redesigned architecture is intended to improve GPU utilization, reduce latency, and preserve power for computation. The provided excerpt ends as Meta begins discussing how it handles workload spikes and hot spots during data and checkpoint loading.
