metaJul 1, 2026

Meta’s AI Storage Blueprint at Scale (opens in new tab)

Meta argues that AI progress increasingly depends on storage that can deliver massive datasets with predictable, low latency. Traditional BLOB-storage designs optimized for durable, cost-efficient HDD storage create metadata and proxying bottlenecks that stall GPUs and slow research. Meta is therefore rebuilding its storage foundation around unified metadata, direct client-to-storage access, and regional deployments colocated with GPUs. ## Storage Architecture and AI’s Growing Demands - Meta operates hundreds of exabyte-scale storage clusters supporting products such as Facebook, Instagram, Meta AI, Ads, and internal databases. - Its storage APIs are built on Tectonic, a horizontally scalable block layer providing: - High durability and availability through erasure coding - HDD and flash tiering - Placement of hot, warm, and cold data - Multi-tenant regional storage - BLOB-storage layers built on Tectonic provide globally scalable object storage and configurable durability/availability policies. - Meta’s training systems historically used an NFS-like filesystem interface over Tectonic, but are increasingly moving to BLOB storage for unified access to massive data lakes and higher performance. ## Why Storage Latency Limits GPU Utilization - AI workloads require bursty and sustained high throughput with predictable worst-case latency. - Training runs use hundreds of thousands of GPUs processing data in batches and periodically synchronizing state. - A single slow GPU can delay synchronization and extend the completion time for every GPU. - Data loaders prefetch future batches while GPUs process current ones, but high-latency storage reads can still create GPU stalls. - These stalls directly increase training costs and extend time to market. ## Problems with the Legacy BLOB Architecture - The older service-oriented design accumulated multiple stateful layers, each with its own metadata store. - A single `getObject("/bucket/path")` request could require lookups across the namelayer, volumeslayer, and containerlayer. - Cross-region metadata requests could add hundreds of milliseconds, and one slow lookup could delay the entire operation. - The architecture’s original assumptions no longer matched AI requirements: - **Latency:** AI needs bounded pMax latency, not merely acceptable average performance. - **Reliability:** AI requires high availability, but does not always need global replication by default. - **Cost:** Flash is necessary for AI-level IOPS, making storage cost-per-byte less important. - **Power:** Power used by storage competes directly with power available for GPUs. ## Rebuilding the Storage Foundation Meta redesigned the system around three major changes: - **Unified metadata schema** - Metadata from separate layers was consolidated into a flat schema backed by ZippyDB. - Path resolution can now use O(1) lookups to map objects to `(blockId, offset, size)` locations. - **Direct data access** - The dataplane proxy was removed. - A “fat client” SDK streams data directly from Tectonic storage servers. - This reduces latency, increases throughput, and lowers storage power consumption. - **Regional deployment** - The BLOB stack can operate regionally or globally. - Regional instances are colocated with GPUs in AI regions, reducing cross-region access. With the new flow, the SDK requests a read plan from the API server, which performs the metadata lookup and returns storage locations. The SDK’s embedded Tectonic BlockClient then reads directly from the underlying blocks, adding essentially no extra dataplane overhead. The redesigned architecture is intended to improve GPU utilization, reduce latency, and preserve power for computation. The provided excerpt ends as Meta begins discussing how it handles workload spikes and hot spots during data and checkpoint loading.

object-storageai-trainingdistributed-storageerasure-coding+3

cloudflareFeb 3, 2026

Improve global upload performance with R2 Local Uploads (opens in new tab)

R2 Local Uploads improves global upload performance by first writing object data near the client, then asynchronously copying it to the bucket’s region. Objects become immediately available and remain strongly consistent during replication. Cloudflare reports up to a 75% reduction in upload request duration for cross-region uploads. ## Faster Global Uploads - Local Uploads targets `PutObject` and `UploadPart` requests made far from the bucket’s location. - Synthetic tests showed median upload TTLB dropping from about 2 seconds to 500 milliseconds. - Tests used 5 MB objects uploaded from Western North America to an Asia-Pacific bucket at roughly 20 requests per second. - The feature is available in open beta and can be enabled in the Cloudflare Dashboard or with: ```bash npx wrangler r2 bucket local-uploads enable [BUCKET] ``` ## The Cross-Region Distance Problem - R2 requests enter through a globally distributed Gateway Worker, which handles authentication and routing. - Object metadata is managed by a distributed Durable Object Metadata Service. - Encrypted object data is stored in R2’s distributed storage infrastructure. - Without Local Uploads, streamed data must travel to the bucket’s region before the upload can complete. - Long-distance transfers can increase latency and introduce upload variability or reliability issues. ## How Local Uploads Works - If the client and bucket are in the same region, R2 uses its normal storage flow. - If they are in different regions: - Data is initially written to storage near the client. - Metadata is published in the bucket’s region. - The object becomes readable as soon as the local write completes. - Background replication later copies the data to the bucket’s primary region. - There is no read-unavailability window while replication is in progress. - Local Uploads is unavailable for jurisdiction-restricted buckets, including EU and FedRAMP buckets. ## When to Use It - Applications have users or devices distributed across multiple regions. - Upload speed and reliability are important. - You want faster writes without moving the bucket’s primary location. - R2’s Metrics page can help identify regional request patterns through the “Request Distribution by Region” graph. ## Replication Architecture - R2 represents the background copy operation as a replication task. - Cloudflare Queues process these tasks asynchronously. - Queues provide: - Rate control for replication. - Automatic retries. - Dead-letter queue support for failures. - Sharding across multiple queues for each storage region. - When publishing object metadata, R2 atomically: - Stores the object metadata. - Creates a pending-replica key describing unfinished replication work. - Creates a timestamp-based replication marker that determines when the task enters a queue. - The pending-replica record includes the replication plan, source and destination locations, mode, priority, and whether the source can be deleted after successful replication. Local Uploads is a strong fit for globally distributed upload-heavy workloads. Enable it when cross-region write latency matters, while keeping in mind the restriction on jurisdiction-constrained buckets.

object-storagecloudflare-workersdurable-objectscloudflare-r2+4

datadogJan 29, 2025

Husky: Efficient compaction at Datadog scale (opens in new tab)

Husky is a distributed event store built on object storage for observability workloads reaching trillions of events per day. Because data is written continuously, rarely updated, and queried both recently and historically, its storage layer must minimize object-store fetches while still supporting high query parallelism. The central design challenge is choosing a compaction and layout strategy that keeps fragments manageable without sacrificing query speed. ## Husky’s Query Execution Model - Ingested events are grouped into files called **fragments** and stored in systems such as Amazon S3, Google Cloud Storage, or Azure Blob Storage. - Metadata for each fragment is stored separately in FoundationDB. - For each query: - Metadata is scanned to identify relevant fragments. - Fragments are distributed among query workers. - Workers scan their assigned data. - Results are merged. - Query cost depends mainly on: - The number of fragments fetched from object storage. - The number of events scanned within those fragments. - Husky therefore focuses on: - Reducing the total number of files through efficient compaction. - Organizing data so queries scan as few irrelevant events as possible. ## The Compaction “Goldilocks” Problem - Compaction combines many small fragments into a larger fragment containing the same data. - FoundationDB transactions atomically replace the old fragments with the compacted one, ensuring queries see a consistent state either before or after compaction. - Ingestion writers buffer events per tenant to avoid producing extremely small files, but they flush periodically to keep newly ingested data queryable quickly. - These flushed fragments may contain only a few thousand events, making queries inefficient when they must fetch thousands of objects and metadata records. ## Balancing Fragment Size Husky must find a fragment size that balances several competing concerns: - **Object storage and metadata overhead** - Fewer, larger fragments reduce the number of fetches and metadata entries. - **Compaction cost** - Larger or more aggressively reorganized fragments require more CPU and more object-storage GET and PUT operations. - **Query parallelism** - Smaller fragments allow more workers to operate concurrently. - Larger fragments reduce distribution overhead but can limit parallelism for large analytical queries. - **Scan efficiency** - Query workers use vectorized execution, which is most effective when scanning sufficiently large batches of rows. - **Data locality and compression** - Compaction can place events with similar timestamps or tags near one another. - This improves compression and allows queries to skip irrelevant data, but requires additional processing and query-pattern analysis. Fragments that are too small create excessive fetch and scheduling overhead. Fragments that are too large reduce parallelism and can make broad queries slower. The goal is a “just right” size suited to typical query patterns. ## Storage Layout and Query Selectivity - Husky organizes events along both: - The time dimension. - Spatial dimensions such as tags. - Keeping commonly queried events close together reduces the amount of data that must be scanned. - Similar data also compresses more effectively. - Achieving this layout increases compaction work, creating a tradeoff between lower query cost and lower maintenance cost. ## Scalable Compaction - Husky’s storage system depends on compaction being efficient enough to run continuously at very large scale. - The design must account not only for the final fragment size, but also for the CPU and object-storage costs required to produce it. - Atomic metadata updates ensure that compaction can occur without exposing partial results or inconsistent table states. Husky’s approach treats compaction as a core part of query performance rather than simple file maintenance. A practical design must tune fragment sizes, merge frequency, and data layout together to minimize total system cost while preserving fast access to both recent events and large historical datasets.

object-storagedatabase-designquery-optimizationfoundationdb+4