cloudflareFeb 3, 2026

Improve global upload performance with R2 Local Uploads (opens in new tab)

R2 Local Uploads improves global upload performance by first writing object data near the client, then asynchronously copying it to the bucket’s region. Objects become immediately available and remain strongly consistent during replication. Cloudflare reports up to a 75% reduction in upload request duration for cross-region uploads. ## Faster Global Uploads - Local Uploads targets `PutObject` and `UploadPart` requests made far from the bucket’s location. - Synthetic tests showed median upload TTLB dropping from about 2 seconds to 500 milliseconds. - Tests used 5 MB objects uploaded from Western North America to an Asia-Pacific bucket at roughly 20 requests per second. - The feature is available in open beta and can be enabled in the Cloudflare Dashboard or with: ```bash npx wrangler r2 bucket local-uploads enable [BUCKET] ``` ## The Cross-Region Distance Problem - R2 requests enter through a globally distributed Gateway Worker, which handles authentication and routing. - Object metadata is managed by a distributed Durable Object Metadata Service. - Encrypted object data is stored in R2’s distributed storage infrastructure. - Without Local Uploads, streamed data must travel to the bucket’s region before the upload can complete. - Long-distance transfers can increase latency and introduce upload variability or reliability issues. ## How Local Uploads Works - If the client and bucket are in the same region, R2 uses its normal storage flow. - If they are in different regions: - Data is initially written to storage near the client. - Metadata is published in the bucket’s region. - The object becomes readable as soon as the local write completes. - Background replication later copies the data to the bucket’s primary region. - There is no read-unavailability window while replication is in progress. - Local Uploads is unavailable for jurisdiction-restricted buckets, including EU and FedRAMP buckets. ## When to Use It - Applications have users or devices distributed across multiple regions. - Upload speed and reliability are important. - You want faster writes without moving the bucket’s primary location. - R2’s Metrics page can help identify regional request patterns through the “Request Distribution by Region” graph. ## Replication Architecture - R2 represents the background copy operation as a replication task. - Cloudflare Queues process these tasks asynchronously. - Queues provide: - Rate control for replication. - Automatic retries. - Dead-letter queue support for failures. - Sharding across multiple queues for each storage region. - When publishing object metadata, R2 atomically: - Stores the object metadata. - Creates a pending-replica key describing unfinished replication work. - Creates a timestamp-based replication marker that determines when the task enters a queue. - The pending-replica record includes the replication plan, source and destination locations, mode, priority, and whether the source can be deleted after successful replication. Local Uploads is a strong fit for globally distributed upload-heavy workloads. Enable it when cross-region write latency matters, while keeping in mind the restriction on jurisdiction-constrained buckets.

asynchronous-replicationcloudflare-workersdurable-objectscloudflare-r2+4

datadogNov 4, 2025

Replication redefined: How we built a low-latency, multi-tenant data replication platform (opens in new tab)

Datadog built a managed, multi-tenant data replication platform to move data reliably across thousands of services without brittle, point-to-point integrations. The effort began by separating analytical search workloads from a shared PostgreSQL database, then evolved into automated pipeline provisioning with Temporal. The platform favors asynchronous replication to improve scalability and resilience, accepting limited replication lag in exchange for lower application latency and reduced operational coupling. ## Scaling Search Beyond PostgreSQL - A shared PostgreSQL database initially provided low-latency access, ACID guarantees, and low operational cost. - As data volumes grew, complex joins and aggregations became increasingly slow. - Datadog’s Metrics Summary page had to join: - 82,000 active metrics - 817,000 metric configurations - Page latency reached approximately 7 seconds at p90, while repeated facet changes generated additional expensive queries. - Index and disk bloat, memory pressure, VACUUM and ANALYZE overhead, and rising I/O wait further reduced throughput. - Rather than continuing to optimize PostgreSQL for analytical search, Datadog moved search and aggregation workloads to a dedicated search platform. - Data was denormalized during replication, producing document-oriented indexes better suited to faceted search. - The resulting system reduced page-load times by as much as 97%—from roughly 30 seconds to 1 second—while maintaining about 500 ms of replication lag. ## Automating Pipeline Provisioning with Temporal Provisioning a replication pipeline required coordinating multiple systems and configuration steps: - Enabling PostgreSQL logical replication with `wal_level`. - Creating users and assigning replication permissions. - Configuring publishers and replication slots. - Deploying Debezium instances to capture PostgreSQL changes. - Creating Kafka topics and mapping them to Debezium instances. - Adding heartbeat tables to monitor replication and prevent excessive WAL retention. - Configuring sink connectors to write Kafka data into the search platform. Manual management became increasingly difficult across many pipelines and data centers. Datadog used Temporal workflows to split provisioning into modular, repeatable tasks and combine them into higher-level orchestrations. This reduced errors, improved consistency, and allowed engineers to create and modify pipelines without repeating complex operational procedures. ## Choosing Asynchronous Replication - Synchronous replication provides strong consistency by waiting for replicas to acknowledge each write. - However, it increases latency and operational complexity, particularly across distributed environments. - Asynchronous replication allows the primary system to acknowledge writes immediately while replicas catch up afterward. - Datadog selected the asynchronous model because it decouples application performance from network latency and replica availability. - The trade-off is temporary replication lag during failures or periods of pressure, but the model offers better scalability and resilience for high-throughput systems. Datadog’s experience suggests that replication should be treated as a managed platform rather than a collection of custom integrations. Separating workloads, automating provisioning, and choosing asynchronous delivery can improve performance and reliability while reducing the operational burden on individual engineering teams.

asynchronous-replicationdatabase-designpostgresqlkafka+4