cloudflare-queues

3 posts

cloudflare

Dogfooding at scale: migrating cdnjs to Cloudflare’s Developer Platform (opens in new tab)

cdnjs now runs entirely on Cloudflare’s Developer Platform after a migration intended to improve maintainability rather than performance. Despite the rise of bundlers and modern JavaScript tooling, cdnjs still serves about 9 billion requests per day because it is free, familiar, immutable, auditable, and widely used by both developers and AI coding assistants. The migration replaces a fragmented GCP, GitHub, VM, and Cloudflare setup with a unified architecture built around Workers, R2, Workflows, Queues, D1, KV, Cache, and Containers. ## cdnjs’s Scale and Continued Relevance - cdnjs serves roughly: - 108,000 requests per second - 9 billion requests per day - Traffic across more than 330 Cloudflare data centers - A 98.6% cache-hit rate - It is used by approximately 12% of websites and holds a 48.3% share of the JavaScript CDN market. - Its simple `<script>`-tag model remains popular because: - URLs and versions are consistent and immutable. - Libraries are available without accounts, API keys, or rate limits. - Files include Subresource Integrity hashes. - The project is open source and community-driven. - AI assistants frequently generate cdnjs URLs because they appear throughout years of tutorials, documentation, GitHub repositories, and Stack Overflow answers. ## Why the Existing Architecture Became a Problem - Cloudflare moved cdnjs file serving to Workers and KV in 2020, improving resilience and enabling pre-compressed Brotli and gzip assets. - The publishing pipeline remained on GCP because Cloudflare previously lacked suitable tools for: - Fetching large package archives - Running CPU-intensive processing - Coordinating multi-step jobs over hours - The old pipeline combined GCP Functions, Google Cloud Storage, Pub/Sub, a git-sync VM, GitHub, Workers KV, and a bare-metal origin. - New features and bug fixes required coordinating deployments across multiple platforms, while observability required manually stitching together unrelated logs. ## Problems with the Legacy Pipeline - **No shared tracing** - Package updates could pass through several systems without a common correlation ID. - Partial failures could leave KV updated while GitHub remained stale, with no alert indicating the divergence. - **Split-brain storage** - File content existed both in Workers KV and a GitHub repository. - Neither system was cleanly authoritative, making reconciliation difficult. - **Storage-driven orchestration** - GCP Cloud Functions triggered one another through object-created events. - Storage effectively acted as a message queue without dead-letter handling, backlog visibility, or reliable replay. - **Operational fragmentation** - npm polling required 26 separately deployed Cloud Functions, one for each alphabetic shard. - Health monitoring required checking all 26 deployments and their logs. - **An oversized GitHub repository** - The repository exceeded 1.1 TB of packed storage. - GitHub could no longer generate archive downloads reliably. - Cloning and forking became impractical. - A 274-entry `.gitignore` accumulated to exclude releases the pipeline could not reject properly. - **Security overhead** - Cloud Functions, a VM, container images, storage buckets, and service-account credentials all required patching, auditing, and protection. - Retiring these components reduced the attack surface and eliminated recently exposed vulnerabilities. ## The New Cloudflare-Based Architecture - The rebuilt system uses Cloudflare’s Developer Platform end to end. - **R2** becomes the single source of truth for file content. - It can store large assets that previously did not fit comfortably in KV, including source maps, large bundles, and font packages. - Its S3-compatible API makes the catalog accessible to external tools and mirrors. - The broader platform combines: - Workers for request handling - Workflows for orchestration - Queues for reliable asynchronous processing - R2 for durable object storage - D1, KV, Workers Cache, and Containers for supporting services - Centralizing the pipeline should make processing state observable, reduce deployment complexity, and eliminate inconsistencies between edge storage and the GitHub repository. ## Practical Conclusion The cdnjs migration demonstrates that a globally critical, high-volume open-source service can evolve from a collection of legacy systems into a unified serverless platform. Its continued value comes not only from speed, but from being free, predictable, immutable, and easy for both humans and automated tools to consume.

cloudflare

Unlocking the Cloudflare app ecosystem with OAuth for all (opens in new tab)

Cloudflare opened self-managed OAuth to all customers so developers can build SaaS integrations, internal platforms, CI/CD workflows, and agentic tools without relying on difficult-to-manage API tokens. The expansion required improvements to permissions, consent, revocation, and phishing protections, as well as a major upgrade to the Hydra-based OAuth engine. Cloudflare used staged migrations, custom database changes, token-handling safeguards, and queued revocations to minimize disruption and preserve users’ security controls. ## Why Cloudflare Expanded OAuth Access - Previously, third-party OAuth integrations were limited to manually approved partners. - Other developers had to use API tokens, which are less convenient and poorly suited to delegated access. - Self-managed OAuth lets customers: - Request narrowly scoped permissions. - Give users clearer consent controls. - Revoke application access from the dashboard. - Build integrations and agentic tools using standard OAuth flows. - Cloudflare improved consent screens to identify the requesting application and its permissions, while making application ownership more visible to reduce phishing risks. ## Planning the Hydra Upgrade - Cloudflare used Hydra, an open-source OAuth engine, but its older deployment could not support the platform’s growing scale and new use cases. - The upgrade was split into two stages: - First, move to the latest 1.X release. - Then, perform the larger 2.X migration. - The 1.X database migrations created operational risks: - Standard index creation could take exclusive locks on critical tables. - Schema changes added columns and moved data between tables. - Hydra’s SDK used `SELECT *`, creating deserialization problems after schema changes. - Cloudflare rewrote migrations to use `CREATE INDEX CONCURRENTLY` and built a custom Hydra version that selected explicit columns. ## Designing a Blue-Green Migration - An in-place 2.X upgrade was rejected because of the volume of schema changes. - A blue-green deployment was chosen, but the migration would take several hours. - Disabling writes would prevent new authorizations and revocations, leaving users unable to manage application access during the upgrade. - Instead, Cloudflare kept writes enabled while reducing the amount of data that could be lost during the cutover: - Token expiry times were temporarily extended to multiple hours, reducing refresh-token writes. - Revocation events were written to Cloudflare Queues. - After switching to the green database, queued revocations could be replayed. - Preserving revocations was essential to prevent applications that users had disabled from regaining access. ## Lessons from the 1.X Upgrade - The custom migrations completed faster than expected without user impact. - A hard cutover was necessary because the old Hydra version could not read tokens created by the new version. - The new version introduced stricter refresh-token invalidation: - Reusing a refresh token invalidated the entire access and refresh-token chain. - This caused problems for high-volume clients such as Wrangler and MCP clients. - Cloudflare added refresh-token coalescing in the Worker routing layer: - Briefly caching requests allowed retries to be served without triggering invalidation. - Hydra 2.X provides a configurable refresh-token grace period, offering a more direct solution for safe retries. ## Executing the 2.X Upgrade - Cloudflare prepared a blue-green migration to avoid several hours of customer-facing downtime. - The strategy depended on reducing token writes, recording all revocations externally, switching databases, and replaying queued events afterward. - The provided article ends while beginning the detailed discussion of the 2.X execution. Cloudflare’s approach demonstrates that opening a security-sensitive platform to broad OAuth usage requires more than exposing an authorization endpoint. Safe adoption depends on explicit permissions, transparent consent, reliable revocation, backward-compatible token behavior, and migration plans that protect users even during infrastructure cutovers.

cloudflare

Improve global upload performance with R2 Local Uploads (opens in new tab)

R2 Local Uploads improves global upload performance by first writing object data near the client, then asynchronously copying it to the bucket’s region. Objects become immediately available and remain strongly consistent during replication. Cloudflare reports up to a 75% reduction in upload request duration for cross-region uploads. ## Faster Global Uploads - Local Uploads targets `PutObject` and `UploadPart` requests made far from the bucket’s location. - Synthetic tests showed median upload TTLB dropping from about 2 seconds to 500 milliseconds. - Tests used 5 MB objects uploaded from Western North America to an Asia-Pacific bucket at roughly 20 requests per second. - The feature is available in open beta and can be enabled in the Cloudflare Dashboard or with: ```bash npx wrangler r2 bucket local-uploads enable [BUCKET] ``` ## The Cross-Region Distance Problem - R2 requests enter through a globally distributed Gateway Worker, which handles authentication and routing. - Object metadata is managed by a distributed Durable Object Metadata Service. - Encrypted object data is stored in R2’s distributed storage infrastructure. - Without Local Uploads, streamed data must travel to the bucket’s region before the upload can complete. - Long-distance transfers can increase latency and introduce upload variability or reliability issues. ## How Local Uploads Works - If the client and bucket are in the same region, R2 uses its normal storage flow. - If they are in different regions: - Data is initially written to storage near the client. - Metadata is published in the bucket’s region. - The object becomes readable as soon as the local write completes. - Background replication later copies the data to the bucket’s primary region. - There is no read-unavailability window while replication is in progress. - Local Uploads is unavailable for jurisdiction-restricted buckets, including EU and FedRAMP buckets. ## When to Use It - Applications have users or devices distributed across multiple regions. - Upload speed and reliability are important. - You want faster writes without moving the bucket’s primary location. - R2’s Metrics page can help identify regional request patterns through the “Request Distribution by Region” graph. ## Replication Architecture - R2 represents the background copy operation as a replication task. - Cloudflare Queues process these tasks asynchronously. - Queues provide: - Rate control for replication. - Automatic retries. - Dead-letter queue support for failures. - Sharding across multiple queues for each storage region. - When publishing object metadata, R2 atomically: - Stores the object metadata. - Creates a pending-replica key describing unfinished replication work. - Creates a timestamp-based replication marker that determines when the task enters a queue. - The pending-replica record includes the replication plan, source and destination locations, mode, priority, and whether the source can be deleted after successful replication. Local Uploads is a strong fit for globally distributed upload-heavy workloads. Enable it when cross-region write latency matters, while keeping in mind the restriction on jurisdiction-constrained buckets.