cloudflare-workflows

3 posts

cloudflare

Run CI/CD for millions of repos — on your platform, on Cloudflare (opens in new tab)

Cloudflare is bringing code storage, CI, and deployment together on its platform. Its CI SDK turns Cloudflare Workflows into TypeScript-defined pipelines that can build, test, and deploy repositories stored in Artifacts. The approach supports both platform-managed CI for customer applications and custom workflows, while adding isolated execution, caching, parallelism, and optional AI-powered self-healing. ## Cloudflare-Hosted CI/CD - Artifacts provides versioned code storage capable of supporting millions of repositories. - Artifact push events can directly trigger Workflow executions through a new `events` configuration field. - A CI job can: - Build code in an isolated environment - Run linters, typechecks, and unit tests - Cache dependencies between steps - Automatically fix failed steps with an AI review agent - Deploy only after successful validation ## CI/CD as a Cloudflare Workflow - A traditional CI/CD pipeline is essentially an ordered sequence of Workflow steps. - Instead of complex YAML configuration, developers can define pipelines in TypeScript using `step.do()`. - The CI SDK combines Workflows with the Sandbox SDK to run commands safely and independently. - Workflow retries and timeouts manage state and failures without requiring developers to call the Sandbox API directly. - Push-triggered jobs no longer require separately configuring event subscriptions, queues, and consumers. ## Dependency Caching and Parallel Execution - An initial install step can download dependencies and tools such as bundlers, linters, and test runners. - Cache inputs can include files such as `package.json` and `bun.lock`. - The resulting sandbox snapshot is stored in an R2 bucket and reused by later steps. - Build, lint, test, and typecheck steps can run concurrently with `Promise.all()`. - A deploy step runs only after all required checks complete successfully. ## Platform-Managed and Custom CI - Platforms can define one reusable CI/CD pipeline for applications created by their customers. - Platform-owned code and customer code can use different pipelines while remaining in the same namespace. - Customers who need specialized behavior can define their own Workflow and run custom CI on their repository. - Both managed and custom CI pipelines can operate simultaneously. ## Extensible and Self-Healing Pipelines - Developers can import `CIWorkflow` from `@cloudflare/ci` and define their own workflow. - Each build or validation step runs in a separate isolated sandbox. - Workflows can invoke an AI agent when a build fails. - The agent can diagnose issues, apply a fix, and push a commit for approval. - Cloudflare provides a self-healing CI example through Project Think. Cloudflare’s recommended model is to treat CI/CD as ordinary TypeScript Workflow code: install dependencies once, run checks in parallel, and deploy only after success. This gives platforms a reusable default pipeline while preserving the flexibility for individual teams or customers to customize their own builds.

cloudflare

Dogfooding at scale: migrating cdnjs to Cloudflare’s Developer Platform (opens in new tab)

cdnjs now runs entirely on Cloudflare’s Developer Platform after a migration intended to improve maintainability rather than performance. Despite the rise of bundlers and modern JavaScript tooling, cdnjs still serves about 9 billion requests per day because it is free, familiar, immutable, auditable, and widely used by both developers and AI coding assistants. The migration replaces a fragmented GCP, GitHub, VM, and Cloudflare setup with a unified architecture built around Workers, R2, Workflows, Queues, D1, KV, Cache, and Containers. ## cdnjs’s Scale and Continued Relevance - cdnjs serves roughly: - 108,000 requests per second - 9 billion requests per day - Traffic across more than 330 Cloudflare data centers - A 98.6% cache-hit rate - It is used by approximately 12% of websites and holds a 48.3% share of the JavaScript CDN market. - Its simple `<script>`-tag model remains popular because: - URLs and versions are consistent and immutable. - Libraries are available without accounts, API keys, or rate limits. - Files include Subresource Integrity hashes. - The project is open source and community-driven. - AI assistants frequently generate cdnjs URLs because they appear throughout years of tutorials, documentation, GitHub repositories, and Stack Overflow answers. ## Why the Existing Architecture Became a Problem - Cloudflare moved cdnjs file serving to Workers and KV in 2020, improving resilience and enabling pre-compressed Brotli and gzip assets. - The publishing pipeline remained on GCP because Cloudflare previously lacked suitable tools for: - Fetching large package archives - Running CPU-intensive processing - Coordinating multi-step jobs over hours - The old pipeline combined GCP Functions, Google Cloud Storage, Pub/Sub, a git-sync VM, GitHub, Workers KV, and a bare-metal origin. - New features and bug fixes required coordinating deployments across multiple platforms, while observability required manually stitching together unrelated logs. ## Problems with the Legacy Pipeline - **No shared tracing** - Package updates could pass through several systems without a common correlation ID. - Partial failures could leave KV updated while GitHub remained stale, with no alert indicating the divergence. - **Split-brain storage** - File content existed both in Workers KV and a GitHub repository. - Neither system was cleanly authoritative, making reconciliation difficult. - **Storage-driven orchestration** - GCP Cloud Functions triggered one another through object-created events. - Storage effectively acted as a message queue without dead-letter handling, backlog visibility, or reliable replay. - **Operational fragmentation** - npm polling required 26 separately deployed Cloud Functions, one for each alphabetic shard. - Health monitoring required checking all 26 deployments and their logs. - **An oversized GitHub repository** - The repository exceeded 1.1 TB of packed storage. - GitHub could no longer generate archive downloads reliably. - Cloning and forking became impractical. - A 274-entry `.gitignore` accumulated to exclude releases the pipeline could not reject properly. - **Security overhead** - Cloud Functions, a VM, container images, storage buckets, and service-account credentials all required patching, auditing, and protection. - Retiring these components reduced the attack surface and eliminated recently exposed vulnerabilities. ## The New Cloudflare-Based Architecture - The rebuilt system uses Cloudflare’s Developer Platform end to end. - **R2** becomes the single source of truth for file content. - It can store large assets that previously did not fit comfortably in KV, including source maps, large bundles, and font packages. - Its S3-compatible API makes the catalog accessible to external tools and mirrors. - The broader platform combines: - Workers for request handling - Workflows for orchestration - Queues for reliable asynchronous processing - R2 for durable object storage - D1, KV, Workers Cache, and Containers for supporting services - Centralizing the pipeline should make processing state observable, reduce deployment complexity, and eliminate inconsistencies between edge storage and the GitHub repository. ## Practical Conclusion The cdnjs migration demonstrates that a globally critical, high-volume open-source service can evolve from a collection of legacy systems into a unified serverless platform. Its continued value comes not only from speed, but from being free, predictable, immutable, and easy for both humans and automated tools to consume.

cloudflare

How we built saga rollbacks for Cloudflare Workflows (opens in new tab)

Cloudflare Workflows now supports saga rollbacks, letting each durable step declare how to compensate for its side effects if a later operation fails. This addresses partial failures in multi-step processes, such as refunding a debit when a subsequent credit cannot complete. Rollbacks execute in reverse order and preserve Workflow durability, while requiring the same idempotency safeguards as normal steps. ## The Saga Problem - Durable Workflows can retry steps and persist state, but completed external operations cannot always be directly undone. - In a bank transfer: - Bank A debits the sender. - Bank B fails to credit the recipient. - The original debit must be reversed with a new credit operation. - The pairing of a forward action and its semantic compensation is known as the saga pattern. ## Manual Compensation Before Rollbacks - Developers had to track which steps completed and write centralized `try`/`catch` logic. - Compensation had to: - Run only for completed operations. - Execute in reverse order. - Continue even if one rollback fails. - Remain durable and retryable. - This approach becomes increasingly complex as workflows gain more steps. ## Rollback Functions on `step.do()` - Rollback logic is now declared directly in the step’s options: ```js await step.do("debit-bank-a", debitFn, { rollback: async ({ output }) => refundFn(output.id), }); ``` - Each step carries its own undo operation, making compensation easier to maintain. - Rollbacks can use the original step output, such as a payment or transaction ID. - If a later step fails, previously registered rollback handlers run automatically in reverse step-start order. ## Idempotency and Partial Failures - Rollback functions must be idempotent because they may be retried. - External operations should use idempotency keys to prevent duplicate refunds, credits, or inventory releases. - A step that fails may still need compensation: - It could have modified an external system before failing. - The operation may have succeeded even though Workflows never received its result. - Rollback handlers must therefore handle `output === undefined`. - If user code catches an error and the Workflow continues, rollback does not immediately start. However, if the Workflow later fails, previously registered handlers can still run. ## Practical Usage - Developers pass an options object with a `rollback` function as the final argument to `step.do()`. - Rollbacks can reverse payments, release resources, or perform other compensating actions. - This removes the need for growing manual catch blocks and explicit rollback ordering while retaining durable execution behavior. Cloudflare’s rollback support is best suited to workflows involving external side effects. Developers should define compensation alongside every reversible step and make both forward and rollback operations safely repeatable.