Durable Execution

3 posts

cloudflare3 min readCurated summary

Bringing more agent harnesses to Cloudflare, starting with Flue

Cloudflare argues that production AI agents need more than an agent harness: they require platform primitives for durable state, execution, storage, and secure compute. It presents a three-layer stack—framework, harness, and runtime—and introduces Flue as the first framework built on the Cloudflare Agents SDK. Flue uses a declarative approach based on Pi, while Cloudflare supplies the infrastructure needed to resume interrupted work and run agents reliably at scale. ## The Three-Layer Agent Stack - **Framework — Flue** - Provides project structure, conventions, integrations, CLI commands, and developer experience. - **Harness — Pi or Project Think** - Runs the agentic loop: calls tools, processes results, manages context, and continues until a task is complete. - **Runtime/platform — Cloudflare Agents SDK** - Supplies compute, state, storage, durable execution, sandboxing, and workflow primitives. - Cloudflare’s goal is to make these runtime capabilities available to any harness or framework. ## Flue’s Declarative Agent Model - Flue 1.0 Beta is built on the Pi harness, which also powers OpenClaw. - Developers describe what an agent knows rather than explicitly scripting its orchestration. - An agent is defined through its: - Model - Skills - Sandbox - Instructions - This allows relatively compact agents to autonomously handle tasks such as reproducing and diagnosing bug reports. ## Flue’s Developer Experience - **Integrated channels** - Preconfigured integrations let agents work in Slack, GitHub, Linear, and Discord. - Channels handle event verification and dispatch boilerplate. - **Headless and UI-ready operation** - Agents can run as background processes. - `@flue/react` provides hooks for streaming agent state, tool execution, and messages into frontend applications. - **Ecosystem integrations** - Commands such as `flue add channel slack` generate Markdown blueprints that coding agents can modify and integrate into a project. ## Durable Execution with Durable Streams - Production agents face host crashes, LLM API timeouts, restarts, and interrupted tool calls. - Flue records prompts, tool responses, model decisions, and other execution events in an append-only log. - This durable event history prevents in-memory state from being lost. - If a process fails, another process can replay the log and resume from the exact point of interruption. ## Deployment Across Clouds - On Node.js, Flue agents run as long-lived processes on VMs, containers, GitHub Actions, or existing servers. - On Cloudflare, each agent runs in its own Durable Object. - This provides: - Isolated storage and compute - Automatic scaling - No need to provision servers or manage sticky sessions - Protection from noisy neighbors - Cloudflare deployments use Agents SDK features including `runFiber()`, `stash()`, and `onFiberRecovered()` for durable execution. - Sandboxed code execution uses `@cloudflare/codemode` and `@cloudflare/shell` with a durable workspace. ## Requirements for Production Agent Harnesses - An agent turn is a multi-step process that may involve token streaming, tool calls, human approval, or delegated subagents. - These operations can last seconds or minutes and may fail at any point. - Persisting only conversation history is insufficient because it does not preserve active execution state, pending tool calls, or the agent’s current position. - Cloudflare’s fiber-based primitives provide checkpointing so interrupted agent turns can recover instead of leaving users with stalled requests. Cloudflare’s recommendation is to treat the framework, harness, and runtime as separate but coordinated layers. Frameworks like Flue make agents easy to build, while the Agents SDK supplies the durable execution and infrastructure primitives required to operate them reliably in production.

Read original(opens in new tab)
netflixOriginal article

How Temporal Powers Reliable Cloud Operations at Netflix | by Netflix Technology Blog | Dec, 2025 | Netflix TechBlog (opens in new tab)

Netflix has significantly enhanced the reliability of its global continuous delivery platform, Spinnaker, by adopting Temporal for durable execution of cloud operations. By migrating away from a fragile, polling-based orchestration model between its internal services, the engineering team successfully reduced transient deployment failures from 4% to a remarkable 0.0001%. This shift has allowed developers to write complex, long-running operational logic as standard code while the underlying platform handles state persistence and fault recovery. ### Limitations of Legacy Orchestration * **The Polling Bottleneck:** Originally, Netflix's orchestration engine (Orca) communicated with its cloud interface (Clouddriver) via a synchronous POST request followed by continuous polling of a GET endpoint to track task status. * **State Fragility:** Clouddriver utilized an internal orchestration engine that relied on in-memory state or volatile Redis storage, meaning if a Clouddriver instance crashed mid-operation, the deployment state was often lost, leading to "zombie" tasks or failed deployments. * **Manual Error Handling:** Developers had to manually implement complex retry logic, exponential backoffs, and state checkpointing for every cloud operation, which was both error-prone and difficult to maintain. ### Transitioning to Durable Execution with Temporal * **Abstraction of Failures:** Temporal provides a "Durable Execution" platform where the state of a workflow—including local variables and thread stacks—is automatically persisted. This allows code to run "as if failures don’t exist," as the system can resume exactly where it left off after a process crash or network interruption. * **Workflows and Activities:** Netflix re-architected cloud operations into Temporal Workflows (orchestration logic) and Activities (idempotent units of work like calling an AWS API). This separation ensures that the orchestration logic remains deterministic while external side effects are handled reliably. * **Eliminating Polling:** By using Temporal’s signaling and long-running execution capabilities, Netflix moved away from the heavy overhead of thousands of services polling for status updates, replacing them with a push-based, event-driven model. ### Impact on Cloud Operations * **Dramatic Reliability Gains:** The most significant outcome was the near-elimination of transient failures, moving from a 4% failure rate to 0.0001%, ensuring that critical updates to the Open Connect CDN and Live streaming infrastructure are executed with high confidence. * **Developer Productivity:** Using Temporal’s SDKs, Netflix engineers can now write standard Java or Go code to define complex deployment strategies (like canary releases or blue-green deployments) without building custom state machines or management layers. * **Operational Visibility:** Temporal provides a native UI and history audit log for every workflow, giving operators deep visibility into exactly which step of a deployment failed and why, along with the ability to retry specific failed steps manually if necessary. For organizations managing complex, distributed cloud infrastructure, adopting a durable execution framework like Temporal is highly recommended. It moves the burden of state management and fault tolerance from the application layer to the platform, allowing engineers to focus on business logic rather than the mechanics of distributed systems failure.

awsOriginal article

Build multi-step applications and AI workflows with AWS Lambda durable functions (opens in new tab)

AWS Lambda durable functions introduce a simplified way to manage complex, long-running workflows directly within the standard Lambda experience. By utilizing a checkpoint and replay mechanism, developers can now write sequential code for multi-step processes that automatically handle state management and retries without the need for external orchestration services. This feature significantly reduces the cost of long-running tasks by allowing functions to suspend execution for up to one year without incurring compute charges during idle periods. ### Durable Execution Mechanism * The system uses a "durable execution" model based on checkpointing and replay to maintain state across function restarts. * When a function is interrupted or resumes from a pause, Lambda re-executes the handler from the beginning but skips already-completed operations by referencing saved checkpoints. * This architecture ensures that business logic remains resilient to failures and can survive execution environment recycles. * The execution state can be maintained for extended periods, supporting workflows that require human intervention or long-duration external processes. ### Programming Primitives and SDK * The feature requires the inclusion of a new open-source durable execution SDK in the function code. * **Steps:** The `context.step()` method defines specific blocks of logic that the system checkpoints and automatically retries upon failure. * **Wait:** The `context.wait()` primitive allows the function to terminate and release compute resources while waiting for a specified duration, resuming only when the time elapses. * **Callbacks:** Developers can use `create_callback()` to pause execution until an external event, such as an API response or a manual approval, is received. * **Advanced Control:** The SDK includes `wait_for_condition()` for polling external statuses and `parallel()` or `map()` operations for managing concurrent execution paths. ### Configuration and Setup * Durable execution must be enabled at the time of the Lambda function's creation; it cannot be retroactively enabled for existing functions. * Once enabled, the function maintains the same event handler structure and service integrations as a standard Lambda function. * The environment is specifically optimized for high-reliability use cases like payment processing, AI agent orchestration, and complex order management. AWS Lambda durable functions represent a major shift for developers who need the power of stateful orchestration but prefer to keep their logic within a single code-based environment. It is highly recommended for building AI workflows and multi-step business processes where state persistence and cost-efficiency are critical requirements.