Cloudflare Workflows now supports saga rollbacks, letting each durable step declare how to compensate for its side effects if a later operation fails. This addresses partial failures in multi-step processes, such as refunding a debit when a subsequent credit cannot complete. Rollbacks execute in reverse order and preserve Workflow durability, while requiring the same idempotency safeguards as normal steps.
## The Saga Problem
- Durable Workflows can retry steps and persist state, but completed external operations cannot always be directly undone.
- In a bank transfer:
- Bank A debits the sender.
- Bank B fails to credit the recipient.
- The original debit must be reversed with a new credit operation.
- The pairing of a forward action and its semantic compensation is known as the saga pattern.
## Manual Compensation Before Rollbacks
- Developers had to track which steps completed and write centralized `try`/`catch` logic.
- Compensation had to:
- Run only for completed operations.
- Execute in reverse order.
- Continue even if one rollback fails.
- Remain durable and retryable.
- This approach becomes increasingly complex as workflows gain more steps.
## Rollback Functions on `step.do()`
- Rollback logic is now declared directly in the step’s options:
```js
await step.do("debit-bank-a", debitFn, {
rollback: async ({ output }) => refundFn(output.id),
});
```
- Each step carries its own undo operation, making compensation easier to maintain.
- Rollbacks can use the original step output, such as a payment or transaction ID.
- If a later step fails, previously registered rollback handlers run automatically in reverse step-start order.
## Idempotency and Partial Failures
- Rollback functions must be idempotent because they may be retried.
- External operations should use idempotency keys to prevent duplicate refunds, credits, or inventory releases.
- A step that fails may still need compensation:
- It could have modified an external system before failing.
- The operation may have succeeded even though Workflows never received its result.
- Rollback handlers must therefore handle `output === undefined`.
- If user code catches an error and the Workflow continues, rollback does not immediately start. However, if the Workflow later fails, previously registered handlers can still run.
## Practical Usage
- Developers pass an options object with a `rollback` function as the final argument to `step.do()`.
- Rollbacks can reverse payments, release resources, or perform other compensating actions.
- This removes the need for growing manual catch blocks and explicit rollback ordering while retaining durable execution behavior.
Cloudflare’s rollback support is best suited to workflows involving external side effects. Developers should define compensation alongside every reversible step and make both forward and rollback operations safely repeatable.
Woowa Brothers argues that incident impact depends not only on how quickly an outage is detected, but also on how quickly an effective First Action is executed. Analysis of more than 70 incidents showed that incidents beginning with a hotfix tended to last nearly twice as long as those beginning with a rollback. The company therefore established a standardized incident lifecycle and metrics system to make early response measurable and improve it through automation and process design.
## Why First Action Matters
- Detection was generally fast in Woowa Brothers’ 2025 incidents, but customer impact often continued for a long time.
- Rollbacks can immediately undo a problematic change, while hotfixes require diagnosis, coding, and redeployment.
- During hotfix preparation, the service may remain impaired and customer impact continues to accumulate.
- Predefined mechanical mitigations—such as rollback or scaling—are particularly effective because they can be executed without lengthy additional decisions.
- First Action is defined by both:
- **What** action was taken
- **When** it was executed after the incident was recognized
## Standardizing Incident Response
- Comparing First Actions across incidents was difficult because teams used different starting points:
- The moment customer impact was recognized
- The moment an incident was announced company-wide
- Other team-specific interpretations
- Woowa Brothers concluded that First Action could only be measured consistently if the entire incident timeline used shared definitions.
- The lifecycle provides a common framework for understanding where an incident is, what action should happen next, and how long each stage takes.
## The Seven-Stage Incident Lifecycle
### Potential-Incident Lifecycle
#### 1. Anomaly
- A service or system shows abnormal behavior and the responsible team detects and acknowledges it.
- Acknowledgment must leave objective evidence, such as:
- An on-call alert acknowledgment
- A customer-service response comment
- An alert comment
- If there is no customer impact, the process may end after internal handling.
- If any part of the ordering process becomes unavailable, the event transitions into the Incident Lifecycle.
### Incident Lifecycle
#### 2. Open
- The service owner recognizes the event as an incident.
- An incident-response channel is created and relevant technical and business organizations are invited to coordinate.
#### 3. Investigating
- Teams assess customer impact and investigate likely causes.
- They first examine recent deployments, configuration changes, and failures in external dependencies.
#### 4. Identified
- Teams execute actions to reduce customer impact.
- Rollbacks and scaling adjustments are prioritized as First Actions.
- Multiple mitigation options may be evaluated and applied in parallel rather than waiting for a single definitive root cause.
#### 5. Monitoring
- Teams verify whether the applied actions are actually reducing customer impact.
- If not, the process returns to investigation and mitigation for another response attempt.
#### 6. Resolved
- Customer impact has been eliminated and the incident is considered resolved.
- Findings and actions are communicated across the organization according to an established process.
#### 7. Closure Time
- The team documents the root-cause analysis and tracks preventive follow-up work.
- Woowa Brothers separates incident reporting from execution of corrective actions to ensure resolution leads to real operational improvement.
## Metrics for Measuring Response Performance
The lifecycle is useful because it connects incident stages to time-based metrics. These metrics are intended to reveal bottlenecks and guide improvement, not serve as goals in themselves.
### MTTD: Mean Time to Detect
- Measures the average time from incident occurrence until detection and acknowledgment.
- Woowa Brothers includes acknowledgment evidence—similar to MTTA—because an alert alone does not prove that response has begun.
- A high MTTD may indicate:
- Inadequate monitoring coverage or thresholds
- Excessive alert noise
- Missing acknowledgment records
### MTTR: Mean Time to Repair
- Measures the average time from incident acknowledgment until service recovery.
- A high MTTR can indicate:
- Poor First Action readiness
- Insufficient service visibility
- Complex recovery procedures
- Slow decisions or communication bottlenecks
- It points to structural improvements such as automation, standardized procedures, and clearer decision-making authority.
### MTTA: Mean Time to Action
- Measures how quickly mitigation is initiated.
- Rather than judging whether a response was “good,” it evaluates whether standardized response mechanisms activate promptly.
- Woowa Brothers divides it into two more specific metrics.
#### MTTFA: Mean Time to First Action
- Measures the time until the first predefined mechanical mitigation, such as a rollback or scaling adjustment.
- A high MTTFA may result from:
- Complicated rollback paths
- Manual scaling operations
- Excessive decisions or preparation required before execution
- The recommended solution is to automate and simplify First Action procedures.
#### MTTEA: Mean Time to Effective Action
- Measures the time from incident occurrence until abnormal metrics begin improving after an effective mitigation is applied.
- It captures whether an action actually worked, not merely whether it was executed.
- The relationship between MTTFA and MTTEA provides useful signals:
- **MTTEA ≈ MTTFA:** the initial action was fast and immediately effective.
- **MTTEA > MTTFA:** the First Action was insufficient and additional response was needed.
- **MTTEA without MTTFA:** no mechanical initial action was possible, or responders relied directly on a hotfix or similar intervention.
- **Increasing MTTEA:** standard scenarios or automation require improvement.
The practical recommendation is to treat incident response as an organizational system rather than an individual skill. Define a shared lifecycle, prioritize fast and reversible mitigations such as rollback, automate their execution, and use metrics like MTTFA and MTTEA to continuously remove response bottlenecks.
