Deploy Safety: Reducing customer impact from change
Slack’s Deploy Safety Program reduced customer-impact hours by 90% from its peak by focusing on safer change across all deployment systems, rather than optimizing individual services in isolation. The program combines measurable reliability goals, automated detection and rollback, blast-radius reduction, and cultural change. Its core lesson is to invest broadly, measure results, and expand approaches that demonstrably reduce customer impact without slowing development. ## Defining the Reliability Problem - Slack became increasingly mission-critical, raising customer expectations for reliability. - In analysis of customer-facing incidents, 73% were triggered by Slack-induced change, especially code deployments. - Incidents occurred across hundreds of services and multiple deployment systems, producing inconsistent levels of customer impact. - Customers reported that interruptions became significantly more disruptive after roughly 10 minutes. - Earlier reliability efforts often focused on individual deployment systems or services, leading to manual processes that slowed innovation and reduced engineering morale. ## North Star Goals and the Deploy Safety Manifesto The initial program goals applied to Slack’s highest-importance services: - Detect and automatically remediate deployment problems within 10 minutes. - Detect and manually remediate problems within 20 minutes. - Identify problematic deployments before they reach 10% of the fleet. - Preserve Slack’s engineering and development velocity. These goals later evolved into a Deploy Safety Manifesto covering all deployment systems and processes, including: - Automated safety improvements. - Deployment guardrails. - Changes to engineering practices and safety culture. ## Measuring Customer Impact Slack defined its primary program metric as: - **Hours of customer impact from high-severity and selected medium-severity change-triggered incidents.** The metric is an imperfect proxy for customer sentiment because: - Incident severity reflects current or anticipated impact, not always the final customer experience. - Medium-severity incidents require additional filtering to determine whether their actual impact is relevant. - It can be difficult to connect an individual engineering project directly to changes in customer sentiment. Slack evaluates the metric using four principles: - Measure outcomes rather than activity. - Distinguish real measurements from proxy metrics. - Apply subjective criteria consistently. - Regularly validate the metric against feedback from leaders who speak directly with customers. ## Choosing Where to Invest At the beginning of the program, Slack did not know which projects would produce the greatest benefit or when results would appear. Incident data is inherently delayed, while customers are experiencing reliability problems immediately. The investment strategy therefore emphasized: - Broad initial investment and a bias toward action. - Addressing known customer pain first. - Expanding successful projects and repeatable patterns. - Reducing investment in areas with limited impact. - Maintaining a flexible roadmap that could change as results emerged. Projects were prioritized according to whether they could: - Detect deployment problems earlier. - Improve automatic remediation time. - Improve manual rollback and remediation time. - Reduce severity by limiting deployment blast radius. ## Improving Webapp Backend Deployments Slack identified Webapp backend deployments as the largest source of change-triggered incidents and iteratively improved their safety: - Built automated metric monitoring. - Added automatic alerts and manual rollback procedures to validate alignment with customer impact. - Introduced automatic deployments and rollback. - Demonstrated that repeated automatic rollbacks could keep customer impact below 10 minutes. - Expanded monitoring to additional metrics. - Optimized manual rollback processes. - Added manual rollback capability for the frontend. - Began consolidating deployment practices through a centralized orchestration system inspired by ReleaseBot and AWS Pipelines. - Extended metrics-based deployment and automatic remediation beyond Bedrock and Kubernetes. These improvements made Webapp backend, frontend, and some infrastructure deployments significantly safer, with continued quarter-over-quarter improvement. ## Iterative Expansion Slack applied the same pattern across other areas: - Try an intervention. - Measure whether customer impact improves. - Invest further when the approach succeeds. - Reuse successful patterns in other systems. - Reduce or redirect investment when results are limited. The article notes that some efforts, such as faster mobile-app issue detection, were successful, while others produced less noticeable improvements. Slack’s experience suggests that deployment safety works best as an ongoing program: establish measurable customer-focused goals, automate detection and recovery, control blast radius, and continuously replicate proven practices without sacrificing delivery speed.
Read original(opens in new tab)