toss·Jun 9, 2026·3 min readCurated summary

How Do TAMs Solve Problems in Fast-Moving Organizations?

TAM CONNECT 2025 brought together Technical Account Managers from Toss and Kakao Pay to compare how they connect technology, business, customers, and internal teams. Although their organizations differ, they face similar challenges: operational complexity, cross-team coordination, recurring incidents, and the need to improve customer experience. The event framed TAMs not as basic support staff, but as technology-driven problem solvers whose role is expanding through automation and AI. ## The Scope of a TAM’s Role - TAMs resolve partner integration issues and advise on API adoption. - They coordinate multiple teams during incidents and improve operational processes. - Their work includes automating recurring problems and influencing product and platform design. - Depending on the situation, a TAM may act like a developer, product manager, or incident leader. - At Toss, their responsibilities span authentication, Face Connect, financial platforms, online and offline payments, and partner APIs. ## Reducing Alert Noise and Preventing Recurring Problems - Toss’s Dayoung Park presented a problem-solving framework that began by redefining which alerts truly require attention. - Excessive notifications can obscure serious incidents, so the team focused on identifying meaningful operational signals. - They structured incident patterns, detected recurring issues automatically, and analyzed the root causes of settlement discrepancies. - The goal was not merely to resolve incidents faster, but to build systems that prevent them from recurring. ## Making Operations Independent of Individual Owners - PayToss’s Gimun Lee discussed reducing dependency on specific people’s knowledge. - Response histories and operational information were shared transparently so anyone could handle an issue. - Their Discord developer community used n8n workflows, LLM-based log analysis, and automatically generated incident-cause and resolution suggestions. - These tools helped the team maintain an average response time of under ten minutes. ## Using Customer Experience to Improve TAM Work - Toss’s Seongmin Chun drew on previous experience working for a customer organization. - Understanding customer frustrations and the information needed during incidents influenced his communication and support practices. - The team used the PDCA cycle to continually improve integration guides, standardize repeated communications, and structure operational processes. - Effective TAM work means not only solving current problems but also ensuring the same problems do not happen again. ## Shared Challenges Across Companies - TAMs must balance customer expectations with internal development teams. - They operate in rapidly changing services with increasingly complex systems. - The role requires simultaneous understanding of technology, business priorities, and customer needs. - TAMs often appear to sit ambiguously between support, operations, development, and business, but their actual work involves structuring complex problems and mobilizing multiple teams. - The event characterized TAMs as technology-based problem solvers rather than simple support personnel. ## AI and the Future of TAM Work - AI is already being applied to: - Log analysis - Incident-cause recommendations - Operations guide generation - Automated responses to recurring inquiries - Anomaly detection - Document search and summarization - As AI handles repetitive responses, TAMs are likely to focus more on complex problem-solving, structural improvements, cross-team coordination, customer-experience design, and operational strategy. TAM CONNECT demonstrated that TAM organizations can learn significantly from one another. As technology and services become more complex, TAMs will likely become increasingly strategic, combining automation and AI with the judgment needed to improve systems, processes, and customer experiences.

Read original(opens in new tab)

Incident ManagementLarge Language ModelsAPIFintech+4

cloudflare·May 1, 2026·3 min readCurated summary

Code Orange: Fail Small is complete. The result is a stronger Cloudflare network

Cloudflare’s “Code Orange: Fail Small” initiative rebuilt key parts of its infrastructure to prevent repeats of the November 18 and December 5, 2025 global outages. The work focused on safer configuration rollouts, smaller failure impact, stronger emergency access, and improved incident communication. Cloudflare concludes that the network is now more resilient, though reliability remains an ongoing effort. ## Safer Configuration Changes - Configuration changes affecting customer traffic are now deployed progressively rather than instantly across the network. - Health monitoring can detect problems early and automatically roll back faulty changes. - Cloudflare introduced **Snapstone**, a unified system that: - Packages configuration changes. - Releases them gradually. - Monitors system health in real time. - Automatically rolls back unsafe deployments. - Snapstone supports different configuration types, including data files and global control flags. - New high-risk configuration pipelines have been identified and brought under the health-mediated deployment process. ## Reducing the Impact of Failure - Product teams reviewed failure modes and removed unnecessary runtime dependencies. - Systems now use the last known good configuration where possible, following a **“fail stale”** strategy. - Where stale configuration is unavailable, teams choose between: - **Fail open:** Continue serving traffic with reduced protection or functionality. - **Fail close:** Stop processing when that is safer than continuing. - The Bot Management outage scenario would now be detected during an early rollout stage, affecting only a small amount of traffic before rollback. - Services are increasingly segmented into independent systems serving different customer cohorts. - For example, the Workers runtime deploys first to less-critical segments, such as free customers, before reaching more critical traffic. - This approach limits the blast radius of faulty deployments and adjusts rollout speed based on customer criticality. - Cloudflare plans to extend cohort-based deployment to more systems. ## Revised Break-Glass and Incident Procedures - Cloudflare audited tools needed for visibility, debugging, and emergency production changes. - It created backup authorization paths for **18 key services**, along with emergency scripts and proxies. - These pathways are designed to remain usable if Cloudflare’s own Zero Trust infrastructure is affected by an outage. - More than 200 engineers participated in an organization-wide emergency drill on April 7, 2026. - Repeated exercises are intended to ensure engineers can use emergency access procedures effectively under pressure. - Cloudflare also began improving how technical incident observations are converted into clear customer communications. Cloudflare’s changes make configuration rollouts safer, reduce failure blast radius, and improve emergency response. The practical recommendation is to treat these safeguards as ongoing operational practices rather than a one-time project, continually testing them and extending them to additional systems.

Read original(opens in new tab)

Incident ManagementCloudflareZero TrustConfiguration Management+4

datadog·Apr 21, 2026·3 min readCurated summary

Steganography at scale: Embedding share URLs in Datadog widget screenshots

Datadog developed invisible pixel-level watermarks so screenshots can retain the context normally preserved by share links. The system embeds a compact widget snapshot ID into a widget’s border, while the full metadata remains in a Redis cache. This approach preserves screenshots’ convenience while enabling recovery of queries, time ranges, settings, and deep links at Datadog’s scale. ## Share Links Versus Screenshots - Copying a Datadog widget creates a backend record and places a unique share URL in the clipboard. - Pasting the URL into a dashboard or notebook restores the widget. - Slack and Teams integrations can render a live preview and link to Graph Explorer. - Screenshots are easier to use and provide a consistent visual snapshot, but normally lose: - Time range - Underlying queries - Visualization type - Dashboard state - Configuration and context ## Encoding Only a Snapshot ID - A complete widget definition averages about 2 kB and may include queries, display settings, legends, time-frame overrides, template variables, dimensions, and deep links. - Rather than embedding all of that data in the image, Datadog stores it in Redis and embeds only a randomly generated key. - The frontend generates the snapshot ID optimistically before the cache write completes, allowing watermarking without waiting for a backend response. - Records are retained for one hour because screenshots are usually shared within seconds or minutes. - At more than 1 billion widget renders per day, IDs must be compact while avoiding cross-customer collisions. - Datadog prefixes the cache key with the organization ID. An 8-byte ID provides roughly 2⁶⁴ possible values, producing an estimated collision probability of about 1 in 37 million under the stated usage assumptions. ## Watermarking the Widget Border - Every dashboard widget has a consistent 1-pixel border, making it a reliable location for encoding data regardless of visualization type. - An initial design represented each bit with a separate colored pixel, but 64 pixels were needed for 8 bytes and could become visible. - The final design stores data in RGB color adjustments: - Each pixel encodes up to 9 bits by offsetting the red, green, and blue channels. - The base color is calculated by subtracting 3 from each channel. - Channel offsets of up to 7 represent the encoded values. - Two sentinel pixels, using a `+7/+7/+7` offset, mark the beginning and end of the watermark. - Eight pixels between the sentinels encode one byte each: - 3 bits in red - 3 bits in green - 2 highest bits in blue ## Design Constraints - The watermark must remain nearly invisible and avoid adding interface elements. - It must work across different widget sizes, color profiles, display densities, and copy-paste workflows. - The border-based method avoids visualization-specific implementations while keeping the encoded region short. Datadog’s approach combines cached metadata with subtle RGB-level encoding, allowing screenshots to function like context-preserving share links without changing their appearance or the user’s workflow.

Read original(opens in new tab)

Incident ManagementDatadogRedisWatermarking+3

meta·Apr 8, 2026·2 min readCurated summary

Trust But Canary: Configuration Safety at Scale

As AI accelerates software development, stronger safeguards are needed to prevent faster mistakes from becoming larger incidents. Meta’s Configurations team uses canarying, progressive rollouts, health checks, and monitoring to detect regressions early. Data and AI also help reduce alert noise and speed up identifying the changes responsible for failures. ## Safe Configuration Rollouts - Meta deploys configuration changes gradually rather than releasing them everywhere at once. - Canarying exposes changes to a small subset of systems or users first. - Progressive rollouts expand the deployment only when monitoring indicates that the change is healthy. - These practices limit the impact of faulty configurations and provide opportunities to stop or reverse a rollout. ## Monitoring and Health Checks - Automated health checks and operational signals help identify regressions soon after deployment. - Monitoring provides evidence for deciding whether a rollout should continue, pause, or be rolled back. - Early detection is especially important at Meta’s scale, where a small configuration error can affect many systems. ## Learning from Incidents - Incident reviews focus on improving tools, processes, and safeguards rather than assigning blame to individuals. - The goal is to make future failures less likely and reduce their potential impact. - These reviews turn operational problems into improvements across the configuration management system. ## AI-Assisted Operations - Data-driven techniques reduce alert noise so engineers can focus on meaningful signals. - AI and machine learning help speed up bisection, narrowing down which change introduced a problem. - Faster diagnosis can shorten recovery times and make progressive deployment practices more effective. The episode recommends combining gradual releases, strong observability, blameless incident reviews, and AI-assisted analysis to keep increasingly rapid development safe at scale.

Read original(opens in new tab)

Incident ManagementAIMachine LearningCanary Deployment+3

woowahan·Feb 2, 2026·5 min readCurated summary

The First Action That Determines the Success or

Woowa Brothers argues that incident impact depends not only on how quickly an outage is detected, but also on how quickly an effective First Action is executed. Analysis of more than 70 incidents showed that incidents beginning with a hotfix tended to last nearly twice as long as those beginning with a rollback. The company therefore established a standardized incident lifecycle and metrics system to make early response measurable and improve it through automation and process design. ## Why First Action Matters - Detection was generally fast in Woowa Brothers’ 2025 incidents, but customer impact often continued for a long time. - Rollbacks can immediately undo a problematic change, while hotfixes require diagnosis, coding, and redeployment. - During hotfix preparation, the service may remain impaired and customer impact continues to accumulate. - Predefined mechanical mitigations—such as rollback or scaling—are particularly effective because they can be executed without lengthy additional decisions. - First Action is defined by both: - **What** action was taken - **When** it was executed after the incident was recognized ## Standardizing Incident Response - Comparing First Actions across incidents was difficult because teams used different starting points: - The moment customer impact was recognized - The moment an incident was announced company-wide - Other team-specific interpretations - Woowa Brothers concluded that First Action could only be measured consistently if the entire incident timeline used shared definitions. - The lifecycle provides a common framework for understanding where an incident is, what action should happen next, and how long each stage takes. ## The Seven-Stage Incident Lifecycle ### Potential-Incident Lifecycle #### 1. Anomaly - A service or system shows abnormal behavior and the responsible team detects and acknowledges it. - Acknowledgment must leave objective evidence, such as: - An on-call alert acknowledgment - A customer-service response comment - An alert comment - If there is no customer impact, the process may end after internal handling. - If any part of the ordering process becomes unavailable, the event transitions into the Incident Lifecycle. ### Incident Lifecycle #### 2. Open - The service owner recognizes the event as an incident. - An incident-response channel is created and relevant technical and business organizations are invited to coordinate. #### 3. Investigating - Teams assess customer impact and investigate likely causes. - They first examine recent deployments, configuration changes, and failures in external dependencies. #### 4. Identified - Teams execute actions to reduce customer impact. - Rollbacks and scaling adjustments are prioritized as First Actions. - Multiple mitigation options may be evaluated and applied in parallel rather than waiting for a single definitive root cause. #### 5. Monitoring - Teams verify whether the applied actions are actually reducing customer impact. - If not, the process returns to investigation and mitigation for another response attempt. #### 6. Resolved - Customer impact has been eliminated and the incident is considered resolved. - Findings and actions are communicated across the organization according to an established process. #### 7. Closure Time - The team documents the root-cause analysis and tracks preventive follow-up work. - Woowa Brothers separates incident reporting from execution of corrective actions to ensure resolution leads to real operational improvement. ## Metrics for Measuring Response Performance The lifecycle is useful because it connects incident stages to time-based metrics. These metrics are intended to reveal bottlenecks and guide improvement, not serve as goals in themselves. ### MTTD: Mean Time to Detect - Measures the average time from incident occurrence until detection and acknowledgment. - Woowa Brothers includes acknowledgment evidence—similar to MTTA—because an alert alone does not prove that response has begun. - A high MTTD may indicate: - Inadequate monitoring coverage or thresholds - Excessive alert noise - Missing acknowledgment records ### MTTR: Mean Time to Repair - Measures the average time from incident acknowledgment until service recovery. - A high MTTR can indicate: - Poor First Action readiness - Insufficient service visibility - Complex recovery procedures - Slow decisions or communication bottlenecks - It points to structural improvements such as automation, standardized procedures, and clearer decision-making authority. ### MTTA: Mean Time to Action - Measures how quickly mitigation is initiated. - Rather than judging whether a response was “good,” it evaluates whether standardized response mechanisms activate promptly. - Woowa Brothers divides it into two more specific metrics. #### MTTFA: Mean Time to First Action - Measures the time until the first predefined mechanical mitigation, such as a rollback or scaling adjustment. - A high MTTFA may result from: - Complicated rollback paths - Manual scaling operations - Excessive decisions or preparation required before execution - The recommended solution is to automate and simplify First Action procedures. #### MTTEA: Mean Time to Effective Action - Measures the time from incident occurrence until abnormal metrics begin improving after an effective mitigation is applied. - It captures whether an action actually worked, not merely whether it was executed. - The relationship between MTTFA and MTTEA provides useful signals: - **MTTEA ≈ MTTFA:** the initial action was fast and immediately effective. - **MTTEA > MTTFA:** the First Action was insufficient and additional response was needed. - **MTTEA without MTTFA:** no mechanical initial action was possible, or responders relied directly on a hotfix or similar intervention. - **Increasing MTTEA:** standard scenarios or automation require improvement. The practical recommendation is to treat incident response as an organizational system rather than an individual skill. Define a shared lifecycle, prioritize fast and reversible mitigations such as rollback, automate their execution, and use metrics like MTTFA and MTTEA to continuously remove response bottlenecks.

Read original(opens in new tab)

Incident ManagementDatabase DesignRollbackRoot Cause Analysis+4

meta·Dec 19, 2025Original article

DrP: Meta's Root Cause Analysis Platform at Scale (opens in new tab)

DrP is Meta’s programmatic root cause analysis (RCA) platform designed to automate incident investigations and reduce the burden of manual on-call tasks. By codifying investigation playbooks into executable "analyzers," the platform significantly lowers the mean time to resolve (MTTR) by 20% to 80% for over 300 teams. This systematic approach replaces outdated manual scripts with a scalable backend that executes 50,000 automated analyses daily, providing immediate context when alerts fire. ## Architecture and Core Components * **Expressive SDK:** Provides a framework for engineers to codify investigation workflows into "analyzers," utilizing a rich library of helper functions and machine learning algorithms. * **Built-in Analysis Tools:** The platform includes native support for anomaly detection, event isolation, time-series correlation, and dimension analysis to identify specific problem areas. * **Scalable Backend:** A multi-tenant execution environment manages a worker pool that handles thousands of requests securely and asynchronously. * **Workflow Integration:** DrP is integrated directly into Meta’s internal alerting and incident management systems, allowing for automatic triggering without human intervention. ## Authoring and Verification Workflow * **Template Bootstrapping:** Engineers use the SDK to generate boilerplate code that captures required input parameters and context in a type-safe manner. * **Analyzer Chaining:** The system allows for seamless dependency analysis by passing context between different analyzers, enabling investigations to span multiple interconnected services. * **Automated Backtesting:** Before deployment, analyzers undergo automated backtesting integrated into the code review process to ensure accuracy and performance. * **Decision Tree Logic:** Investigation steps are modeled as decision trees within the code, allowing the analyzer to follow different paths based on the data it retrieves. ## Execution and Post-Processing * **Trigger-based Analysis:** When an alert is activated, the backend automatically queues the relevant analyzer, ensuring findings are available as soon as an engineer begins triaging. * **Automated Mitigation:** A post-processing system can take direct action based on investigation results, such as creating tasks or submitting pull requests to resolve identified issues. * **DrP Insights:** This system periodically reviews historical analysis outputs to identify and rank the top causes of alerts, helping teams prioritize long-term reliability fixes. * **Alert Annotation:** Results are presented in both human-readable text and machine-readable formats, directly annotating the incident logs for the on-call responder. ## Practical Conclusion Organizations managing large-scale distributed systems should transition from static markdown playbooks to executable investigation code. By implementing a programmatic RCA framework like DrP, teams can scale their troubleshooting expertise and significantly reduce "on-call fatigue" by automating the repetitive triage steps that typically consume the first hour of an incident.

Incident ManagementAIMachine LearningObservability+4

aws·Dec 2, 2025Original article

New and enhanced AWS Support plans add AI capabilities to expert guidance (opens in new tab)

AWS has announced a major transformation of its support plans, moving from a reactive model to a proactive, AI-driven approach for issue prevention and workload optimization. By integrating AI-powered capabilities with deep technical expertise, these enhanced plans aim to help organizations identify potential operational risks before they impact business performance. This new tier-based structure provides businesses with varying levels of contextual assistance, ranging from intelligent automated recommendations to direct access to specialized engineering teams. ### Business Support+ * Introduces intelligent, AI-powered assistance designed to provide contextual recommendations for developers, startups, and small businesses. * Features a seamless transition from AI tools to human experts, with critical case response times reduced to 30 minutes—twice as fast as previous standards. * Provides personalized workload optimization suggestions based on the user's specific environment via a low-cost monthly subscription. ### Enterprise Support * Assigns a designated Technical Account Manager (TAM) who utilizes data-driven insights and AI tools to mitigate risks and identify optimization opportunities. * Grants access to the AWS Security Incident Response service at no additional fee, centralizing the tracking, monitoring, and investigation of security events. * Guarantees a 15-minute response time for production-critical issues, with support engineers receiving AI-generated context to ensure faster, more personalized resolution. * Includes access to hands-on workshops and interactive programs to foster continuous technical growth within the organization. ### Unified Operations Support * Provides the highest level of context-aware assistance through a dedicated core team including a TAM, a Domain Engineer, and a Senior Billing and Account Specialist. * Delivers industry-leading 5-minute response times for critical incidents, supported by around-the-clock monitoring and AI-powered proactive risk identification. * Offers on-demand access to specialized experts in migration, incident management, and security through the customer’s preferred collaboration channels. These updates reflect AWS’s commitment to using generative AI to shorten resolution times and provide more personalized architectural guidance. Organizations should evaluate their operational complexity and required response times to select the plan that best aligns with their mission-critical cloud needs.

Incident ManagementAIAWSCloud Monitoring+3

woowahan·Dec 2, 2025Original article

How Woowa Brothers Detects (opens in new tab)

Woowa Brothers addresses the inevitability of system failures by shifting from traditional resource-based monitoring to a specialized Service Anomaly Detection system. By focusing on high-level service metrics such as order volume and login counts rather than just CPU or memory usage, they can identify incidents that directly impact the user experience. This approach ensures near real-time detection and provides a structured response framework to minimize damage during peak service hours. ### The Shift to Service-Level Monitoring * Traditional monitoring focuses on infrastructure metrics like CPU and memory, but it is impossible to monitor every system variable, leading to "blind spots" in failure detection. * Service metrics, such as real-time login counts and payment success rates, are finite and offer a direct reflection of the actual customer experience. * By monitoring these core indicators, the SRE team can detect anomalies that system-level alerts might overlook, ensuring that no failure goes unnoticed. ### Requirements for Effective Anomaly Detection * **Real-time Performance:** Alerts must be triggered in near-real-time to allow for immediate intervention before the impact scales. * **Explainability:** The system favors transparent logic over "black-box" AI models, allowing developers to quickly understand why an alert was triggered and how to improve the detection logic. * **Integrated Response:** Beyond just detection, the system must provide a clear response process so that any engineer, regardless of experience, can follow a standardized path to resolution. ### Technical Implementation and Logic * The system leverages the predictable, pattern-based nature of delivery service traffic, which typically peaks during lunch and dinner. * The team chose a Median-based approach to generate "Prediction" values from historical data, as it is more robust against outliers and easier to analyze than complex methods like IQR or 2-sigma. * Detection is determined by comparing "Actual" values against "Warning" and "Critical" thresholds derived from the predicted median. * To prevent false positives caused by temporary spikes, the system tracks "threshold reach counts," requiring a metric to stay in an abnormal state for a specific number of consecutive cycles before firing a Slack alert. ### Optimization of Alert Accuracy * Each service metric requires a tailored "settling period" to find the optimal balance between detection speed and accuracy. * Setting a high threshold reach count improves accuracy but slows down detection, while a low count accelerates detection at the risk of increased false positives. * Alerts are delivered via Slack with comprehensive context, including current status and urgency, to facilitate rapid decision-making. For organizations running high-traffic services, prioritizing service-level indicators (SLIs) over infrastructure metrics can significantly reduce the time to detect critical failures. Implementing simple, explainable statistical models like the Median approach allows teams to maintain a reliable monitoring system that evolves alongside the service without the complexity of uninterpretable AI models.

Incident ManagementMonitoringAnomaly DetectionSre+2

naver·Nov 27, 2025Original article

Naver TV (opens in new tab)

Naver’s Integrated Search team is transitioning from manual fault response to an automated system using LLM Agents to manage the increasing complexity of search infrastructure. By integrating Large Language Models into the DevOps pipeline, the system evolves through accumulated experience, moving beyond simple alert monitoring to intelligent diagnostic analysis and action recommendation. ### Limitations of Traditional Fault Response * **Complex Search Flows:** Naver’s search architecture involves multiple interdependent layers, which makes manual root cause analysis slow and prone to human error. * **Fragmented Context:** Existing monitoring requires developers to manually synthesize logs and metrics from disparate telemetry sources, leading to high cognitive load during outages. * **Delayed Intervention:** Human-led responses often suffer from a "detection-to-action" lag, especially during high-traffic periods or subtle service regressions. ### Architecture of DevOps Agent v1 * **Initial Design:** Focused on automating basic data gathering and providing preliminary textual reports to engineers. * **Infrastructure Integration:** Built using a specialized software stack designed to bridge frontend (FE) and backend (BE) telemetry within the search infrastructure. * **Standardized Logic:** The v1 agent operated on a fixed set of instructions to perform predefined diagnostic tasks when triggered by specific system alarms. ### Evolution to DevOps Agent v2 * **Overcoming V1 Limitations:** The first iteration struggled with maintaining deep context and providing diverse actionable insights, necessitating a more robust agentic structure. * **Enhanced Memory and Learning:** V2 incorporates a more sophisticated architecture that allows the agent to reference historical failure data and learn from past incident resolutions. * **Advanced Tool Interaction:** The system was upgraded to handle more complex tool-calling capabilities, allowing the agent to interact more deeply with internal infrastructure APIs. ### System Operations and Evaluation * **Trigger Queue Management:** Implements a queuing system to efficiently process and prioritize multiple concurrent system alerts without overwhelming the diagnostic pipeline. * **Anomaly Detection:** Utilizes advanced detection methods to distinguish between routine traffic fluctuations and genuine service anomalies that require LLM intervention. * **Rigorous Evaluation:** The agent’s performance is measured through a dedicated evaluation framework that assesses the accuracy of its diagnoses against known ground-truth incidents. ### Scaling and Future Challenges * **Context Expansion:** Efforts are focused on integrating a wider range of metadata and environmental context to provide a holistic view of system health. * **Action Recommendation:** The system is moving toward suggesting specific recovery actions, such as rollbacks or traffic rerouting, rather than just identifying the problem. * **Sustainability:** Ensuring the DevOps Agent remains maintainable and cost-effective as the underlying search infrastructure and LLM models continue to evolve. Organizations managing high-scale search traffic should consider LLM-based agents as integrated infrastructure components rather than standalone tools. Moving from reactive monitoring to a proactive, experience-based agent system is essential for reducing the mean time to recovery (MTTR) in complex distributed environments.

Incident ManagementAILarge Language ModelsDevOps+3

slack·Oct 7, 2025·4 min readCurated summary

Deploy Safety: Reducing customer impact from change

Slack’s Deploy Safety Program reduced customer-impact hours by 90% from its peak by focusing on safer change across all deployment systems, rather than optimizing individual services in isolation. The program combines measurable reliability goals, automated detection and rollback, blast-radius reduction, and cultural change. Its core lesson is to invest broadly, measure results, and expand approaches that demonstrably reduce customer impact without slowing development. ## Defining the Reliability Problem - Slack became increasingly mission-critical, raising customer expectations for reliability. - In analysis of customer-facing incidents, 73% were triggered by Slack-induced change, especially code deployments. - Incidents occurred across hundreds of services and multiple deployment systems, producing inconsistent levels of customer impact. - Customers reported that interruptions became significantly more disruptive after roughly 10 minutes. - Earlier reliability efforts often focused on individual deployment systems or services, leading to manual processes that slowed innovation and reduced engineering morale. ## North Star Goals and the Deploy Safety Manifesto The initial program goals applied to Slack’s highest-importance services: - Detect and automatically remediate deployment problems within 10 minutes. - Detect and manually remediate problems within 20 minutes. - Identify problematic deployments before they reach 10% of the fleet. - Preserve Slack’s engineering and development velocity. These goals later evolved into a Deploy Safety Manifesto covering all deployment systems and processes, including: - Automated safety improvements. - Deployment guardrails. - Changes to engineering practices and safety culture. ## Measuring Customer Impact Slack defined its primary program metric as: - **Hours of customer impact from high-severity and selected medium-severity change-triggered incidents.** The metric is an imperfect proxy for customer sentiment because: - Incident severity reflects current or anticipated impact, not always the final customer experience. - Medium-severity incidents require additional filtering to determine whether their actual impact is relevant. - It can be difficult to connect an individual engineering project directly to changes in customer sentiment. Slack evaluates the metric using four principles: - Measure outcomes rather than activity. - Distinguish real measurements from proxy metrics. - Apply subjective criteria consistently. - Regularly validate the metric against feedback from leaders who speak directly with customers. ## Choosing Where to Invest At the beginning of the program, Slack did not know which projects would produce the greatest benefit or when results would appear. Incident data is inherently delayed, while customers are experiencing reliability problems immediately. The investment strategy therefore emphasized: - Broad initial investment and a bias toward action. - Addressing known customer pain first. - Expanding successful projects and repeatable patterns. - Reducing investment in areas with limited impact. - Maintaining a flexible roadmap that could change as results emerged. Projects were prioritized according to whether they could: - Detect deployment problems earlier. - Improve automatic remediation time. - Improve manual rollback and remediation time. - Reduce severity by limiting deployment blast radius. ## Improving Webapp Backend Deployments Slack identified Webapp backend deployments as the largest source of change-triggered incidents and iteratively improved their safety: - Built automated metric monitoring. - Added automatic alerts and manual rollback procedures to validate alignment with customer impact. - Introduced automatic deployments and rollback. - Demonstrated that repeated automatic rollbacks could keep customer impact below 10 minutes. - Expanded monitoring to additional metrics. - Optimized manual rollback processes. - Added manual rollback capability for the frontend. - Began consolidating deployment practices through a centralized orchestration system inspired by ReleaseBot and AWS Pipelines. - Extended metrics-based deployment and automatic remediation beyond Bedrock and Kubernetes. These improvements made Webapp backend, frontend, and some infrastructure deployments significantly safer, with continued quarter-over-quarter improvement. ## Iterative Expansion Slack applied the same pattern across other areas: - Try an intervention. - Measure whether customer impact improves. - Invest further when the approach succeeds. - Reuse successful patterns in other systems. - Reduce or redirect investment when results are limited. The article notes that some efforts, such as faster mobile-app issue detection, were successful, while others produced less noticeable improvements. Slack’s experience suggests that deployment safety works best as an ongoing program: establish measurable customer-focused goals, automate detection and recovery, control blast radius, and continuously replicate proven practices without sacrificing delivery speed.

Read original(opens in new tab)

Incident ManagementKubernetesAws PipelinesDeployment Orchestration+2

datadog·Jun 3, 2025·3 min readCurated summary

Detecting faulty deployments: Our journey from unlabeled data to supervised learning

Deployments are a major source of software incidents, making rapid detection of faulty releases essential. Datadog developed Automatic Faulty Deployment Detection to identify releases associated with significant, deployment-related increases in error rates, despite having no reliable labeled dataset. Their solution evolved into an iterative, unsupervised ensemble of statistical checks designed to balance precision, recall, and the diverse behavior of customer applications. ## Challenges in Detecting Faulty Deployments - No universal ground truth exists because teams define “faulty” differently depending on their applications. - Faulty deployments are rare, creating severe class imbalance: - Random manual labeling would produce few useful examples. - Even a low false-positive rate could result in poor precision. - Applications have widely varying traffic and error patterns: - Seasonal applications naturally experience periodic changes. - Low-traffic services need longer observation periods. - Frequent deployments can make it difficult to identify which release caused an incident. ## Defining a Faulty Deployment Datadog focused on deployments that caused a significant and sustained increase in error rate. The definition relied on three attributes: - **Impact** - The total number of errors must be meaningfully higher than the baseline. - The increase must be significantly worse than in previous versions. - **Temporal correlation** - The error increase should align with the introduction of the new version. - **Persistence** - The elevated error rate must continue over time rather than reflecting temporary deployment noise. ## Building an Iterative Detection Framework - The initial system applied simple statistical rules to the first 60 minutes after each deployment. - Manual annotation was used to estimate precision, but this required substantial effort and did not reveal recall. - Datadog created an iterative framework composed of checks for different deployment requirements. - Checks included: - Comparing error rates before and after deployment. - Comparing a release with previous versions. - Accounting for periodic traffic and errors. - Handling sparse traffic patterns. - The checks were combined into a unanimous-voting ensemble: a deployment was flagged only when every check classified it as faulty. - The process began with a high-recall model, then: - Manually reviewed predicted faults. - Analyzed false positives. - Added new checks and adjusted thresholds to improve precision and recall. - Incident data and version rollbacks provided additional signals for finding faulty deployments the model had missed. ## Balancing Detection Speed and Recall - The model used the first hour after deployment to gather enough data to determine whether increased errors were persistent. - Increasing the observation period can improve confidence but delays detection. - The framework became progressively more sophisticated, adapting to: - Periodic error and traffic patterns. - Sparse traffic. - Multiple concurrent application versions. The practical recommendation is to begin with simple, high-recall statistical rules, then iteratively improve them through targeted manual review, false-positive analysis, and additional operational signals such as incidents and rollbacks. This approach can support other anomaly-detection problems where labels are scarce, failures are rare, and application behavior varies significantly.

Read original(opens in new tab)

Incident ManagementDatabase DesignApplication Performance MonitoringStatistical Analysis+4

datadog·Sep 23, 2024·3 min readCurated summary

How we optimized LLM use for cost, quality, and safety to facilitate writing postmortems

Bits AI uses large language models to generate first drafts of incident postmortems from Datadog Incident Management metadata and Slack discussions. The goal is to reduce documentation effort without replacing the human-led reflection that makes postmortems valuable. Datadog found that production-quality LLM features require extensive prompt experimentation, safeguards against hallucinations, careful model selection, and strong privacy and feedback mechanisms. ## Why LLMs Are Difficult to Use for Postmortems - Postmortems must remain factual, consistent, and useful, unlike more creative LLM applications. - LLMs can ignore instructions, produce inconsistent formatting, repeat information, or hallucinate plausible but false details. - More than 100 hours were spent refining section structures and instructions so they worked across varied incidents. - Building these systems requires a combination of software engineering, product management, data science, and technical writing. ## Balancing Structured and Unstructured Incident Data - Structured Datadog data includes incident metadata, alerts, graphs, and manually entered fields such as customer impact. - Slack discussions provide current, detailed, and unstructured context from the incident response process. - Manually maintained fields are often more reliable but can become outdated, while LLMs can continually reassess live discussions. - Combining both sources helped the model understand the full incident and prioritize information appropriately. ## Cost, Speed, and Quality Trade-offs - Datadog compared models including GPT-3.5 and GPT-4. - Results varied by as much as 50 times in cost and speed. - GPT-4 generally produced more accurate output but was substantially slower and more expensive. - Model choice therefore depended on the required balance between quality, latency, and operating cost. ## Preserving Human Control and Privacy - AI-generated drafts were intended to support authors, not become final documents automatically. - Keeping engineers involved preserves the discovery and learning process that postmortems are meant to encourage. - Because incidents may contain sensitive information or secrets, Datadog scrubbed data before sending it to an LLM. - The system was designed to ensure authors remained responsible for reviewing and improving the draft. ## Experimentation and Reliability Safeguards - A custom API extracted and structured incident data from Datadog, Slack, and other sources. - The framework enabled rapid testing of different datasets, models, input configurations, and output token limits. - Datadog refined prompts and instructions to guide the model toward useful recommendations while retaining author agency. - Lowering model temperature and tuning other parameters reduced irrelevant or fabricated content. - A related feature generated concise incident summaries for responders joining Slack channels, providing a faster testing environment for LLM-based incident analysis. ## Evaluating Generated Drafts - Datadog used both qualitative and quantitative evaluation methods. - Authors of earlier postmortems compared their human-written documents with AI-generated drafts for the same incidents. - Evaluation included criteria such as factual accuracy and conciseness, with the provided article ending before the full evaluation approach was described. - Continuous feedback was used to refine prompts, models, and the overall drafting workflow. Overall, reliable AI-assisted postmortems require more than selecting a capable model. The strongest approach combines carefully prepared incident data, privacy controls, model and prompt experimentation, systematic evaluation, and mandatory human review.

Read original(opens in new tab)

Incident ManagementLarge Language ModelsDatadogSlack+4