awsMay 28, 2026

Introducing the next generation of AWS Resilience Hub for generative AI-based SRE resilience journey | Amazon Web Services (opens in new tab)

AWS introduces the next generation of AWS Resilience Hub to help organizations manage application resilience consistently across large portfolios. The update adds a new business-oriented application model, modular resilience policies, dependency discovery, generative AI failure-mode analysis, and organization-wide reporting. Its goal is to help SREs and development teams define expectations, identify weaknesses, implement fixes, and demonstrate resilience through testing. ## Modular Resilience Policies - Policies are built from composable requirements rather than a single fixed policy type. - Requirements can include: - Availability service-level objectives (SLOs) - Multi-AZ and multi-Region disaster recovery - Recovery time objectives (RTOs) - Recovery point objectives (RPOs) - Backup data recovery requirements - Example policies can define 99.95% availability, a 15-minute RTO, and a 5-minute RPO for financial applications. ## Business-Oriented Application Modeling - A **system** represents a business application. - **User journeys** describe critical end-user paths and business outcomes. - **Services** represent deployable units such as microservices, AWS resources, code, and observability components. - Resilience Hub automatically maps resource relationships into a topology showing data flow, containment, and permissions. ## Dependency Discovery - Resilience Hub analyzes VPC DNS query logs to identify dependencies that may not be documented. - It can discover: - AWS services - Internal endpoints - Third-party endpoints - Unexpected cross-Region calls - Critical external dependencies - Dependency discovery can be enabled or disabled for individual services. ## Generative AI Failure-Mode Assessments - Assessments analyze services against: - Configured resilience policies - AWS Well-Architected best practices - The AWS Resilience Analysis Framework - Findings explain the failure mode, its architectural impact, recommended remediation, and related policy requirement. - Users can add or modify assertions to guide the AI agents and improve assessment accuracy. - Findings can be marked as resolved after remediation or irrelevant when they do not apply. ## Getting Started and Enterprise Management - Users must configure an invoker IAM role granting read-only access to AWS resources. - Cross-account roles or AWS Organizations service-linked roles can support multi-account environments. - AWS Organizations integration allows a delegated administrator to assess resilience across the enterprise without signing into individual accounts. - A typical workflow is: - Create a resilience policy. - Create a system and service. - Associate resources through tags, CloudFormation, Terraform, or Amazon EKS. - Enable dependency discovery. - Run a failure-mode assessment. - Review and implement recommendations. ## Migration, Availability, and Pricing - Migration APIs convert older Resilience Hub policies and applications into the new model, including mapping multiple related applications into one system with multiple services. - The new version is generally available in supported AWS commercial Regions. - Pricing uses a service-based model and includes two failure-mode assessments per service per month, with automated dependency assessment available optionally. - AWS offers a free trial. The updated Resilience Hub is most useful for organizations that need consistent resilience standards and evidence across many AWS accounts and applications. Teams should begin by defining reusable policies, modeling critical services and dependencies, and using the AI assessments to prioritize remediation.

site-reliability-engineeringgen-aiterraforminfrastructure-as-code+4

lineApr 22, 2026

Implementing SLO/SLI for Improved Reliability Part 3 - Service Application Cases (opens in new tab)

SLI/SLO adoption is not merely a matter of choosing metrics; it requires redefining how a service is understood from the user’s perspective. LINE’s SRE team applies this approach by identifying critical user journeys, measuring reliability with clear criteria, and setting realistic objectives. The resulting data helps teams balance reliability, engineering costs, feature delivery, and incident response. ## The Mindset Behind SLI/SLO ### Understanding the Service and Users - Begin by identifying the services and features users depend on. - Map user journeys and select critical user journeys (CUJs) based on: - How frequently users use a feature - Whether the feature is essential to the service - Its relationship to business objectives - Aligning SLOs with business goals ensures that reliability work supports organizational priorities. ### Communication and Collaboration - SLI/SLOs should be defined and managed collaboratively rather than by a single team. - Product or service owners define CUJs because they understand the user experience best. - Infrastructure teams provide scalable systems for collecting and managing metrics. - SREs build the measurement tools and processes used to monitor and improve reliability. - Shared ownership is essential so SLOs can guide both daily operations and new feature launches. ## Implementing SLI/SLOs ### Analyze Critical User Journeys - List the services and functions provided to users. - Ask: - Which features are used most often? - Which features are indispensable? - LINE examples include: - Account registration - Sending and receiving messages - User authentication and encryption - LINE Login - Profile information - The goal is not to include every feature, but to select the most important ones from the user’s perspective. ### Define Service Level Indicators For each CUJ, determine: - **Measurement location:** Choose the point that best represents the user experience, such as a gateway, frontend, or backend. - **Measurement API:** Select a representative API to avoid unnecessarily complex calculations. - **Success criteria:** Establish clear boundaries between successful and failed requests. Common SLI criteria include: - **Latency:** Define a percentile, such as the 99.9th percentile, and the maximum acceptable response time. - **Success rate:** Define the required percentage of successful responses during the measurement period. For example, a messaging service might require 99.9% of requests to complete within 500 milliseconds and 99.999% of all requests to receive successful responses. If a CUJ cannot be measured reliably or its success criteria cannot be defined clearly, it may be excluded or supported with a dedicated measurement metric. ### Set SLO Targets - Define the reliability level the service must maintain over a specific period. - An example target is achieving the defined latency and success-rate criteria for 99.9% of a 28-day period. - Targets must be realistic: - Excessively high targets increase operational and infrastructure costs. - Excessively low targets can result in poor user experiences. - SLOs should balance reliability requirements with available resources. ### Visualize Reliability - Provide dashboards that allow all stakeholders to understand the current SLO status quickly. - Show overall SLO performance and error-budget consumption, with detailed dashboards for individual CUJs. - Keep dashboards simple and easy to scan rather than displaying excessive information. - Use visual indicators such as: - Green for healthy performance - Orange for warning conditions - Red for missed objectives ## How SLI/SLOs Are Used ### Quantifying Reliability - Replace vague descriptions such as “the service is slow” with measurable statements. - Teams can identify issues such as latency exceeding a 400-millisecond SLI threshold or success rates falling below 99.99%. - Dashboards also help correlate periods of poor performance with incidents or operational changes. ### Guiding Resource Allocation - SLOs show whether reliability targets are being met. - Error budgets indicate how much additional failure or downtime is acceptable. - When performance exceeds the SLO and the error budget is healthy, teams can invest more aggressively in: - New features - Faster release cycles - Product experimentation - When little error budget remains, resources can instead focus on prevention, remediation, and reliability improvements. ### Supporting On-Call Operations - LINE uses alerts triggered by changes in error-budget status to help on-call teams recognize and respond to service issues. - SLO reviews are also incorporated into regular meetings and preventive reliability work. SLI/SLO implementation works best as a shared, user-focused operating model. By combining clear CUJs, measurable criteria, realistic targets, and actionable dashboards, teams can make informed decisions about when to prioritize innovation and when to prioritize stability.

site-reliability-engineeringobservabilityresource-allocationservice-level-indicator+4

datadogJun 6, 2022

Engineering Spotlight: Tay Nishimura (opens in new tab)

Tay Nishimura’s career shows that succeeding in tech is often less about fitting a standard engineering mold and more about finding work that matches one’s strengths. Although she initially struggled with the speed and coding demands of software development, her rigor, visual thinking, and careful approach became valuable in site reliability engineering. Her transition was enabled by self-directed learning, community education, and ToyNet, an open source networking simulator that demonstrated her practical abilities. ## Entering Tech from Mathematics - Tay began as a mathematics major focused on real analysis, then added computer science after advice from a professor. - Internships at Amazon and Google introduced her to the technology industry. - She found a sharp contrast between academia and industry: - School rewarded theoretical rigor. - Industry emphasized practical, fast, and agile solutions. - Tay also felt like an outsider because she had little exposure to computers growing up. ## Struggling with Traditional Software Engineering - Coding did not come naturally to Tay’s visual way of thinking. - She translated code into drawings to understand and modify it, then converted those ideas back into code. - This process produced high-quality, careful work but made her slower than colleagues expected. - A manager suggested product management and site reliability engineering as possible alternatives. - Tay discovered that her deliberate pace was useful for SRE work, particularly when evaluating failure modes and making critical changes. - Because her company offered no path into those roles, she eventually left rather than continue facing increasing stress. ## Discovering Networking and Technical Program Work - Tay’s next role had a software engineer title but involved work closer to product or technical program management. - She learned that job titles and responsibilities vary significantly between companies. - With better work-life balance, she began studying computer networking in her free time. - She created visual diagrams and learning modules to explain switches, routers, and packet flows. - These efforts became Project Reclass, a nonprofit teaching technical skills to incarcerated people and military veterans. - The program used improvised equipment, such as fake routers and switches, to teach concepts in environments where real networking hardware was unavailable. ## Building ToyNet During the Pandemic - After her company laid off its entire office during COVID-19, Tay decided to pursue SRE directly. - When prisons suspended in-person education, Project Reclass adapted by creating a digital networking simulator. - Tay architected ToyNet, an open source platform built with: - React - A Flask backend - Containerized Mininet instances for network emulation - Users can connect simulated routers, switches, and hosts, configure IP addresses, and run commands such as `ping` and `arp`. - ToyNet was designed to work for incarcerated learners with restricted internet access. - Deploying it in the cloud also gave Tay practical experience that helped compensate for limited professional cloud experience. - Companies interested in the project were more likely to advance her through the interview process, eventually leading to Datadog. ## Finding the Right Environment at Datadog - At Datadog, Tay learned Kubernetes, chaos engineering, network traffic control, and Go. - She found that her rigor and visual thinking were assets rather than liabilities. - While learning Datadog’s Chaos Controller codebase, she mapped files and dependencies by drawing boxes and arrows. - Her experience suggests that engineers do not need to learn or reason in a single conventional way; the right environment can turn an apparent weakness into a strength. Tay’s path recommends experimenting broadly, studying independently, and building concrete projects that reveal how you think and solve problems. The most suitable tech role may emerge only after moving between companies and disciplines rather than forcing yourself to succeed in an ill-fitting position.

site-reliability-engineeringk8sreactopen-source+4