Curated summary
Utilizing SLI/SLO to Improve Reliability Part 1: SLI/SLO Framework and the Development Story of Service Status Check Tool LINE Status
Repeated SLI/SLO adoption revealed a common process that could be standardized across services. The team turned that process into a reusable framework and built “LINE Status,” an internal tool that automatically presents service health according to user experience rather than raw alerts. Together, these initiatives create a shared organizational language for understanding reliability and its impact on users.
A Reusable SLI/SLO Framework
After applying SLI/SLOs to several platforms and services, the SRE team identified recurring patterns independent of service type. They organized these patterns into a five-stage framework:
Select critical user journeys (CUJs) and define SLIs
- Identify the experiences most important to users.
- Define measurable SLIs that represent those experiences.
Design instrumentation and metrics
- Build or adapt metrics suitable for each CUJ.
- Use standardized naming based on Prometheus or OpenTelemetry.
Create dashboards and recording rules
- Provide Grafana dashboards for quickly assessing SLO achievement.
- Precompute complex PromQL operations to improve query performance.
Set SLOs and alerts
- Begin with flexible targets, such as 99.9% availability over a 28-day rolling window, allowing roughly 40 minutes of downtime.
- Define runbooks for responding to alerts.
- Refine targets after operational data and experience accumulate.
Establish error-budget governance
- Balance release speed against reliability.
- Review objectives monthly or quarterly.
- Adjust SLOs and processes as needed.
The framework is currently distributed as a Confluence template containing guidance and FAQs, reducing the communication effort required from SREs during initial adoption.
Moving from Alerts to User-Centered Service Status
As more services adopted SLI/SLOs, the team wanted a consistent way to understand the health of services they did not directly operate.
- The existing public LINE Status API page focused on external users and was updated manually during major incidents.
- The new internal tool was intended to:
- Represent the status of individual service components.
- Update automatically from SLI/SLO alerts and outage data.
- Show whether user experience was being affected.
- Rather than simply reflecting whether an alert or outage existed, status was based on CUJ-related SLI performance and SLO achievement.
- Only representative, high-value CUJs were exposed, avoiding unnecessary technical detail.
LINE Status Architecture and Interface
LINE Status was designed as more than an alert list. It collects events through webhooks, stores them in a separate database, and uses that data to track both current status and historical changes.
- Technical SLI/SLO terms are translated into user-facing functions such as “Message Sending” or “Read Receipts.”
- Status colors provide an immediate overview:
- Green: normal
- Yellow: event detected
- Red: outage
- The main page provides:
- An overview of all services.
- CUJ status within each service card.
- AI-generated one-line summaries.
- Service detail pages provide:
- Recently affected items near the top.
- Timeline-based event displays.
- Monthly historical events.
- The history page shows:
- The scope of impact for each service during an event.
- Past events organized by month.
The initial implementation took about a month and was refined through colleague feedback. The author also used AI-assisted “vibe coding” for the frontend, emphasizing that clear, detailed requirements were more important than the development tool itself.
Connecting the Framework and LINE Status
Once a service adopts SLI/SLOs through the framework, it can be registered in LINE Status. This connects the definition of reliability objectives with an organization-wide view of service health.
- Developers and operators can use the same CUJ-based standards.
- Teams can focus on whether users are affected instead of interpreting isolated alerts.
- During incidents, the tool helps identify impacted experiences quickly.
- Over time, the approach may improve decision-making speed and cross-team communication.
The team plans to refine CUJs, SLIs, and status-transition rules through continued operational experience.
The practical goal is to make SLI/SLOs a common language for describing service health, enabling reliability practices to scale without depending heavily on individual teams or specialists.
Related reading
Continue with another curated summary.
How to build CI/CD observability at scale
Read originalFrom Custom to Open: Scalable Network Probing and HTTP/3 Readiness with Prometheus
Read originalTwenty years of Amazon S3 and building what’s next | Amazon Web Services
Read originalSee risk, fix risk: introducing Remediation in Cloudflare CASB
Read original