How We Built an SRE Bot That Reduced Our Team’s Repetitive Work by 90%
LINE Home DevOps created an SRE bot to reduce the repetitive work caused by growing services, Flava cloud migration, and increasing developer requests. By making Slack the central interface and automating Jira, Confluence, and workflow updates, the team reduced deployment-request handling from roughly 30 minutes to under one minute. The bot also improved tracking, consistency, and response speed, helping SREs move away from constant firefighting. ## Repetitive SRE Work and Its Costs - Developers frequently asked how to inspect Flava pod logs, request permissions, interpret errors, and access staging environments. - Deployment requests required manual movement between Slack, Confluence, and Jira: - Finding release checklists - Copying information into Jira - Creating missing Fix Versions - Linking Epics and active sprints - Sharing ticket links and deployment documentation - Each deployment request previously took about 30 minutes to an hour. - Manual processing caused omissions and mistakes, especially during urgent releases. - General requests were buried in Slack mentions, making ownership and completion status difficult to track. - Measurement showed that each SRE spent nearly half a day per week on repetitive work. ## Slack-Centered Automation The team adopted the principle that developers should only need Slack, while SREs should be able to manage work with a few clicks. - **Slack as the single source of truth:** Requests begin and remain trackable in Slack. - **Zero manual work:** Rule-based Jira and documentation tasks are automated. - **Immediate visibility:** Status changes and results are posted to Slack in real time. - **Permission control:** Only authorized SRE members can claim or complete requests. ## Key Technical Decisions ### Slack Workflows Instead of Slash Commands - Slash commands are easy to implement but depend on users entering correctly formatted text. - Slack Workflows provide structured forms with required-field validation. - Because Workflows are native Slack functionality, the team avoided building a separate user interface. - The lower usage barrier made adoption more likely. ### Asynchronous Processing - Slack requires event responses within three seconds. - Sequential calls to Jira, Confluence, and other APIs could exceed that limit. - The bot immediately acknowledges the request, then performs external work in the background. - Successes and failures are reported in the Slack thread, keeping processing transparent. ### Redis-Based State Management - In-memory state would be lost whenever the bot restarted. - Slack metadata APIs were considered too slow for real-time interactions such as emoji clicks. - Redis was selected for sub-100-millisecond lookups and persistent state. - A 30-day TTL limits stale data. - Redis transactions using `WATCH/MULTI/EXEC` ensure consistent updates when multiple SREs interact simultaneously. ### Hexagonal Architecture - The bot uses ports and adapters to isolate business logic from external systems. - The architecture separates: - Inbound Slack event adapters - Application use cases and business logic - Outbound Jira, Confluence, and Redis adapters - External API or SDK changes can be handled without modifying core business logic. - This structure also makes testing and future feature development easier. ## Automated Request Scenarios ### Deployment Requests - Developers submit required project, release-version, checklist, and other details through a Slack Workflow. - The bot automatically: - Creates a missing Jira Fix Version - Creates and configures the Jira ticket - Links the Epic - Adds the ticket to the active sprint - Finds the relevant deployment manual - Posts the result to the Slack thread - An SRE can click 👀 to claim the work. - Clicking ✅ completes the Jira ticket and posts a completion notification. - SRE effort falls from about 30 minutes to under one minute, with minimal risk of missing required fields. ### Emergency Deployments - Selecting an urgent request automatically sets Jira Priority to `Highest`. - The bot immediately announces the request in Slack. - An SRE can claim it with 👀, perform the deployment, and complete it with ✅. - The process reduces delays from roughly 30–40 minutes to about one minute. ### General SRE Requests - Requests such as production-access permissions are submitted through a structured Slack Workflow. - The bot creates a Jira ticket, links the Epic, assigns the active sprint, and sets an appropriate priority. - Slack retains the ticket link and status, eliminating the need to search through message history later. - SREs claim and complete the request using the same emoji-based workflow. The main recommendation is to automate repetitive, rule-based operations at the point where requests already occur. A Slack-centered, asynchronous bot with durable state and clean system boundaries can reduce manual effort while making ownership, progress, and completion visible to everyone.
Read original(opens in new tab)