DevOps | Techlist.io

lineFeb 13, 20263 min readCurated summary

Claude Code Action: Platformizing AI Code

LINE NEXT transformed Claude Code from an individual productivity tool into an organization-wide code review platform integrated with GitHub Actions. The goal was to reduce review-quality variation, standardize policies, and make AI feedback part of the existing pull request workflow. Its central design separates simple repository-level invocation from centrally managed execution, prompts, permissions, and infrastructure. ## Why AI Code Review Needed to Be Platformized - As LINE NEXT’s services and repositories grew, human code review quality varied according to each reviewer’s experience and preferences. - Developers were already using Claude Code locally, but individual usage created several problems: - Inconsistent review criteria and perspectives - No organization-wide quality process - AI feedback disconnected from pull request workflows - Difficulty providing new employees with a consistent review experience - DevOps therefore treated the issue as a decentralized quality-process problem rather than merely a tooling problem. ## Why GitHub Actions and Claude Code - GitHub Actions was already the foundation for CI/CD and automation across LINE NEXT repositories. - It allowed the team to: - Apply a common workflow repository by repository - Centrally manage execution environments and permissions - Avoid requiring each service team to build additional infrastructure - Claude Code Action integrated directly with pull requests: - Developers could trigger reviews with an `@claude` mention. - Results appeared as GitHub comments or PR reviews. - Developers did not need to learn a separate interface. - A shared GitHub App Runner environment provided consistent execution and centralized security controls. ## Centralized Caller–Executor Architecture - Service repositories act as **callers**: - They invoke the standard workflow. - They provide only basic parameters such as service name and review type. - A centrally managed DevOps repository acts as the **executor**: - Stores prompts and review personas - Defines review policies and priorities - Manages permissions and authentication - Contains the actual execution logic - This design makes AI review an organization-wide platform capability rather than a separate configuration maintained by every project. ### Benefits of Central Control - **Consistent quality:** Central prompts and personas ensure common review depth, tone, security checks, stability checks, and priorities. - **Faster adoption:** New repositories need only add the standard workflow and specify a few parameters. - **Improved governance:** GitHub Apps, centrally managed secrets, and shared runners make it possible to track who accessed which code and with what permissions. - **Lower operational overhead:** Service teams use the platform without managing AI infrastructure themselves. ## Handling Fork-Based Pull Requests - The official Claude Code Action initially assumed that a PR branch existed in the base repository’s `origin`. - For pull requests created from forks, this caused failures such as: ```text couldn't find remote ref ``` - The original implementation fetched and checked out the branch by name: ```text git fetch origin <branch> git checkout <branch> ``` - This failed because fork branches exist in the external repository, not necessarily in the base repository. - From a platform perspective, this was a structural limitation because it blocked external contributors and collaboration repositories. - The proposed direction was to redesign the execution flow rather than simply add an exception, using GitHub’s special pull-request reference: ```text refs/pull/<PR number>/head ``` This approach allows the workflow to retrieve the actual pull request head commit regardless of whether the PR originated from the main repository or a fork.

Read original

DevOps Database Design Kubernetes CI/CD+4

tossJan 20, 2026Original article

The story of how I destroyed (opens in new tab)

Toss Payments modernized its inherited legacy infrastructure by building an OpenStack-based private cloud to operate alongside public cloud providers in an Active-Active hybrid configuration. By overcoming extreme technical debt—including servers burdened with nearly 2,000 manual routing entries—the team achieved a cloud-agnostic deployment environment that ensures high availability and cost efficiency. The transformation demonstrates how a small team can successfully implement complex open-source infrastructure through automation and the rigorous technical internalization of Cluster API and OpenStack. ### The Challenge of Legacy Networking - The inherited infrastructure relied on server-side routing rather than network equipment, meaning every server carried its own routing table. - Some legacy servers contained 1,997 individual routing entries, making manual management nearly impossible and preventing efficient scaling. - Initial attempts to solve this via public cloud (AWS) faced limitations, including rising costs due to exchange rates, lack of deep visibility for troubleshooting, and difficulties in disaster recovery (DR) configuration between public and on-premise environments. ### Scaling OpenStack with a Two-Person Team - Despite having only two engineers with no prior OpenStack experience, the team chose the open-source platform to maintain 100% control over the infrastructure. - The team internalized the technology by installing three different versions of OpenStack dozens of times and simulating various failure scenarios. - Automation was prioritized using Ansible and Terraform to manage the lifecycle of VMs and load balancers, enabling new instance creation in under 10 seconds. - Deep technical tuning was applied, such as modifying the source code of the Octavia load balancer to output custom log formats required for their specific monitoring needs. ### High Availability and Monitoring Strategy - To ensure reliability, the team built three independent OpenStack clusters operating in an Active-Active configuration. - This architecture allows for immediate traffic redirection if a specific cluster fails, minimizing the impact on service availability. - A comprehensive monitoring stack was implemented using Zabbix, Prometheus, Mimir, and Grafana to collect and visualize every essential metric across the private cloud. ### Managing Kubernetes with Cluster API - To replicate the convenience of Public Cloud PaaS (like EKS), the team implemented Cluster API to manage the Kubernetes lifecycle. - Cluster API treats Kubernetes clusters themselves as resources within a management cluster, allowing for standardized and rapid deployment across the private environment. - This approach ensures that developers can deploy applications without needing to distinguish between the underlying cloud providers, fulfilling the goal of "cloud-agnostic" infrastructure. ### Practical Recommendation For organizations dealing with massive technical debt or high public cloud costs, the Toss Payments model suggests that a "Private-First" hybrid approach is viable even with limited headcount. The key is to avoid proprietary black-box solutions and instead invest in the technical internalization of open-source tools like OpenStack and Cluster API, backed by a "code-as-infrastructure" philosophy to ensure scalability and reliability.

DevOps Kubernetes Infrastructure As Code High Availability+4

naverNov 27, 2025Original article

Naver TV (opens in new tab)

Naver’s Integrated Search team is transitioning from manual fault response to an automated system using LLM Agents to manage the increasing complexity of search infrastructure. By integrating Large Language Models into the DevOps pipeline, the system evolves through accumulated experience, moving beyond simple alert monitoring to intelligent diagnostic analysis and action recommendation. ### Limitations of Traditional Fault Response * **Complex Search Flows:** Naver’s search architecture involves multiple interdependent layers, which makes manual root cause analysis slow and prone to human error. * **Fragmented Context:** Existing monitoring requires developers to manually synthesize logs and metrics from disparate telemetry sources, leading to high cognitive load during outages. * **Delayed Intervention:** Human-led responses often suffer from a "detection-to-action" lag, especially during high-traffic periods or subtle service regressions. ### Architecture of DevOps Agent v1 * **Initial Design:** Focused on automating basic data gathering and providing preliminary textual reports to engineers. * **Infrastructure Integration:** Built using a specialized software stack designed to bridge frontend (FE) and backend (BE) telemetry within the search infrastructure. * **Standardized Logic:** The v1 agent operated on a fixed set of instructions to perform predefined diagnostic tasks when triggered by specific system alarms. ### Evolution to DevOps Agent v2 * **Overcoming V1 Limitations:** The first iteration struggled with maintaining deep context and providing diverse actionable insights, necessitating a more robust agentic structure. * **Enhanced Memory and Learning:** V2 incorporates a more sophisticated architecture that allows the agent to reference historical failure data and learn from past incident resolutions. * **Advanced Tool Interaction:** The system was upgraded to handle more complex tool-calling capabilities, allowing the agent to interact more deeply with internal infrastructure APIs. ### System Operations and Evaluation * **Trigger Queue Management:** Implements a queuing system to efficiently process and prioritize multiple concurrent system alerts without overwhelming the diagnostic pipeline. * **Anomaly Detection:** Utilizes advanced detection methods to distinguish between routine traffic fluctuations and genuine service anomalies that require LLM intervention. * **Rigorous Evaluation:** The agent’s performance is measured through a dedicated evaluation framework that assesses the accuracy of its diagnoses against known ground-truth incidents. ### Scaling and Future Challenges * **Context Expansion:** Efforts are focused on integrating a wider range of metadata and environmental context to provide a holistic view of system health. * **Action Recommendation:** The system is moving toward suggesting specific recovery actions, such as rollbacks or traffic rerouting, rather than just identifying the problem. * **Sustainability:** Ensuring the DevOps Agent remains maintainable and cost-effective as the underlying search infrastructure and LLM models continue to evolve. Organizations managing high-scale search traffic should consider LLM-based agents as integrated infrastructure components rather than standalone tools. Moving from reactive monitoring to a proactive, experience-based agent system is essential for reducing the mean time to recovery (MTTR) in complex distributed environments.

DevOps AI Large Language Models Incident Management+3

lineJul 16, 2025Original article

LY's Tech Conference, ' (opens in new tab)

LY Corporation’s Tech-Verse 2025 conference highlighted the company's strategic pivot toward becoming an AI-centric organization through the "Catalyst One Platform" initiative. By integrating the disparate infrastructures of LINE and Yahoo! JAPAN into a unified private cloud, the company aims to achieve massive cost efficiencies while accelerating the deployment of AI agents across its entire service ecosystem. This transformation focuses on empowering engineers with AI-driven development tools to foster rapid innovation and deliver a seamless, "WOW" experience for global users. ### Infrastructure Integration and the Catalyst One Platform To address the redundancies following the merger of LINE and Yahoo! JAPAN, LY Corporation is consolidating its technical foundations into a single internal ecosystem known as the Catalyst One Platform. * **Private Cloud Advantage:** The company maintains its own private cloud to achieve a four-fold cost reduction compared to public cloud alternatives, managed by a lean team of 700 people supporting 500,000 servers. * **Unified Architecture:** The integration spans several layers, including Infrastructure (Project "DC-Hub"), Cloud (Project "Flava"), and specialized Data and AI platforms. * **Next-Generation Cloud "Flava":** This platform integrates existing services to enhance VM specifications, VPC networking, and high-performance object storage (Ceph and Dragon). * **Information Security:** A dedicated "SafeOps" framework is being implemented to provide governance and security across all integrated services, ensuring a safer environment for user data. ### AI Strategy and Service Agentization A core pillar of LY’s strategy is the "AI Agentization" of all its services, moving beyond simple features to proactive, personalized assistance. * **Scaling GenAI:** Generative AI has already been integrated into 44 different services within the group. * **Personalized Agents:** The company is developing the capacity to generate millions of specialized agents that can be linked together to support the unique needs of individual users. * **Agent Ecosystem:** The goal is to move from a standard platform model to one where every user interaction is mediated by an intelligent agent. ### AI-Driven Development Transformation Beyond user-facing services, LY is fundamentally changing how its engineers work by deploying internal AI development solutions to all staff starting in July. * **Code and Test Automation:** Proof of Concept (PoC) results showed a 96% accuracy rate for "Code Assist" and a 97% reduction in time for "Auto Test" procedures. * **RAG Integration:** The system utilizes Retrieval-Augmented Generation (RAG) to leverage internal company knowledge and guidelines, ensuring high-quality, context-aware development support. * **Efficiency Gains:** By automating repetitive tasks, the company intends for engineers to shift their focus from maintenance to creative service improvement and innovation. The successful integration of these platforms and the aggressive adoption of AI-driven development tools suggest that LY Corporation is positioning itself to be a leader in the "AI-agent" era. For technical organizations, LY's model serves as a case study in how large-scale mergers can leverage private cloud infrastructure to fund and accelerate a company-wide AI transition.

DevOps AI AI Agents Generative AI+4

figmaAug 12, 20243 min readCurated summary

The Atlassian Method: The Power of Developer Joy | Figma Blog

Atlassian treats “developer joy” as more than a smoother developer experience: it is a company-wide philosophy centered on reducing friction and protecting the craft of development. After poor productivity and satisfaction in 2022, Atlassian standardized tools, improved processes, and empowered engineering teams to address frustrations directly. The effort produced major gains, including higher satisfaction, faster pull-request cycles, more frequent deployments, and improved roadmap delivery. ## Developer Joy as a Company Priority - Developer experience typically covers workflows, tools, and processes; developer joy focuses more deeply on the values, standards, and craft of development. - Atlassian found that developers lose more than eight hours per week—about 20% of their time—to inefficiencies. - Common sources of friction include: - Searching for information and solutions - Poor tooling and redundant systems - Cross-functional coordination with design and other teams - Planning and process overhead - In 2022, Atlassian frequently missed public roadmap commitments, while developer satisfaction fell below 50%. ## Operationalizing Joy - Atlassian created a cross-functional “champions” program to identify and eliminate organizational frustrations. - The initiative focused on: - **Systems:** Auditing tools and processes, standardizing where possible, and removing redundant tools—the company’s “Noah’s Ark of tooling.” - **Culture:** Establishing coding standards, shared values, and metrics that emphasized quality and craft, not just efficiency. - Every engineering team allocated 10% of its time to developer productivity improvements. - This gave developers an “ownership stake” in solving the problems affecting their work. - Developer joy became a company-wide OKR reported by teams every month. ## Measuring the Business Value - Atlassian accepted short-term tradeoffs, prioritizing engagement and productivity improvements over immediate revenue optimization. - The company measured both quantitative and qualitative outcomes. - Within a few months, it achieved: - A 50% increase in developer satisfaction - A 50% reduction in median pull-request cycle time - A threefold increase in deployment frequency - Delivery of all customer roadmap commitments instead of repeated delays - An increase in internal CSAT from below 50% to 80% - The results reinforced the idea that investments in tools, processes, and clear measurement compound over time. ## Toward Team Joy - The success of developer joy helped Atlassian build broader support for applying the same principles beyond individual developers. - The initiative began moving toward the larger concept of “team joy,” extending the focus to collaboration and the shared experience of delivering work. Organizations seeking similar results should treat developer productivity as an ongoing company responsibility: give teams dedicated time to improve their systems, reduce unnecessary complexity, and measure satisfaction alongside delivery performance.

Read original

DevOps Software Development Developer Experience Agile+3

microsoftJul 18, 20243 min readCurated summary

Managed DevOps Pools – The Origin Story

Microsoft’s vast, diverse engineering organization had accumulated more than 5,000 self-hosted Azure DevOps pools, creating duplicated tooling, inconsistent reliability, security gaps, and compliance challenges. Its One Engineering System (1ES) team addressed this with 1ES Hosted Pools, a standardized service for flexible, secure, and scalable CI/CD infrastructure. Adoption reduced costs by more than 60%, cut remaining self-hosted pools to a few dozen, and eventually led to the external Managed DevOps Pools offering. ## The Scale and Challenges of Self-Hosted Infrastructure - Microsoft supports over 100,000 engineers across many businesses, programming languages, operating systems, hardware platforms, build engines, and test frameworks. - By 2021, teams had created: - More than 5,000 self-hosted Azure DevOps pools - Hundreds of thousands of agents - Teams needed capabilities unavailable from Microsoft-hosted agents, including: - Larger compute sizes - Private-network connectivity - Custom images - Stateful agents - Long-running tests - The decentralized approach caused: - Duplicate engineering effort - Uneven support and reliability - Poor resource utilization and higher costs - Inconsistent patching and security practices - Difficult and time-consuming compliance audits ## 1ES Hosted Pools - 1ES developed a standardized internal service for custom Azure DevOps infrastructure. - Teams could connect agents to private resources such as package registries, secret managers, and on-premises services. - They could bring custom images, using centrally maintained images as their base. - Business continuity features allowed backup pools and failover to other Azure regions. - Agents were stateless by default, but teams could reuse stateful agents for better performance through local caches. - Stateful agents were automatically recycled based on age or available disk space. - Teams could select Azure VM families and sizes suited to their workload. - Standby agents could be pre-warmed on schedules or automatically provisioned using historical demand. ## Operational and Business Benefits - **Lower costs:** Infrastructure bills fell by more than 60% through improved utilization, better SKU selection, and selective use of Azure Spot VMs. - **Faster development:** Teams spent less time maintaining CI/CD infrastructure and more time building products. - **Simpler compliance:** Standardized telemetry made audits easier and allowed security and compliance improvements to be deployed centrally. - **Greater mobility:** Developers changing teams no longer had to learn different infrastructure-management systems. - **Improved security:** Features such as Azure Confidential VMs, Trusted Launch, and Secure TPM became available across pools. - **Reduced fragmentation:** By 2024, Microsoft had reduced its remaining self-hosted pools from more than 5,000 to only a few dozen. ## From Internal Platform to Managed DevOps Pools - 1ES first built Hosted Pools as an internal “Host On Behalf Of” service to validate whether centralized management could reduce self-hosting. - Success inside Microsoft, combined with customer demand, led to the external **Managed DevOps Pools (MDP)** service. - Organizations using VM Scale Set agents or self-hosted agents can migrate to MDP to gain standardized scaling, security, compliance, and operational support. - The external offering initially does not include every feature available in 1ES Hosted Pools, though additional capabilities may be added later. Centralizing CI/CD infrastructure can eliminate redundant platform work while improving cost efficiency, security, compliance, and developer productivity. Managed DevOps Pools extends Microsoft’s internal solution to organizations facing similar self-hosting challenges.

Read original

DevOps CI/CD Cloud Infrastructure Azure+3