Managed DevOps Pools – The Origin Story
Microsoft’s vast, diverse engineering organization had accumulated more than 5,000 self-hosted Azure DevOps pools, creating duplicated tooling, inconsistent reliability, security gaps, and compliance challenges. Its One Engineering System (1ES) team addressed this with 1ES Hosted Pools, a standardized service for flexible, secure, and scalable CI/CD infrastructure. Adoption reduced costs by more than 60%, cut remaining self-hosted pools to a few dozen, and eventually led to the external Managed DevOps Pools offering. ## The Scale and Challenges of Self-Hosted Infrastructure - Microsoft supports over 100,000 engineers across many businesses, programming languages, operating systems, hardware platforms, build engines, and test frameworks. - By 2021, teams had created: - More than 5,000 self-hosted Azure DevOps pools - Hundreds of thousands of agents - Teams needed capabilities unavailable from Microsoft-hosted agents, including: - Larger compute sizes - Private-network connectivity - Custom images - Stateful agents - Long-running tests - The decentralized approach caused: - Duplicate engineering effort - Uneven support and reliability - Poor resource utilization and higher costs - Inconsistent patching and security practices - Difficult and time-consuming compliance audits ## 1ES Hosted Pools - 1ES developed a standardized internal service for custom Azure DevOps infrastructure. - Teams could connect agents to private resources such as package registries, secret managers, and on-premises services. - They could bring custom images, using centrally maintained images as their base. - Business continuity features allowed backup pools and failover to other Azure regions. - Agents were stateless by default, but teams could reuse stateful agents for better performance through local caches. - Stateful agents were automatically recycled based on age or available disk space. - Teams could select Azure VM families and sizes suited to their workload. - Standby agents could be pre-warmed on schedules or automatically provisioned using historical demand. ## Operational and Business Benefits - **Lower costs:** Infrastructure bills fell by more than 60% through improved utilization, better SKU selection, and selective use of Azure Spot VMs. - **Faster development:** Teams spent less time maintaining CI/CD infrastructure and more time building products. - **Simpler compliance:** Standardized telemetry made audits easier and allowed security and compliance improvements to be deployed centrally. - **Greater mobility:** Developers changing teams no longer had to learn different infrastructure-management systems. - **Improved security:** Features such as Azure Confidential VMs, Trusted Launch, and Secure TPM became available across pools. - **Reduced fragmentation:** By 2024, Microsoft had reduced its remaining self-hosted pools from more than 5,000 to only a few dozen. ## From Internal Platform to Managed DevOps Pools - 1ES first built Hosted Pools as an internal “Host On Behalf Of” service to validate whether centralized management could reduce self-hosting. - Success inside Microsoft, combined with customer demand, led to the external **Managed DevOps Pools (MDP)** service. - Organizations using VM Scale Set agents or self-hosted agents can migrate to MDP to gain standardized scaling, security, compliance, and operational support. - The external offering initially does not include every feature available in 1ES Hosted Pools, though additional capabilities may be added later. Centralizing CI/CD infrastructure can eliminate redundant platform work while improving cost efficiency, security, compliance, and developer productivity. Managed DevOps Pools extends Microsoft’s internal solution to organizations facing similar self-hosting challenges.
Read original(opens in new tab)