techlist.io
KREN
microsoft

Managed DevOps Pools – The Origin Story (opens in new tab)

ci-cddevopscloud-infrastructureazureazure-devopsmanaged-devops-poolsinfrastructure-management

Microsoft’s vast, diverse engineering organization had accumulated more than 5,000 self-hosted Azure DevOps pools, creating duplicated tooling, inconsistent reliability, security gaps, and compliance challenges. Its One Engineering System (1ES) team addressed this with 1ES Hosted Pools, a standardized service for flexible, secure, and scalable CI/CD infrastructure. Adoption reduced costs by more than 60%, cut remaining self-hosted pools to a few dozen, and eventually led to the external Managed DevOps Pools offering.

The Scale and Challenges of Self-Hosted Infrastructure

  • Microsoft supports over 100,000 engineers across many businesses, programming languages, operating systems, hardware platforms, build engines, and test frameworks.
  • By 2021, teams had created:
    • More than 5,000 self-hosted Azure DevOps pools
    • Hundreds of thousands of agents
  • Teams needed capabilities unavailable from Microsoft-hosted agents, including:
    • Larger compute sizes
    • Private-network connectivity
    • Custom images
    • Stateful agents
    • Long-running tests
  • The decentralized approach caused:
    • Duplicate engineering effort
    • Uneven support and reliability
    • Poor resource utilization and higher costs
    • Inconsistent patching and security practices
    • Difficult and time-consuming compliance audits

1ES Hosted Pools

  • 1ES developed a standardized internal service for custom Azure DevOps infrastructure.
  • Teams could connect agents to private resources such as package registries, secret managers, and on-premises services.
  • They could bring custom images, using centrally maintained images as their base.
  • Business continuity features allowed backup pools and failover to other Azure regions.
  • Agents were stateless by default, but teams could reuse stateful agents for better performance through local caches.
  • Stateful agents were automatically recycled based on age or available disk space.
  • Teams could select Azure VM families and sizes suited to their workload.
  • Standby agents could be pre-warmed on schedules or automatically provisioned using historical demand.

Operational and Business Benefits

  • Lower costs: Infrastructure bills fell by more than 60% through improved utilization, better SKU selection, and selective use of Azure Spot VMs.
  • Faster development: Teams spent less time maintaining CI/CD infrastructure and more time building products.
  • Simpler compliance: Standardized telemetry made audits easier and allowed security and compliance improvements to be deployed centrally.
  • Greater mobility: Developers changing teams no longer had to learn different infrastructure-management systems.
  • Improved security: Features such as Azure Confidential VMs, Trusted Launch, and Secure TPM became available across pools.
  • Reduced fragmentation: By 2024, Microsoft had reduced its remaining self-hosted pools from more than 5,000 to only a few dozen.

From Internal Platform to Managed DevOps Pools

  • 1ES first built Hosted Pools as an internal “Host On Behalf Of” service to validate whether centralized management could reduce self-hosting.
  • Success inside Microsoft, combined with customer demand, led to the external Managed DevOps Pools (MDP) service.
  • Organizations using VM Scale Set agents or self-hosted agents can migrate to MDP to gain standardized scaling, security, compliance, and operational support.
  • The external offering initially does not include every feature available in 1ES Hosted Pools, though additional capabilities may be added later.

Centralizing CI/CD infrastructure can eliminate redundant platform work while improving cost efficiency, security, compliance, and developer productivity. Managed DevOps Pools extends Microsoft’s internal solution to organizations facing similar self-hosting challenges.