Capacity Efficiency at Meta: How Unified AI Agents Optimize Performance at Hyperscale
Meta’s Capacity Efficiency Program uses AI agents to automate both the discovery and resolution of infrastructure performance issues. By combining standardized tools with encoded expertise from senior efficiency engineers, the platform turns investigations that once took hours into minutes and has recovered hundreds of megawatts of power. The approach aims to let Meta scale efficiency improvements across more product areas without proportionally increasing engineering headcount. ## Capacity Efficiency at Hyperscale - At Meta’s scale, even a 0.1% performance regression can significantly increase power consumption across systems serving more than 3 billion people. - The program has two complementary functions: - **Offense:** Proactively identify and implement optimizations. - **Defense:** Detect production regressions, identify their causes, and deploy mitigations. - Human investigation is often the bottleneck, requiring engineers to analyze profiling data, review documentation and prior fixes, inspect deployments, and search internal discussions. - AI automation can reduce roughly 10 hours of manual diagnosis to about 30 minutes. ## A Unified Platform for AI Efficiency Agents - Meta built one platform for both offensive and defensive workflows because they share the same basic structure: - Gather relevant technical context. - Apply domain-specific reasoning. - Produce a code change for review. - **MCP tools** provide standardized interfaces for querying profiling data, retrieving experiment results, examining configuration history, searching code, and accessing documentation. - **Skills** encode expert reasoning, including which tools to use and how to interpret their results. - The same tools support both use cases, while specialized skills handle different optimization and regression scenarios. ## Defense: Automated Regression Resolution - FBDetect monitors noisy production time series and can identify regressions as small as 0.005%. - Traditional root-cause analysis correlates the regression with recent pull requests or configuration changes. - Previously, teams often rolled back problematic changes—reducing engineering velocity—or left them unresolved, allowing resource waste to accumulate. - The AI Regression Solver: - Identifies affected functions and regression symptoms. - Locates the responsible pull request, files, and changed lines. - Applies mitigation expertise appropriate to the codebase, language, or regression type. - Generates a corrective pull request and sends it to the original author for review. - Faster resolution prevents small regressions from compounding across Meta’s infrastructure. ## Offense: Converting Opportunities into Code - Efficiency opportunities describe potential improvements to existing code, but implementing them traditionally required substantial investigation and engineering time. - Meta’s AI workflow gathers: - Opportunity metadata. - Optimization documentation. - Examples of similar fixes. - Relevant files and functions. - Validation criteria. - Skills then apply specialized knowledge, such as memoizing a function to reduce CPU usage. - The agent generates a guarded candidate fix, checks syntax and style, validates that it addresses the intended issue, and presents the change in an engineer’s editor for review or one-click application. - This expands the number of optimization opportunities engineers can pursue manually. ## Scaling Efficiency with AI - The platform has already recovered hundreds of megawatts of power—enough to supply hundreds of thousands of U.S. homes for a year. - Automated regression handling reduces ongoing waste, while automated opportunity resolution increases the volume of proactive improvements. - The long-term goal is a self-sustaining efficiency engine in which AI handles the long tail of investigations and fixes, allowing engineers to focus on new products and higher-value work. Meta’s approach recommends treating performance expertise as reusable, composable software: standardize access to engineering data, encode proven reasoning into skills, and let agents carry issues from detection through ready-to-review code changes.
Read original(opens in new tab)