Lights Out, Systems On: Validating Instant Power Loss Readiness (opens in new tab)
Meta’s Instantaneous PowerLoss Storm is a disaster-readiness testing program designed to prepare data centers for sudden, zero-notice power loss. It extends existing fault-tolerance mechanisms across facilities, servers, storage, compute, and the Twine orchestrator, while addressing region-wide failures and autonomous recovery. Through incremental testing and carefully defined tradeoffs, Meta aims to make losing an entire region as manageable as losing a smaller fault domain. ## Defense-in-Depth for Instant Failures - Power-loss tolerance was built into the full data-center stack, including mechanical and electrical systems, server racks, storage, compute, and Twine. - Batteries and Power Loss Siren (PLS) preserve in-memory data when racks lose power. - Twine services use region-wide asynchronous unavailability events (UEs) to coordinate shutdown and recovery. - Existing mechanisms had been tested against smaller fault domains, but region-wide failures introduced new challenges involving scale, replica placement, and autonomous startup. ## Solving Region Bootstrap Problems - Restarting a region may require millions of services to start simultaneously and discover their dependencies. - Circular dependencies among Twine control-plane services—such as Scheduler, Allocator, Broker, and Zelos—could prevent the orchestrator from starting itself. - Belljar CI/CD tests continuously identify critical startup dependencies before deployment. - A Twine recovery kit, supported by Belljar and Twrko, provides a manual “jumpstart” mechanism for breaking unexpected dependency cycles. - Meta also encountered a “boomerang” problem in which UEs shut down the control-plane services responsible for generating and distributing those signals. - The simpler solution was to let control-plane services ignore power-related shutdown UEs, preventing orphaned services that could not be reaped or recovered. ## Balancing Reliability and Engineering Velocity - Absolute tolerance to instant power loss could require costly or overly complex infrastructure and might create false positives during normal operations. - Meta defined unacceptable impacts as: - Storage or database data loss - Permanent damage to data-center facilities - Sustained disruption beyond one region - The company accepted bounded risks such as transient service errors, limited rack failures, and temporary staleness in routing or region-availability information. - Issues were considered tolerable when they could be remediated after the incident within a reasonable mean time to respond (MTTR). ## Incremental Validation Through PowerLoss Storms - Because testing a full region carried significant risk, Meta validated readiness progressively: - Dependency tests in new and pre-production regions - Exercises in shadow regions that mirror production - Tests in small production regions - Full tests in large regions supporting storage, AI, and data-warehouse workloads - During a Storm, Meta injects a power-supply fault to immediately de-energize an entire region. - After a short, realistic MTTR, remedial drain actions isolate the region from global controllers and schedulers. - The tests avoid preemptive preparation so they accurately represent an unexpected power failure. - Repeated exercises train both systems and engineers to handle regional loss with the resilience normally expected from smaller fault domains. Meta’s approach is to expand disaster readiness gradually: define unacceptable consequences, build layered recovery mechanisms, test at increasing scale, and use each exercise to improve both architecture and operational practice.
