How Workers powers our internal maintenance scheduling pipeline
Cloudflare built an automated maintenance scheduler on Cloudflare Workers to prevent overlapping infrastructure changes from disrupting connectivity or customer-specific routing. The system evaluates the full network state, identifies conflicts across maintenance events, and alerts operators before unsafe schedules are approved. Its key design shift was from loading all operational data into one Worker to using graph-based, on-demand data retrieval that respects Workers’ memory limits. ## Why Manual Maintenance Planning Was No Longer Enough - Cloudflare operates data centers in more than 330 cities, making manual coordination increasingly unreliable. - Maintenance can create conflicts when: - Redundant edge routers in the same metro area are taken offline simultaneously. - All data centers selected by a customer’s Dedicated CDN Egress IPs (“Aegis”) pool become unavailable. - These failures could cause higher latency, connectivity loss, or 5xx errors. - The scheduler centralizes network state and warns operators when maintenance windows overlap in unsafe ways. ## Modeling Operational Safety as Constraints - Each safety rule begins with proposed maintenance items, such as routers or server groups. - The system finds calendar events whose time windows overlap with the proposed change. - It then combines those events with product data, including Aegis pools and their associated data center IDs. - For example, if an Aegis customer’s pool uses data centers 21 and 45, scheduling both for simultaneous downtime violates the constraint that at least one must remain online. - Operators receive conflict notifications and can reschedule maintenance before it affects customers. ## Reducing Data Usage on Workers - The initial design loaded server relationships, product configurations, and health metrics into a single Worker. - This quickly caused out-of-memory errors. - The scheduler instead loads only data relevant to the maintenance location and affected relationships. - A router maintenance request in Frankfurt, for example, does not need unrelated infrastructure data from Australia. ## Graph Processing with Typed Associations - Cloudflare modeled infrastructure and product relationships as a graph: - **Objects** represent entities such as routers, data centers, and Aegis pools. - **Associations** represent relationships between those entities. - Inspired by Facebook’s TAO system, the team created an interface supporting operations such as: - `object_get()` to retrieve an object. - `assoc_get()` to stream typed relationships. - `assoc_count()` to count related objects. - Constraints can retrieve only the associations they need, such as which Aegis pools include a particular data center and how many data centers each pool contains. - Parallel lookups and deduplication reduce both execution time and memory consumption. Cloudflare’s scheduler demonstrates how Workers can serve as a centralized safety layer for complex infrastructure operations. The practical recommendation is to represent operational dependencies as typed graphs and fetch relationship data incrementally, rather than loading the entire network and product state into each execution.
Read original(opens in new tab)