Shedding old code with ecdysis: graceful restarts for Rust services at Cloudflare (opens in new tab)
Cloudflare’s open-source Rust library **ecdysis** enables zero-downtime restarts for high-volume network services. It preserves listening sockets and existing connections while a new process initializes, avoiding refused connections and dropped requests. After five years of production use, Cloudflare uses it to safely deploy fixes, security patches, and new features across its global infrastructure. ## Why Conventional Restarts Fail - Stopping the old process before starting the new one creates a period when no process is listening. - New clients receive `ECONNREFUSED`; even a 100 ms gap can drop hundreds of connections at a busy location. - Existing connections—including file uploads, video streams, WebSockets, and gRPC streams—are terminated when the old process exits. - `SO_REUSEPORT` allows multiple processes to bind the same port, but can orphan connections: - The kernel assigns an incoming `SYN` to one listening socket. - If that process exits before calling `accept()`, the queued connection is terminated. - This makes simply overlapping two independently bound processes unsafe for graceful upgrades. ## The ecdysis Restart Model ecdysis uses a process-forking approach pioneered by NGINX: - The parent calls `fork()` to create a child. - The child replaces itself with the new executable using `execve()`. - The child inherits the listening socket file descriptors through a named pipe shared with the parent. - The parent continues serving traffic while the child initializes. - Once the child signals readiness, the parent closes its copy of the listening socket and drains existing connections. - Both processes may briefly accept connections during the transition, but this is intentional and avoids coverage gaps. ## Crash Safety and Upgrade Requirements - The old process can fully shut down after the replacement is ready. - The new process receives time to initialize before taking over. - If initialization fails—for example, because of invalid configuration—the child exits while the parent continues serving normally. - Upgrades are serialized so that only one runs at a time, preventing cascading failures. - The unchanged listening socket ensures that new connections are not refused during the handoff. ## Rust and System Integration - ecdysis provides native Tokio stream wrappers for asynchronous Rust services. - Synchronous services can use it without an async runtime. - With the `systemd_notify` feature enabled, it integrates with systemd lifecycle notifications. - Configuring a service with `Type=notify-reload` allows systemd to track graceful upgrades correctly. Cloudflare’s approach demonstrates that graceful restarts require coordination between processes rather than simply starting a second server. Services needing reliable zero-downtime upgrades can use ecdysis to preserve connections, tolerate failed deployments, and safely roll out new Rust binaries.
