How GitHub uses eBPF to improve deployment safety (opens in new tab)
GitHub uses eBPF to prevent deployment scripts from depending on GitHub or other services that may be unavailable during an outage. The approach applies network restrictions only to deployment processes, preserving normal traffic for stateful production hosts. By combining Linux cgroups with eBPF’s `BPF_PROG_TYPE_CGROUP_SKB` hooks, GitHub can detect or block unsafe outbound calls before they create circular deployment dependencies. ## The Deployment Circular Dependency - GitHub hosts its own source code on `github.com`, creating a basic dependency: GitHub may need GitHub to deploy a fix. - GitHub mitigates this with: - A code mirror used for “fix-forward” deployments. - Prebuilt assets used to roll back changes. - Additional circular dependencies can still be introduced by deployment scripts, internal services, or tools that download binaries dynamically. ## Three Types of Circular Dependencies The post uses a hypothetical MySQL outage to illustrate how deployment recovery can fail: - **Direct dependencies** - A deployment script downloads the latest release of an open-source tool from GitHub. - If GitHub cannot serve release data, the deployment cannot complete. - **Hidden dependencies** - A required tool is already installed locally but checks GitHub for updates when it runs. - The tool may fail or hang if that update check cannot connect. - **Transient dependencies** - The deployment calls another internal service, such as a migrations service. - That service then attempts to download a binary from GitHub, causing the failure to propagate back to the deployment. ## Why Manual Review Is Insufficient - Teams responsible for stateful hosts traditionally review deployment scripts for circular dependencies. - Many indirect or unexpected dependencies are discovered only during incidents, when they can delay recovery. - Blocking `github.com` at the host level would be too disruptive because stateful hosts continue serving customer traffic during deploys, drains, and restarts. ## Per-Process Network Filtering with eBPF - eBPF allows custom programs to run inside the Linux kernel and attach to low-level operations such as networking. - GitHub focused on `BPF_PROG_TYPE_CGROUP_SKB`, which can inspect network egress for a specific cgroup. - Linux cgroups provide process grouping, isolation, and resource controls without requiring Docker. - The proposed design: - Create a dedicated cgroup. - Place only the deployment script and its processes inside it. - Restrict or monitor outbound network access for that group. - Leave the host’s ordinary production traffic unaffected. ## Proof of Concept with Go and eBPF - The proof of concept uses Go and the `cilium/ebpf` library. - The library simplifies: - Compiling and loading eBPF programs. - Attaching programs to kernel hooks. - Reading and updating eBPF maps. - The example attaches an egress program to `/sys/fs/cgroup/system.slice`. - An eBPF array map tracks the number of egress packets, while the Go program periodically reads and reports the counter. - The same mechanism can be extended from packet counting to selectively allowing or blocking network traffic. GitHub’s approach moves dependency validation from manual inspection into enforcement at runtime. Restricting only deployment processes with cgroups and eBPF provides a practical way to make emergency deployments independent of services that may be down, without blocking the production workloads sharing the same host.