Modernizing the Meta Ads Service With an Open-Source Kernel Scheduler (opens in new tab)

At Meta’s scale, small latency regressions can materially affect ad relevance, ranking, and revenue. When Linux kernel 6.9’s EEVDF scheduler reduced ad-serving performance, Meta used sched_ext to deploy a workload-specific BPF scheduler without modifying the kernel. The solution reduced p99 ads-retrieval latency by 28%, saved 3.28 MW of power, increased ads ranked by 1.1%, and enabled further improvements through rapid user-space updates.

Why Ads Latency Matters

• Meta’s ads platform processes more than 5 million requests per second, or over 400 billion requests daily.
• Lower p99 latency allows more relevant ads to be retrieved and ranked within each request.
• General-purpose schedulers such as CFS and EEVDF balance CPU usage without understanding which threads are most important to ad delivery.
• Ads-specific scheduling can prioritize work on the critical request path while deferring less-sensitive tasks.

The Kernel Upgrade Problem

• During a move from Linux 6.4 to 6.9, Meta found that EEVDF introduced a latency regression.
• The regression reduced the number of ads ranked in responses.
• Some servers had to remain on Linux 6.4, creating operational fragmentation and technical debt.
• sched_ext provided a way to address the regression without waiting for another kernel release.

How sched_ext Customizes Scheduling

• sched_ext is an upstream, BPF-based framework that entered Linux 6.12.
• It lets developers implement scheduling policies in BPF programs responding to events such as:
  • Thread wake-ups
  • Run-queue insertion
  • Dispatching the next thread
  • CPU idle-state transitions
• Meta’s policy divides CPUs into two dynamically sized pools:
  • Latency-critical request-path threads
  • Less latency-sensitive background work
• Keeping related work on the same CPUs improves L3 cache locality and reduces DRAM access.
• The scheduler is loaded by a user-space binary, so new policies can be deployed by restarting the scheduler process rather than rebuilding the kernel.

Performance and Operational Results

The initial deployment on the largest ads-serving server type achieved:

• 28% lower p99 latency on the ads retrieval path
• 1.1% more weighted ads ranked
• 3.28 MW of fleet-wide power savings

Two subsequent policy updates produced additional gains:

• A further 60% reduction in service p99 latency
• 18% fewer timeout errors on the critical path
• Delivery in days instead of the months typically required for kernel changes

From Fix to Optimization Platform

• sched_ext gives Meta an independent scheduling-development path alongside upstream Linux evolution.
• BPF updates support rapid experimentation with:
  • Cache-aware thread placement
  • ROI-based executor routing
  • NUMA-aware scheduling
• Because sched_ext is upstream, other organizations can implement workload-specific policies without maintaining a Linux kernel fork.
• Meta plans to use application-level hints, such as request importance, to adjust scheduling slices and queue priority dynamically.

sched_ext demonstrates that application-aware scheduling can produce measurable business, latency, and energy benefits. For workloads with priorities that general-purpose schedulers cannot see, an extensible BPF-based scheduler offers a practical way to optimize continuously without coupling improvements to kernel release cycles.