Launching Cloudflare’s Gen 13 servers- trading cache for cores for 2x edge compute performance

Cloudflare’s Gen 13 servers use AMD EPYC 5th Gen Turin processors to provide up to twice as many cores as Gen 12. However, Turin’s much smaller per-core cache caused the legacy FL1 request-handling layer to suffer severe latency increases, despite higher throughput. Cloudflare found that tuning alone could not fully solve the problem, reinforcing the need for FL2, a Rust-based rewrite designed to scale with cores rather than depend heavily on cache.

Turin’s Core-Heavy Architecture

• Gen 13 Turin processors offer:
  • Up to 192 cores and 384 SMT threads, compared with Gen 12’s 96 cores.
  • Improved instructions per cycle through the Zen 5 architecture.
  • Up to 32% lower power consumption per core.
  • DDR5-6400 support for greater memory bandwidth.
• The tradeoff is substantially less cache:
  • Gen 12 Genoa-X provides 12 MB of L3 cache per core through 3D V-Cache.
  • The 192-core Turin 9965 provides only 2 MB per core.
• This architecture favors aggregate throughput but challenges workloads dependent on cache locality.

FL1’s Cache and Latency Problems

• FL1, based on NGINX and LuaJIT, was optimized for Gen 12’s large cache.
• AMD uProf measurements showed:
  • Dramatically higher L3 cache miss rates on Turin.
  • More requests requiring slow DRAM access.
  • Increasing latency as CPU utilization and cache contention rose.
• An L3 hit takes roughly 50 CPU cycles, while a DRAM fetch can take more than 350 cycles.
• As a result, Gen 13’s additional cores delivered throughput gains but introduced unacceptable latency penalties.

Throughput Gains at an Unacceptable Cost

• With FL1, Gen 13 produced:
  • 10% more throughput on the 128-core Turin 9755.
  • 31% more on the 160-core Turin 9845.
  • 62% more on the 192-core Turin 9965.
• The Turin 9965 offered the strongest total-cost-of-ownership benefits.
• However, latency increased by more than 50% at high CPU utilization, which would negatively affect customer experience and violate performance requirements.

Hardware and Resource Tuning

• Cloudflare tested several mitigations with AMD:
  • Hardware prefetcher and Data Fabric Probe Filter adjustments produced only marginal improvements.
  • Adding FL1 workers increased throughput but took resources away from other services.
  • CPU pinning and isolation provided limited benefits.
• AMD’s Platform Quality of Service (PQOS) was used to control cache and memory-bandwidth sharing across Turin’s Core Complex Dies.

Cache Isolation with PQOS

• Reserving part of a single CCD’s cache for FL1 produced less than 5% additional throughput.
• Configurations assigning FL1 50–75% of each CCD’s cache also delivered less than 5% improvement and caused minor degradation elsewhere.
• A socket-level approach was more successful:
  • Six of twelve CCDs, aligned with a NUMA domain, were dedicated to FL1.
  • This provided more than 15% incremental throughput while keeping latency acceptable.
• These results showed that workload placement and cache locality could help, but they were not a complete substitute for software designed around Turin’s cache profile.

Cloudflare’s broader solution was FL2, a Rust-based rewrite of its core request-handling layer. By reducing dependence on large per-core caches, FL2 enabled Gen 13’s higher core count to translate into scalable edge-compute performance without the latency penalties seen with FL1.