Performance Tuning | Techlist.io

cloudflareMar 23, 20263 min readCurated summary

Launching Cloudflare’s Gen 13 servers- trading cache for cores for 2x edge compute performance

Cloudflare’s Gen 13 servers use AMD EPYC 5th Gen Turin processors to provide up to twice as many cores as Gen 12. However, Turin’s much smaller per-core cache caused the legacy FL1 request-handling layer to suffer severe latency increases, despite higher throughput. Cloudflare found that tuning alone could not fully solve the problem, reinforcing the need for FL2, a Rust-based rewrite designed to scale with cores rather than depend heavily on cache. ## Turin’s Core-Heavy Architecture - Gen 13 Turin processors offer: - Up to 192 cores and 384 SMT threads, compared with Gen 12’s 96 cores. - Improved instructions per cycle through the Zen 5 architecture. - Up to 32% lower power consumption per core. - DDR5-6400 support for greater memory bandwidth. - The tradeoff is substantially less cache: - Gen 12 Genoa-X provides 12 MB of L3 cache per core through 3D V-Cache. - The 192-core Turin 9965 provides only 2 MB per core. - This architecture favors aggregate throughput but challenges workloads dependent on cache locality. ## FL1’s Cache and Latency Problems - FL1, based on NGINX and LuaJIT, was optimized for Gen 12’s large cache. - AMD uProf measurements showed: - Dramatically higher L3 cache miss rates on Turin. - More requests requiring slow DRAM access. - Increasing latency as CPU utilization and cache contention rose. - An L3 hit takes roughly 50 CPU cycles, while a DRAM fetch can take more than 350 cycles. - As a result, Gen 13’s additional cores delivered throughput gains but introduced unacceptable latency penalties. ## Throughput Gains at an Unacceptable Cost - With FL1, Gen 13 produced: - 10% more throughput on the 128-core Turin 9755. - 31% more on the 160-core Turin 9845. - 62% more on the 192-core Turin 9965. - The Turin 9965 offered the strongest total-cost-of-ownership benefits. - However, latency increased by more than 50% at high CPU utilization, which would negatively affect customer experience and violate performance requirements. ## Hardware and Resource Tuning - Cloudflare tested several mitigations with AMD: - Hardware prefetcher and Data Fabric Probe Filter adjustments produced only marginal improvements. - Adding FL1 workers increased throughput but took resources away from other services. - CPU pinning and isolation provided limited benefits. - AMD’s Platform Quality of Service (PQOS) was used to control cache and memory-bandwidth sharing across Turin’s Core Complex Dies. ## Cache Isolation with PQOS - Reserving part of a single CCD’s cache for FL1 produced less than 5% additional throughput. - Configurations assigning FL1 50–75% of each CCD’s cache also delivered less than 5% improvement and caused minor degradation elsewhere. - A socket-level approach was more successful: - Six of twelve CCDs, aligned with a NUMA domain, were dedicated to FL1. - This provided more than 15% incremental throughput while keeping latency acceptable. - These results showed that workload placement and cache locality could help, but they were not a complete substitute for software designed around Turin’s cache profile. Cloudflare’s broader solution was FL2, a Rust-based rewrite of its core request-handling layer. By reducing dependence on large per-core caches, FL2 enabled Gen 13’s higher core count to translate into scalable edge-compute performance without the latency penalties seen with FL1.

Read original

Performance Tuning Database Design Rust Nginx+4

datadogFeb 22, 20213 min readCurated summary

How we minimized the overhead of Kubernetes in our job system

Kubernetes can improve machine management and scalability, but its scheduling and runtime overhead can significantly reduce job throughput if configured poorly. Datadog found its Kubernetes-based job system used more CPU and completed jobs 40–50% more slowly than the previous VM-based system. By designing a controlled experiment, choosing better metrics, and tuning pod resource requests, the team recovered performance to roughly VM parity while investigating the overhead of running one parent process per pod. ## Designing a Comparable Experiment - The initial comparison was difficult because the Kubernetes and VM deployments differed in: - Number of nodes - Number of worker-parent clusters - Workload and enqueue rate - A controlled experiment was created using: - Identical `c5.2xlarge` machines - The same kernel version, `3.13.0-141` - Both systems repeatedly running a simple Python job - Each Kubernetes pod contained one parent process and its worker processes, making pod count equivalent to parent-process count per node. - The older kernel did not include CPU mitigations, avoiding that variable in the comparison. ## Choosing Useful Performance Metrics ### Measuring node effort - Load average initially appeared useful for measuring machine utilization. - Kubernetes background processes—such as cluster polling and pod-state checks—artificially increased load average. - Load average counts runnable processes rather than the amount of CPU time they actually consume. - The team therefore used CPU idle time instead: - It measures unused CPU capacity. - It reflects actual CPU work rather than the number of active processes. ### Measuring system performance - The job system optimized for throughput rather than latency. - Throughput was measured by the number of jobs completed within 30 seconds. - Latency remained useful for detecting queueing problems, but throughput was the primary success metric. ## Tuning Kubernetes Resource Requests - The main performance gains came from improving pod scheduling. - The target was six pods per `c5.2xlarge` node. - Initially, each pod requested: - One full CPU core - More memory than necessary - Since the node had eight cores and approximately 1.5 GiB of memory consumed by Kubernetes and system services, only four pods could be scheduled. - Requests were reduced to: - `100m` CPU, or 100 millicores - `500 MB` memory - CPU tuning generally enabled six pods per node, although some nodes still scheduled only five. - Further memory reduction was needed because system daemons consumed enough memory to prevent six pods from fitting on some nodes. - Resource requests affect scheduling minimums, while limits constrain containers after they start. - These request changes did not slow jobs because the pods still received sufficient resources to operate. ## One Parent Process per Pod - The team considered placing multiple parent processes in each pod to reduce potential pod overhead. - One parent plus its workers was a natural application unit and simplified orchestration. - The decision depended on how much overhead each pod introduced: - High overhead would favor fewer, larger pods. - Low overhead would favor one parent per pod for simpler management. - Using `pstree`, the team identified six job-system instances per node and traced their process trees through components such as: - `containerd-shim` - `tini` - The application process - They estimated that each pod included overhead associated with three containers, particularly `containerd-shim`. - CPU overhead was then investigated using `perf sched`. The practical lesson is to compare equivalent workloads, measure actual CPU consumption rather than relying blindly on load average, and tune Kubernetes requests for the desired packing density. Resource requests should be large enough for reliable operation but not so large that they unnecessarily prevent pods from being scheduled together.

Read original

Performance Tuning Database Design Kubernetes Python+4