Cpu Optimization | Techlist.io

cloudflareMar 23, 20263 min readCurated summary

Launching Cloudflare’s Gen 13 servers- trading cache for cores for 2x edge compute performance

Cloudflare’s Gen 13 servers use AMD EPYC 5th Gen Turin processors to provide up to twice as many cores as Gen 12. However, Turin’s much smaller per-core cache caused the legacy FL1 request-handling layer to suffer severe latency increases, despite higher throughput. Cloudflare found that tuning alone could not fully solve the problem, reinforcing the need for FL2, a Rust-based rewrite designed to scale with cores rather than depend heavily on cache. ## Turin’s Core-Heavy Architecture - Gen 13 Turin processors offer: - Up to 192 cores and 384 SMT threads, compared with Gen 12’s 96 cores. - Improved instructions per cycle through the Zen 5 architecture. - Up to 32% lower power consumption per core. - DDR5-6400 support for greater memory bandwidth. - The tradeoff is substantially less cache: - Gen 12 Genoa-X provides 12 MB of L3 cache per core through 3D V-Cache. - The 192-core Turin 9965 provides only 2 MB per core. - This architecture favors aggregate throughput but challenges workloads dependent on cache locality. ## FL1’s Cache and Latency Problems - FL1, based on NGINX and LuaJIT, was optimized for Gen 12’s large cache. - AMD uProf measurements showed: - Dramatically higher L3 cache miss rates on Turin. - More requests requiring slow DRAM access. - Increasing latency as CPU utilization and cache contention rose. - An L3 hit takes roughly 50 CPU cycles, while a DRAM fetch can take more than 350 cycles. - As a result, Gen 13’s additional cores delivered throughput gains but introduced unacceptable latency penalties. ## Throughput Gains at an Unacceptable Cost - With FL1, Gen 13 produced: - 10% more throughput on the 128-core Turin 9755. - 31% more on the 160-core Turin 9845. - 62% more on the 192-core Turin 9965. - The Turin 9965 offered the strongest total-cost-of-ownership benefits. - However, latency increased by more than 50% at high CPU utilization, which would negatively affect customer experience and violate performance requirements. ## Hardware and Resource Tuning - Cloudflare tested several mitigations with AMD: - Hardware prefetcher and Data Fabric Probe Filter adjustments produced only marginal improvements. - Adding FL1 workers increased throughput but took resources away from other services. - CPU pinning and isolation provided limited benefits. - AMD’s Platform Quality of Service (PQOS) was used to control cache and memory-bandwidth sharing across Turin’s Core Complex Dies. ## Cache Isolation with PQOS - Reserving part of a single CCD’s cache for FL1 produced less than 5% additional throughput. - Configurations assigning FL1 50–75% of each CCD’s cache also delivered less than 5% improvement and caused minor degradation elsewhere. - A socket-level approach was more successful: - Six of twelve CCDs, aligned with a NUMA domain, were dedicated to FL1. - This provided more than 15% incremental throughput while keeping latency acceptable. - These results showed that workload placement and cache locality could help, but they were not a complete substitute for software designed around Turin’s cache profile. Cloudflare’s broader solution was FL2, a Rust-based rewrite of its core request-handling layer. By reducing dependence on large per-core caches, FL2 enabled Gen 13’s higher core count to translate into scalable edge-compute performance without the latency penalties seen with FL1.

Read original

Cpu Optimization Database Design Rust Nginx+4

datadogSep 18, 20253 min readCurated summary

From hand-tuned Go to self-optimizing code: Building BitsEvolve

Datadog found that small Go-level optimizations can produce substantial infrastructure savings when applied to heavily used, autoscaled services. Manual work—such as removing bounds checks and prioritizing common input paths—delivered improvements ranging from 25% to over 90% in targeted functions. These successes also revealed the need to automate expert optimization techniques through systems like Datadog’s internal BitsEvolve. ## Finding Hotspots That Matter - Micro-optimizations are worthwhile when: - Functions run millions or billions of times. - Services are aggressively autoscaled, allowing CPU savings to reduce machine counts. - Resource usage drops measurably. - Datadog focused on high-throughput services processing timeseries tags and values. - Individual hotspots sometimes represented only 0.5% of compute, but repeated savings could add up to tens of thousands of dollars annually. - The broader goal was a 5–10% reduction in CPU usage across many improvements. ## Removing Bounds Checks from `NormalizeTag` - `NormalizeTag` called `isNormalizedASCIITag`, a frequently executed validator for ASCII tag strings. - AI coding tools suggested changes that were correct but produced no measurable performance gains. - Examining Go assembly with Compiler Explorer revealed two `runtime.panicBounds` calls per loop iteration. - Restructuring the loop eliminated unnecessary bounds checks and enabled further tuning. - The function became 25% faster, reducing service CPU usage by 0.75% and producing projected annual savings of tens of thousands of dollars. ## Using Observability to Optimize for Real Inputs - `NormalizeTagArbTagValue` handled arbitrary input, including invalid UTF-8 and binary data, and consumed 4.5% of CPU in its processing service. - Production data showed: - Nearly all inputs were ASCII. - UTF-8 appeared in fewer than 3% of cases. - Invalid UTF-8 represented less than 0.01% of inputs. - A fast path optimized for common ASCII data made the function more than 90% faster without reducing correctness or safety. - The change generated projected annual savings of hundreds of thousands of dollars. - The result demonstrated that observability is essential: optimization decisions should reflect actual workloads rather than hypothetical edge cases. ## From Manual Optimization to Automation - Deep performance tuning requires specialized knowledge of profiling, compiler behavior, assembly, and workload analysis. - Although the results can be valuable, the process is time-consuming and difficult to scale across a large organization. - Datadog wanted to move beyond isolated “heroic” optimizations toward a repeatable and automated process. - The manual techniques used by performance engineers became the foundation for heuristics in BitsEvolve, an internal agentic system intended to optimize code systematically. Datadog’s experience suggests that organizations should combine production observability with compiler-level analysis, prioritize high-impact hot paths, and automate proven optimization patterns so performance gains do not depend solely on a small group of experts.

Read original

Cpu Optimization Database Design Observability Go+4

datadogSep 30, 20212 min readCurated summary

How we optimized our Akka application using Datadog’s Continuous Profiler

Datadog discovered that an unexpected 20–30% CPU overhead came from Akka’s use of `ForkJoinPool`, not from the log-processing code they initially suspected. Profiling showed that an actor handling intermittent latency metrics repeatedly caused worker threads to park and unpark. Moving that actor to a busier, more stable dispatcher reduced CPU usage by about 30%. ## How Profiling Revealed the Problem - Datadog used Akka to parallelize log-event processing through actors and dispatchers. - An optimization to log parsing produced little improvement, despite reducing parsing CPU time. - Continuous Profiler flame graphs showed increased CPU time in: - `ForkJoinPool.scan()` - `Unsafe.park()` - Thread-level analysis revealed that the default Akka dispatcher—not the expected dedicated work pool—was responsible. - Many of the affected threads were executing a latency-reporting actor. ## Why `ForkJoinPool` Was Consuming CPU - `ForkJoinPool` dynamically manages worker threads: - It creates threads when work increases. - It suspends idle threads with `Unsafe.park()`. - It resumes them with `Unsafe.unpark()`. - It terminates idle workers after a default period. - The latency actor received a few hundred events per second, processed them within milliseconds, and then remained idle until the next batch. - Because the pool allowed up to 32 threads—matching the number of processor cores—it repeatedly activated and suspended many workers. - These frequent parking and unparking operations created short CPU spikes and excessive time in `ForkJoinPool.scan()`. ## The Dispatcher Change - The team moved the latency actor from Akka’s default dispatcher to the main `work-dispatcher`. - The work dispatcher already handled a steadier stream of log-processing tasks, keeping its worker threads active. - This required only a configuration change assigning the actor to `work-dispatcher`. - CPU usage fell by roughly 30% across services. - The default dispatcher also shrank from 32 threads to 2, confirming that unnecessary thread activation was the cause. ## Recommendations - Monitor CPU time spent in `ForkJoinPool.scan()`, especially when it exceeds roughly 10–15%. - Limit the number of Akka actor instances. - Set a suitable maximum thread count for each pool. - Reduce the number of separate thread pools where practical. - Use task queues to absorb frequent, short-lived workload spikes. - Aim to keep the number of active `ForkJoinPool` workers relatively stable and avoid repeated parking and unparking.

Read original

Cpu Optimization Database Design Java Profiling+4