From hand-tuned Go to self-optimizing code: Building BitsEvolve
Datadog found that small Go-level optimizations can produce substantial infrastructure savings when applied to heavily used, autoscaled services. Manual work—such as removing bounds checks and prioritizing common input paths—delivered improvements ranging from 25% to over 90% in targeted functions. These successes also revealed the need to automate expert optimization techniques through systems like Datadog’s internal BitsEvolve. ## Finding Hotspots That Matter - Micro-optimizations are worthwhile when: - Functions run millions or billions of times. - Services are aggressively autoscaled, allowing CPU savings to reduce machine counts. - Resource usage drops measurably. - Datadog focused on high-throughput services processing timeseries tags and values. - Individual hotspots sometimes represented only 0.5% of compute, but repeated savings could add up to tens of thousands of dollars annually. - The broader goal was a 5–10% reduction in CPU usage across many improvements. ## Removing Bounds Checks from `NormalizeTag` - `NormalizeTag` called `isNormalizedASCIITag`, a frequently executed validator for ASCII tag strings. - AI coding tools suggested changes that were correct but produced no measurable performance gains. - Examining Go assembly with Compiler Explorer revealed two `runtime.panicBounds` calls per loop iteration. - Restructuring the loop eliminated unnecessary bounds checks and enabled further tuning. - The function became 25% faster, reducing service CPU usage by 0.75% and producing projected annual savings of tens of thousands of dollars. ## Using Observability to Optimize for Real Inputs - `NormalizeTagArbTagValue` handled arbitrary input, including invalid UTF-8 and binary data, and consumed 4.5% of CPU in its processing service. - Production data showed: - Nearly all inputs were ASCII. - UTF-8 appeared in fewer than 3% of cases. - Invalid UTF-8 represented less than 0.01% of inputs. - A fast path optimized for common ASCII data made the function more than 90% faster without reducing correctness or safety. - The change generated projected annual savings of hundreds of thousands of dollars. - The result demonstrated that observability is essential: optimization decisions should reflect actual workloads rather than hypothetical edge cases. ## From Manual Optimization to Automation - Deep performance tuning requires specialized knowledge of profiling, compiler behavior, assembly, and workload analysis. - Although the results can be valuable, the process is time-consuming and difficult to scale across a large organization. - Datadog wanted to move beyond isolated “heroic” optimizations toward a repeatable and automated process. - The manual techniques used by performance engineers became the foundation for heuristics in BitsEvolve, an internal agentic system intended to optimize code systematically. Datadog’s experience suggests that organizations should combine production observability with compiler-level analysis, prioritize high-impact hot paths, and automate proven optimization patterns so performance gains do not depend solely on a small group of experts.
Read original(opens in new tab)