Squeezing every millisecond: How we rebuilt the Datadog Lambda Extension in Rust
Datadog rewrote its AWS Lambda extension from Go into Rust to overcome the performance limits of adapting its large, host-oriented Datadog Agent to Lambda’s constrained environment. The redesign reduced cold-start latency by 82%, memory usage by 40%, and binary size from 55 MB to 7 MB. The project succeeded by narrowing the problem, enforcing performance budgets from the beginning, and designing specifically for Lambda’s execution model. ## Why the Original Extension Needed to Change - The Lambda extension runs as a sidecar process, collecting logs, metrics, traces, profiles, and process data asynchronously. - It was originally based on the Datadog Agent, which is designed for hosts, containers, and clusters. - The Agent’s fairness, buffering, caching, and high-throughput features introduced unnecessary overhead in Lambda. - Optimization attempts included: - Removing dependencies with build tags - Compressing binaries with UPX - Eliminating unnecessary `init` methods - Exploring Go plugins for lazy loading - These changes could not reduce additional cold-start latency below roughly 450–500 milliseconds. ## Why a Rewrite—and Why Rust - Rewrites are risky because they can lose undocumented invariants, reproduce subtle bugs, and create the burden of supporting two systems. - The team concluded that Lambda represented a fundamentally different scale and workload from the general-purpose Datadog Agent. - Rust was well suited because: - Memory safety reduces the risk of crashes and data races. - Extension crashes also terminate the Lambda function and trigger another cold start. - Rust produces small binaries with limited runtime overhead. - Lambda targets a narrow platform set: Amazon Linux on x86 and Arm. - Compile-time concurrency guarantees support reliable multithreaded code. - A hackathon prototype demonstrated enough potential to begin the full rewrite, named Project Bottlecap. ## Project Bottlecap’s Design Constraints - The extension had to minimize interference with the function handler, especially because many Lambda functions serve latency-sensitive APIs. - Telemetry work should occur after the handler returns whenever possible. - The team also minimized post-runtime duration—the CPU time added after normal function execution. - Performance was monitored from the start: - Dashboards and alerts tracked cold-start overhead. - Every pull request was benchmarked. - Regressions were investigated before merging. - The team accepted targeted tradeoffs for speed, including manually implementing AWS API calls and request signing instead of using SDKs that added too much overhead. - The design emphasized optionality because Lambda workloads range from small API functions to large asynchronous batch jobs. - Planned flush strategies included: - Flushing at the end of an invocation for infrequently called or CPU-constrained functions - Periodic or in-invocation flushing for workloads needing different latency and resource tradeoffs The practical lesson is that software optimized for large, long-running systems may be fundamentally unsuitable for serverless runtimes. When optimization reaches a hard performance floor, a focused rewrite—constrained by the target environment and measured continuously—can deliver major gains.
Read original(opens in new tab)