techlist.io
KREN

Rust

37 posts

cloudflare3 min readCurated summary

Launching Cloudflare’s Gen 13 servers- trading cache for cores for 2x edge compute performance

Cloudflare’s Gen 13 servers use AMD EPYC 5th Gen Turin processors to provide up to twice as many cores as Gen 12. However, Turin’s much smaller per-core cache caused the legacy FL1 request-handling layer to suffer severe latency increases, despite higher throughput. Cloudflare found that tuning alone could not fully solve the problem, reinforcing the need for FL2, a Rust-based rewrite designed to scale with cores rather than depend heavily on cache. ## Turin’s Core-Heavy Architecture - Gen 13 Turin processors offer: - Up to 192 cores and 384 SMT threads, compared with Gen 12’s 96 cores. - Improved instructions per cycle through the Zen 5 architecture. - Up to 32% lower power consumption per core. - DDR5-6400 support for greater memory bandwidth. - The tradeoff is substantially less cache: - Gen 12 Genoa-X provides 12 MB of L3 cache per core through 3D V-Cache. - The 192-core Turin 9965 provides only 2 MB per core. - This architecture favors aggregate throughput but challenges workloads dependent on cache locality. ## FL1’s Cache and Latency Problems - FL1, based on NGINX and LuaJIT, was optimized for Gen 12’s large cache. - AMD uProf measurements showed: - Dramatically higher L3 cache miss rates on Turin. - More requests requiring slow DRAM access. - Increasing latency as CPU utilization and cache contention rose. - An L3 hit takes roughly 50 CPU cycles, while a DRAM fetch can take more than 350 cycles. - As a result, Gen 13’s additional cores delivered throughput gains but introduced unacceptable latency penalties. ## Throughput Gains at an Unacceptable Cost - With FL1, Gen 13 produced: - 10% more throughput on the 128-core Turin 9755. - 31% more on the 160-core Turin 9845. - 62% more on the 192-core Turin 9965. - The Turin 9965 offered the strongest total-cost-of-ownership benefits. - However, latency increased by more than 50% at high CPU utilization, which would negatively affect customer experience and violate performance requirements. ## Hardware and Resource Tuning - Cloudflare tested several mitigations with AMD: - Hardware prefetcher and Data Fabric Probe Filter adjustments produced only marginal improvements. - Adding FL1 workers increased throughput but took resources away from other services. - CPU pinning and isolation provided limited benefits. - AMD’s Platform Quality of Service (PQOS) was used to control cache and memory-bandwidth sharing across Turin’s Core Complex Dies. ## Cache Isolation with PQOS - Reserving part of a single CCD’s cache for FL1 produced less than 5% additional throughput. - Configurations assigning FL1 50–75% of each CCD’s cache also delivered less than 5% improvement and caused minor degradation elsewhere. - A socket-level approach was more successful: - Six of twelve CCDs, aligned with a NUMA domain, were dedicated to FL1. - This provided more than 15% incremental throughput while keeping latency acceptable. - These results showed that workload placement and cache locality could help, but they were not a complete substitute for software designed around Turin’s cache profile. Cloudflare’s broader solution was FL2, a Rust-based rewrite of its core request-handling layer. By reducing dependence on large per-core caches, FL2 enabled Gen 13’s higher core count to translate into scalable edge-compute performance without the latency penalties seen with FL1.

Read original(opens in new tab)
RustDatabase DesignNginxParallel Processing+4
cloudflare3 min readCurated summary

Inside Gen 13- how we built our most powerful server yet

Cloudflare’s Gen 13 server is a hardware redesign aligned with its Rust-based FL2 request-processing stack. By choosing a 192-core AMD EPYC Turin 9965, doubling memory, expanding storage and networking, and adding stronger security and accelerator support, Cloudflare targets up to twice the throughput of Gen 12 while remaining within latency limits. The design also improves efficiency, rack density, and operational simplicity. ## Gen 13 at a Glance - Uses a 2U, single-socket design. - Key specifications: - 192-core AMD EPYC 9965 processor - 768 GB of DDR5-6400 memory - Three 7.68 TB E1.S PCIe 5.0 NVMe drives - Dual 100 GbE OCP 3.0 networking - 1,300W Titanium-grade power supply - ASPEED AST2600 BMC and AST1060 hardware root of trust - Compared with Gen 12, Gen 13 provides: - Up to 2× throughput - Up to 50% better performance per watt - Up to 60% more throughput per rack at the same power budget - Twice the memory capacity, 1.5× the storage, and 4× the network bandwidth - PCIe encryption in addition to memory encryption - Better support for high-heat PCIe accelerators ## Choosing the CPU - Gen 12 used a 96-core AMD EPYC Genoa-X 9684X with: - 400W TDP - 1,152 MB of L3 cache - Cloudflare evaluated three Turin processors: - 9755: strongest per-core performance - 9845: lower socket power and fewer cores - 9965: highest core count and better efficiency - The Turin 9965 was selected with 192 cores and 384 threads, doubling Gen 12’s hardware threads. - Its L3 cache is much smaller—384 MB total, or 2 MB per core versus Gen 12’s 12 MB per core—but FL2 workloads depend less on large L3 caches than the previous FL1 stack. - FL2 scales nearly linearly with additional cores, allowing the 9965 to deliver up to 100% higher throughput. - Production testing showed the 9965 achieved the best aggregate requests per second and favorable performance per watt at its 500W TDP. - Higher compute density also means fewer servers to provision, patch, monitor, and operate. - Turin’s support for DDR5-6400, PCIe 5.0, and CXL 2.0 provides a longer upgrade and security-support runway. ## Memory Bandwidth and Capacity - Gen 13 doubles memory from 384 GB to 768 GB while retaining 4 GB per core. - All twelve memory channels are populated using one 64 GB DDR5-6400 ECC RDIMM per channel. - This configuration delivers approximately 614 GB/s of peak memory bandwidth per socket, a 33.3% increase over Gen 12. - Using identical DIMMs across all channels enables balanced interleaving, distributing memory accesses across the full memory subsystem. - The design is intended to prevent the 192-core processor from being starved of data during highly parallel workloads. Cloudflare’s central design choice was to match hardware to the characteristics of FL2 rather than preserve Gen 12’s cache-heavy strategy. For workloads that scale well across cores, the Turin 9965 and fully populated memory system offer higher throughput, better rack economics, and simpler fleet operations.

Read original(opens in new tab)
RustAmd EpycDdr5Nvme+3
aws3 min readCurated summary

Twenty years of Amazon S3 and building what’s next | Amazon Web Services

Amazon S3 began in 2006 as a simple web service for storing and retrieving objects, but its emphasis on security, durability, availability, performance, and elasticity enabled it to become foundational infrastructure. Over two decades, it scaled from roughly one petabyte to hundreds of exabytes while preserving API compatibility, reducing prices, and expanding beyond object storage. Amazon’s long-term vision is for S3 to serve as a universal foundation for data, analytics, and AI workloads. ## The Original S3 Philosophy - S3 introduced two basic operations: - `PUT` to store an object - `GET` to retrieve it - The service abstracted away complex infrastructure so developers could focus on applications. - Its five enduring design principles are: - **Security:** Data is protected by default. - **Durability:** Designed for 11 nines of durability, with a lossless operating model. - **Availability:** Failure is assumed and handled throughout the system. - **Performance:** Storage capacity can grow without degrading performance. - **Elasticity:** Capacity expands and contracts automatically. ## From One Petabyte to Hundreds of Exabytes - At launch, S3 had approximately: - One petabyte of capacity - 400 storage nodes across 15 racks and three data centers - 15 Gbps of bandwidth - A maximum object size of 5 GB - A price of $0.15 per GB - Today, S3: - Stores more than 500 trillion objects. - Serves over 200 million requests per second. - Operates across 123 Availability Zones in 39 AWS Regions. - Supports objects up to 50 TB—10,000 times larger than the original limit. - Storage prices have fallen by roughly 85%, to slightly above 2 cents per GB. - S3 Intelligent-Tiering has saved customers more than $6 billion in storage costs. - The S3 API has become an industry standard, with many other storage systems offering compatible interfaces. ## Backward Compatibility and Long-Term Reliability - Code written against S3 in 2006 still works without modification. - AWS has repeatedly replaced disks, storage systems, and request-processing code while preserving access to older data. - This compatibility reflects S3’s goal of remaining infrastructure that “just works” despite continuous internal change. ## Engineering for Durability and Scale - Microservices continuously inspect every byte across the fleet. - Auditor services detect degradation and automatically trigger repair and re-replication. - Automated formal methods mathematically verify correctness in areas such as: - The index subsystem - Cross-Region replication - Access policies - AWS has progressively rewritten performance-critical components in Rust over the past eight years. - Rust improves performance while preventing memory-safety bugs and other classes of errors at compile time. - S3 follows the principle that scale should improve the service: larger, more distributed workloads become increasingly decorrelated, improving reliability for all customers. ## S3 as a Foundation for Data and AI Amazon’s future vision is for customers to store data once in S3 and work with it directly, avoiding costly copies and specialized systems. - **S3 Tables** provides managed Apache Iceberg tables with automated maintenance to improve query performance and reduce storage costs. - **S3 Vectors** supports semantic search and retrieval-augmented generation, with up to 2 billion vectors per index and sub-100 ms query latency. - Within five months of launch, customers created over 250,000 indexes, ingested more than 40 billion vectors, and executed over 1 billion queries. - **S3 Metadata** enables centralized, faster data discovery without recursively listing large buckets. These additions extend S3 from inexpensive object storage into a broader platform for analytics, search, and AI while retaining its scale and cost advantages.

Read original(opens in new tab)
RustDatabase DesignAPIMicroservices+4
discord3 min readCurated summary

Discord Patch Notes: February 4, 2026

Discord’s February 4, 2026 patch focuses on performance, streaming, permissions, mobile consistency, and bug fixes across desktop, iOS, and Android. The largest improvements include faster desktop rendering, zoomable screenshares, a new Slowmode bypass permission, and continued migration of Discord’s audio/video backend to Rust. The update also adds usability refinements and resolves numerous interface, profile, notification, and stability issues. ## Performance and Infrastructure - Desktop navigation and interaction are significantly faster, especially on systems with limited processing capacity. - The improvements were primarily achieved by replacing slow CSS selectors rather than optimizing APIs or components. - More than 80% of Discord’s audio/video traffic now runs through the Rust-based backend. - Stream previews load substantially faster during the start-stream process. ## Screensharing and Media - Users can zoom and pan screenshares and game streams with a mouse wheel or laptop trackpad. - Android now correctly plays videos containing multiple audio tracks, such as microphone and game audio. - Currently playing voice messages stop correctly when users log out. - Android Group DM notifications now show the group name instead of the sender’s name, matching iOS behavior. ## Server Permissions and Time Formatting - A new **Bypass Slowmode** permission lets server administrators trust selected members to bypass slowmode restrictions. - The permission can be integrated into existing roles through Role Settings until February 23. - Desktop now includes an `@time` command for generating localized Linux timestamps that automatically adapt to each viewer’s time zone. ## General Usability Improvements - Vibing Wumpus has been updated to version 2.0 and can be summoned with: - `Ctrl+Alt+Shift+W` on Windows/Linux - `Cmd+Option+Shift+W` on macOS - Pressing Escape while editing a desktop profile no longer closes the entire profile modal. - Copying with nothing selected no longer clears the clipboard. - Browse Channels on mobile has officially left beta. - Clicking role mentions now shows users assigned to that role for all stable users. - Invite Friends to Server supports both usernames and display names. - Ignoring a friend request correctly changes the action from “Accept Friend Request” to “Add Friend.” ## Profile, Themes, and Server Interface Fixes - Fixed cursor-jumping while editing the About Me section. - Avatar Decorations now render correctly in per-server profile editing. - Profile Theme gradients fill their modal correctly. - Server tags now appear accurately in desktop profile previews. - Fixed alignment and styling issues in Members, Emoji, Connections, and Display Name Styles interfaces. - iOS Client Themes no longer scroll at extremely high speed. - Channel categories on iOS no longer resemble unread channels. - Android no longer displays an extra bottom bar that blocks interaction. ## Stability and Bug Fixes - Fixed crashes involving related stickers in servers without text channels. - Prevented duplicate Checkpoint and gift-reception modals. - Emoji editing no longer creates duplicates when Finish is clicked twice quickly. - Desktop Inbox filtering works correctly again. - Fixed a desktop lockup that could occur after adding a game to a profile without saving. - Mobile Community setup now renders properly. - The Shop displays multiple columns correctly on affected Android devices. - Keybinds involving keys above F10 now persist correctly. - Event previews properly render Markdown. - On iOS, GIF avatars selected through the GIF picker remain animated instead of becoming static images. - Additional fixes address onboarding overlays, dangerous-download modal spacing, role selection transparency, event buttons, server uploads, and localized status text. Discord’s update is primarily a quality and performance release. Users should see the greatest immediate benefits from faster desktop interactions, improved streaming controls, better mobile behavior, and more precise server administration tools, although individual fixes may continue rolling out by platform.

Read original(opens in new tab)
RustAndroidiOSCSS+2
cloudflare3 min readCurated summary

Shedding old code with ecdysis: graceful restarts for Rust services at Cloudflare

Cloudflare’s open-source Rust library **ecdysis** enables zero-downtime restarts for high-volume network services. It preserves listening sockets and existing connections while a new process initializes, avoiding refused connections and dropped requests. After five years of production use, Cloudflare uses it to safely deploy fixes, security patches, and new features across its global infrastructure. ## Why Conventional Restarts Fail - Stopping the old process before starting the new one creates a period when no process is listening. - New clients receive `ECONNREFUSED`; even a 100 ms gap can drop hundreds of connections at a busy location. - Existing connections—including file uploads, video streams, WebSockets, and gRPC streams—are terminated when the old process exits. - `SO_REUSEPORT` allows multiple processes to bind the same port, but can orphan connections: - The kernel assigns an incoming `SYN` to one listening socket. - If that process exits before calling `accept()`, the queued connection is terminated. - This makes simply overlapping two independently bound processes unsafe for graceful upgrades. ## The ecdysis Restart Model ecdysis uses a process-forking approach pioneered by NGINX: - The parent calls `fork()` to create a child. - The child replaces itself with the new executable using `execve()`. - The child inherits the listening socket file descriptors through a named pipe shared with the parent. - The parent continues serving traffic while the child initializes. - Once the child signals readiness, the parent closes its copy of the listening socket and drains existing connections. - Both processes may briefly accept connections during the transition, but this is intentional and avoids coverage gaps. ## Crash Safety and Upgrade Requirements - The old process can fully shut down after the replacement is ready. - The new process receives time to initialize before taking over. - If initialization fails—for example, because of invalid configuration—the child exits while the parent continues serving normally. - Upgrades are serialized so that only one runs at a time, preventing cascading failures. - The unchanged listening socket ensures that new connections are not refused during the handoff. ## Rust and System Integration - ecdysis provides native Tokio stream wrappers for asynchronous Rust services. - Synchronous services can use it without an async runtime. - With the `systemd_notify` feature enabled, it integrates with systemd lifecycle notifications. - Configuring a service with `Type=notify-reload` allows systemd to track graceful upgrades correctly. Cloudflare’s approach demonstrates that graceful restarts require coordination between processes rather than simply starting a second server. Services needing reliable zero-downtime upgrades can use ecdysis to preserve connections, tolerate failed deployments, and safely roll out new Rust binaries.

Read original(opens in new tab)
RustSystemdEcdysisGraceful Restarts+3
meta3 min readCurated summary

Rust at Scale: An Added Layer of Security for WhatsApp

WhatsApp has deployed a Rust-based media security layer across billions of devices to defend against malware hidden in images, videos, PDFs, and other attachments. The system, called Kaleidoscope, validates file formats and identifies suspicious content before it reaches vulnerable downstream libraries. WhatsApp’s large-scale rollout demonstrates Rust’s production readiness and supports the company’s broader shift toward memory-safe languages. ## Media Handling as a Security Boundary - WhatsApp’s default end-to-end encryption protects messages, but shared media can still contain maliciously crafted files. - Attackers may exploit vulnerabilities in: - Operating system libraries - Media parsers - WhatsApp itself - Dangerous attachments can appear harmless, particularly when malware is concealed in images or videos. ## Lessons from the 2015 Stagefright Vulnerability - Android’s Stagefright vulnerability affected operating-system media-processing libraries. - Applications could not directly patch the vulnerable libraries, while users often took months to update their devices. - WhatsApp adapted its existing cross-platform C++ `wamedia` library to identify malformed MP4 files that could trigger vulnerable parsers. - This allowed WhatsApp to protect users faster than relying solely on operating-system updates. - Because the library automatically processes untrusted downloads, WhatsApp identified it as a strong candidate for memory-safe implementation. ## Replacing C++ with Rust - WhatsApp developed the Rust implementation alongside the original C++ version rather than performing a gradual rewrite. - Differential fuzzing, unit tests, and integration tests verified compatibility. - Key challenges included: - Increased binary size from the Rust standard library - Build-system support for WhatsApp’s many target platforms - The final implementation replaced approximately 160,000 lines of C++ with 90,000 lines of Rust, including tests. - Rust provided performance and runtime memory-use improvements. - The library was deployed across Android, iOS, Mac, Web, wearables, and other platforms. ## Kaleidoscope’s File Checks - Kaleidoscope expands beyond basic MP4 validation by checking for: - Non-conforming structures that could exploit parser differences - Embedded files and scripts in PDFs - Files that disguise their type through spoofed extensions or MIME types - Known dangerous formats such as executables and applications - These checks support safer handling in WhatsApp’s user interface and help defend against malicious attachments and unofficial clients. - The system cannot prevent every attack, but it adds an important defense-in-depth layer. ## WhatsApp’s Broader Security Strategy - WhatsApp distributes the libraries each month to billions of phones, computers, watches, and browsers across WhatsApp, Messenger, and Instagram. - The company describes this as the largest deployment of Rust code across diverse end-user platforms. - Its wider security program includes: - End-to-end encrypted messages, calls, and backups - Key transparency and additional calling protections - Fuzzing, static analysis, audits, and attack-surface monitoring - CVE reporting and an expanded bug bounty program - WhatsApp’s vulnerability strategy focuses on minimizing attack surface, strengthening remaining C and C++ code, and choosing memory-safe languages for new development. - Existing protections include control-flow integrity, hardened allocators, safer buffer APIs, specialized developer training, and automated analysis. WhatsApp plans to accelerate Rust adoption, particularly for security-sensitive, cross-platform components that process untrusted input. Its media library rollout provides evidence that Rust can deliver both memory safety and performance at global consumer scale.

Read original(opens in new tab)
RustApplication SecurityCross PlatformMemory Safety+3
cloudflare2 min readCurated summary

What came first- the CNAME or the A record

A memory-optimization change in Cloudflare’s 1.1.1.1 resolver accidentally reordered DNS records, placing CNAMEs after A/AAAA records. Although DNS record order is generally considered irrelevant, some clients—including glibc’s `getaddrinfo`—process answers sequentially and require CNAMEs to appear first. The resulting failures affected users globally until Cloudflare reverted the release. ## Incident Timeline - **December 2, 2025:** The record-reordering change was added. - **December 10:** It reached the testing environment. - **January 7, 2026:** Global deployment began. - **January 8, 17:40 UTC:** The change reached 90% of servers. - **18:19:** The incident was declared. - **18:27:** The release was reverted. - **19:55:** The revert completed and the impact ended. ## How CNAME Chains Are Resolved - A hostname may point through multiple aliases before reaching an A or AAAA record: - `www.example.com → cdn.example.com → server.cdn-provider.com → 198.51.100.1` - Each record has its own TTL and may expire independently. - If only part of a chain expires, 1.1.1.1 can reuse the cached portion and resolve only the missing records. - The resolver then combines the cached CNAME records with newly resolved address records. ## The Memory Optimization That Changed Ordering - Previously, the resolver created a new list: - Insert the existing CNAME chain first. - Append the newly resolved A/AAAA records afterward. - The optimization avoided allocations and copies by appending new CNAME records directly to the existing answer list. - This caused some responses to place address records before CNAME records. ## Why Some DNS Clients Failed - Many clients treat answer-section ordering as irrelevant, but some parse records sequentially. - These clients: - Start by looking for records matching the original queried name. - Update the expected name when they encounter a CNAME. - Accept the corresponding A or AAAA record only after that update. - With the expected order, the client sees the CNAME first and then accepts the address record. - With the address record first, it ignores the address because it does not yet match the expected name. After encountering the CNAME, there are no records left to process, so it reports an empty response. - The affected implementation included glibc’s `getaddrinfo`, widely used for DNS resolution on Linux. The incident demonstrates that even seemingly insignificant DNS response ordering can matter in practice. Resolver implementations should preserve CNAME-before-address ordering, and DNS clients should avoid assuming that record order is meaningful unless the protocol explicitly requires it.

Read original(opens in new tab)
RustDnsNetworkingGlibc+3
discord3 min readCurated summary

Discord Patch Notes: December 8, 2025

Discord’s December 8, 2025 patch focuses on faster uploads, quicker video playback, a redesigned Desktop User Settings experience, and broad reliability improvements. Major infrastructure updates include resumable media uploads, Rust-based retry logic, and CDN changes that substantially reduce video startup latency. The release also contains numerous desktop, mobile, chat, localization, and UI bug fixes. ## Uploads, Performance, and Settings - Media uploads now support resuming after interruptions, which is especially useful for large files. - New Rust-based retry logic reduced upload latency by about 15% overall: - iOS: 21% faster - Android: 10% faster - CDN configuration changes improved video embed startup times: - P90 latency fell from 600 ms to 275 ms. - P95 latency fell from 1,250 ms to 658 ms. - Rare cases involving videos taking up to 40 seconds to start were also addressed. - Desktop User Settings received a redesign intended to make options easier to find, with metrics indicating improved usability. ## Platform and General Improvements - Mobile apps upgraded from React Native 0.78 to 0.81.4. - Desktop upgraded to Electron 37. - Server Hub search is now more performant, particularly for hubs connected to more than 50 servers. - Linux clients no longer display duplicate titlebars. - Various desktop rendering issues were fixed, including stray white pixels, incorrect scrollbars, spacing problems, and MacOS traffic-light alignment. - Server folders, unread channels, invite searches, profile previews, and image previews received multiple correctness fixes. - Emoji Studio now properly re-centers images when using Reset Image. - Search in Settings no longer produces empty categories for certain queries. - Several localization issues were corrected, including “Edit Profiles” and “Search in DMs.” - Users can now dismiss the Upcoming Events banner in voice channels. - Event invites, event popouts, custom-location sharing, and event Join buttons were repaired. - Android and iOS fixes address server folders, server lists, nameplates, role settings, and profile customization controls. ## Chat and Messaging Fixes - Multiline typing in the Desktop Message Forward interface no longer causes visual shifting. - Spoiler timestamps now render correctly before content is revealed. - Mobile fixes improve message-row sizing, reply navigation, forum interactions, and thread rendering. - Android code blocks once again use monospace formatting. - Sticker Picker and Emoji Picker navigation now behaves correctly on Android. - Scrollbars remain interactive when thread previews are present. - Inbox navigation to thread-creation messages has been fixed. - Forum post search no longer includes excessive blank space. - Additional iOS fixes address swiping to reply, navigating to message replies, and displaying the thread spine connecting avatars to threads. Overall, the update combines meaningful performance improvements with a large collection of visual and interaction fixes. Users who depend on large uploads, embedded video, mobile chat, or Desktop settings should benefit most, though the changes may continue rolling out across platforms.

Read original(opens in new tab)
RustAndroidiOSElectron+4
datadog1 min readCurated summary

Evolving our real-time timeseries storage again: Built in Rust for performance at scale | Datadog

The provided content does not include the tech blog post itself. It contains Datadog’s navigation menu and a link to an engineering article titled “Rust Timeseries Engine,” but no article text to summarize. Please provide the blog post content or its URL, and I can summarize it in the requested format.

Read original(opens in new tab)
RustObservabilityTime Series DatabaseDatabase Engine
datadog4 min readCurated summary

Evolving our real-time timeseries storage again: Built in Rust for performance at scale

Datadog built a sixth-generation real-time timeseries database in Rust to keep pace with rapidly growing metric volume, cardinality, and query complexity. The new engine is designed for high throughput and low latency, reportedly achieving 60× higher ingestion performance and 5× faster peak-scale queries. Its development reflects a long evolution from general-purpose databases toward a purpose-built system with tighter control over storage, I/O, and execution. ## Datadog’s Metrics Storage Architecture - The metrics platform includes ingestion, enrichment, real-time and long-term storage, querying, and alerting. - This post focuses on real-time storage, which is split into two independently deployed services: - **RTDB:** Stores raw metric tuples of `<timeseries_id, timestamp, value>`, performs aggregations, and serves recent data. - **Index database:** Stores metric identifiers and their tags as `<timeseries_id, tags>`. - A storage router distributes incoming metrics across RTDB nodes based on load. - The query service contacts the relevant RTDB and index nodes, retrieves results, and combines them. - Each RTDB node includes: - An ingestion subsystem - A storage engine - A durability snapshot module - A gRPC query layer - Throttlers for resource management - A shared control plane coordinating these components ## Generation 1: Cassandra - Cassandra provided strong write scalability and a familiar operational model. - It was influenced by systems such as OpenTSDB and HBase. - Its main weaknesses were: - Limited flexibility for real-time queries - Difficulty supporting complex alerting and analytical workloads - Inefficient retrieval of large datasets - These limitations prompted Datadog to move to Redis. ## Generation 2: Redis - Redis improved read performance and offered a flexible, easy-to-understand storage model. - Datadog avoided Redis’s built-in clustering for reliability reasons, requiring the team to operate many independent instances. - Important drawbacks included: - Single-threaded execution limiting snapshotting during live traffic - Severe but uncommon memory-management and threading failures - Serialization and cross-process communication overhead - Inefficient memory layout, disk I/O, and CPU usage at scale - Redis nevertheless provided valuable operational insight and clarified the need for a purpose-built engine with direct control over I/O and system resources. ## Generation 3: MDBM and Memory-Mapped I/O - MDBM provided a memory-mapped key-value store based on `mmap`. - The operating system’s page cache loaded database pages on demand, making disk-backed data behave similarly to in-memory structures. - This simplified storage interactions initially, but performance degraded as workloads intensified. - Memory-mapped I/O introduced subtle performance and correctness concerns, leading Datadog to conclude that explicit I/O management would scale better. ## Generation 4: A Go-Based B+ Tree - Datadog replaced MDBM with a custom B+ tree written in Go. - The engine supported a thread-per-core-oriented design, with Go’s scheduler providing a useful foundation. - This change significantly improved throughput and latency. - It also created a platform that could be optimized more aggressively for Datadog’s workload. ## Generation 5: DDSketch and RocksDB - Datadog introduced DDSketch to support distribution metrics and accurate percentile estimation. - The existing Go engine was optimized for scalar floating-point values and was difficult to extend for sketches. - RocksDB was therefore integrated to store DDSketch data, offering flexibility and strong performance. - Over time, maintaining separate storage technologies created pressure to build a unified engine capable of handling multiple metric types efficiently. ## The Move Toward a New Engine - The progression from Cassandra to Redis, MDBM, a custom Go B+ tree, and RocksDB shows a pattern of replacing general-purpose components as scale and workload diversity increased. - Each generation solved important problems but introduced new operational or architectural trade-offs. - Datadog ultimately needed a unified, purpose-built storage system with: - High-throughput ingestion - Low-latency queries - Better support for high-cardinality data - Efficient handling of different metric types - More direct control over concurrency, memory, and I/O - The sixth generation addresses these requirements through a Rust-based real-time timeseries database. Datadog’s experience suggests that general-purpose storage systems can be effective early on, but sustained growth eventually favors a specialized engine. The practical lesson is to optimize existing infrastructure first while developing a purpose-built replacement before scale and workload complexity make incremental fixes insufficient.

Read original(opens in new tab)
RustDatabase DesignRedisGrpc+4
datadog3 min readCurated summary

How we built a real-time, client-side noise suppression library without server dependencies

Datadog’s CoScreen team needed high-quality noise suppression that could run in real time on client devices and integrate with WebRTC. Since existing solutions were either too slow, server-dependent, expensive, or difficult to embed, they built and open-sourced **dtln-rs**, a portable Rust library based on the DTLN model. It processes one second of audio in about 33 ms on an M1 MacBook Pro and supports WebAssembly, Node.js, and native clients. ## Introducing dtln-rs - dtln-rs is a lightweight, open-source noise reduction library based on the Dual-Signal Transformation LSTM Network (DTLN). - It can produce: - A WebAssembly module - A native Rust library - A Node.js native module - The library is designed to integrate with WebRTC-based applications. - Datadog also released a demo showing how to embed the filter in an application or webpage. ## Demonstrating Real-World Noise Suppression - The project was motivated by common remote-work disruptions, including lawn mowers and other background noise. - In one test, the filter removed a neighbor’s lawn mower so effectively that a colleague could not tell it was running. - The team used this result as evidence that the embedded library could provide meaningful value to CoScreen users. ## How DTLN Enables Real-Time Processing - AI noise suppression learns to distinguish desired speech from unwanted background sounds. - DTLN uses a short-time Fourier transform (STFT) to divide audio into smaller segments and analyze the magnitude of different frequencies. - It also uses phase information, which describes the starting position of each frequency in the sound wave. - A model analyzes magnitude and phase data to determine which parts are speech and which are noise. - Its LSTM-based architecture can adapt to different environments, such as: - Air-conditioner hum - Cafe conversations - Paper rustling - The combination of deep learning and efficient signal processing allows DTLN to operate with near-instantaneous latency. ## Why Existing Noise Suppression Solutions Were Insufficient - Many advanced machine-learning models require powerful backend servers, with processed audio sent back over the network. - This approach adds latency, infrastructure complexity, and operating costs. - WebRTC remains widely adopted but generally relies on older, built-in noise reduction techniques. - Earlier solutions such as RNNoise can reduce noise but often do not match the quality of newer commercial systems. - Although Web Audio and WebAssembly make custom client-side processing possible, implementation still requires substantial engineering effort. - Large companies can deploy specialized servers and models trained on enormous speech datasets, but smaller teams may not have the resources to do so. - CoScreen’s search for an alternative led to DTLN, which could run in real time on standard hardware and be embedded directly into client applications. ## Practical Recommendation For WebRTC applications needing client-side, real-time noise suppression, dtln-rs offers a portable alternative to expensive server-based services. Its Rust foundation and support for WebAssembly, Node.js, and native targets make it suitable for web, desktop, and embedded clients.

Read original(opens in new tab)
RustDatabase DesignDeep LearningWebassembly+4
figma2 min readCurated summary

Supporting Faster File Load Times with Memory Optimizations in Rust | Figma Blog

Figma improved server-side file loading by reducing the memory overhead of its Rust data structures. Replacing per-node `BTreeMap`s with compact sorted vectors made deserialization faster and cut memory usage for large files by nearly 25%, despite worse theoretical operation complexity. The team also explored packing field IDs into unused pointer bits, potentially storing the same information in fewer bytes. ## Smaller, Memory-Efficient Maps - Figma files consist of nodes, each represented by properties such as type, parent, position, and dimensions. - Nodes were stored as `BTreeMap<u16, u64 pointer>` structures because ordered iteration was required for serialization. - Profiling showed these maps consumed more than 60% of a file’s memory, even though they stored metadata rather than large data payloads. - The schema contains fewer than 200 possible fields, and nodes typically contain only a subset of them—about 60 properties on average. - Figma replaced each `BTreeMap` with a sorted flat vector of `(field ID, pointer)` pairs. - Although vectors have theoretically slower insertion, lookup, and editing, their compact linear layout is more cache-friendly and faster during deserialization. - The deployed change reduced memory usage by nearly 25% for large files and improved file-loading performance. ## Saving More Memory with Bit Stuffing - The team also investigated storing the field ID inside the pointer itself. - While pointers are nominally 64 bits, x86 systems currently use only the lower 48 bits for memory addresses, leaving 16 bits available. - Figma’s field IDs require exactly 16 bits, allowing a single `u64` to contain both: - A 16-bit field ID - A 48-bit memory pointer - This representation could eliminate the separate field-ID storage and further reduce memory overhead. - The approach had not yet been productionized because relying on unused pointer bits is architecture-dependent and could change in the future. Figma’s results demonstrate that practical memory layout and CPU cache behavior can outweigh Big O complexity. For compact, bounded data structures, flat vectors—and carefully considered bit packing—can deliver substantial improvements in both memory efficiency and load speed.

Read original(opens in new tab)
RustPerformance OptimizationMemory OptimizationData Structures+2
discord3 min readCurated summary

Discord Patch Notes: June 3, 2025

Discord’s June 3, 2025 patch focuses on performance, reliability, media playback, and cross-platform consistency. Major improvements include ARM infrastructure migration, faster video startup, better mobile image compression, persistent voice-message playback, and early adoption of shared Rust components. The release also includes framework upgrades and a broad set of UI, navigation, billing, accessibility, and platform-specific bug fixes. ## Infrastructure and Performance - Discord is migrating parts of its infrastructure to ARM hardware. - The change has reduced per-core load and improved latency. - ARM machines also lower energy consumption. - Keyframe-generation changes reduced video and stream startup latency by more than 10% on average. - Discord migrated one core store to a shared, multi-platform Rust implementation. - Early results showed improvements in memory and CPU usage, crashes, and other reliability metrics. - More stores and APIs are planned for migration. - Desktop clients were upgraded to Electron 35. - Mobile clients moved to React Native 0.78, while all clients upgraded to React 19, with incremental performance improvements and no significant regressions. ## Voice, Images, and App Links - Voice messages on desktop and web now support playback-speed controls. - Playback position is preserved when users close the app or navigate away. - Mobile image compression now adapts to source resolution. - Lower-resolution images receive higher-quality embeds. - Median upload latency has decreased across platforms. - Discord revamped app-link handling to resolve Android linking failures and improve compatibility with modern apps. ## General Bug Fixes - Fixed an iOS Billing Settings issue involving an empty entry field and moved Manage Nitro to the top of the page. - Fixed Android poll results that could not be swiped horizontally. - Soundboard uploads now work correctly in Firefox. - Corrected mobile profile scrolling that exposed blank space. - Fixed multi-line status rendering, Custom Status emoji alignment, and unreadable light-mode text. - Starting a Desktop settings search with “M” no longer freezes the application. - Mac popouts now correctly render Unicode emojis in full-screen mode. - Fixed duplicate server entries caused by lurking through Discovery. - Corrected server settings save prompts, profile badge previews, folder rendering in Safari, and profile banner updates on iOS. - Fixed Soundboard picker placement, event-link copying, Shop navigation, 2FA login-state messaging, and Expression Picker spellcheck replacement. - Removed a redundant Linux title bar and corrected channel-list alignment for servers with many channels. - Profile edits now prompt consistently for saving across platforms. - Fixed stale forwarding-search results, invite-modal corner rendering, light-mode member text, overlay popups, privacy-page dividers, folder mention badges, and stream-invite display names. - Improved Discovery scaling for different window sizes. - Fixed “Go to Shop” links from avatar decoration and profile-effect previews. - Corrected rendering issues involving passkey backup codes and several additional interface elements. The changes had been merged but might still be rolling out by platform. Users encountering remaining issues can report them through Discord’s community bug megathread or test upcoming iOS builds through TestFlight.

Read original(opens in new tab)
RustReactAndroidiOS+3
datadog3 min readCurated summary

Squeezing every millisecond: How we rebuilt the Datadog Lambda Extension in Rust

Datadog rewrote its AWS Lambda extension from Go into Rust to overcome the performance limits of adapting its large, host-oriented Datadog Agent to Lambda’s constrained environment. The redesign reduced cold-start latency by 82%, memory usage by 40%, and binary size from 55 MB to 7 MB. The project succeeded by narrowing the problem, enforcing performance budgets from the beginning, and designing specifically for Lambda’s execution model. ## Why the Original Extension Needed to Change - The Lambda extension runs as a sidecar process, collecting logs, metrics, traces, profiles, and process data asynchronously. - It was originally based on the Datadog Agent, which is designed for hosts, containers, and clusters. - The Agent’s fairness, buffering, caching, and high-throughput features introduced unnecessary overhead in Lambda. - Optimization attempts included: - Removing dependencies with build tags - Compressing binaries with UPX - Eliminating unnecessary `init` methods - Exploring Go plugins for lazy loading - These changes could not reduce additional cold-start latency below roughly 450–500 milliseconds. ## Why a Rewrite—and Why Rust - Rewrites are risky because they can lose undocumented invariants, reproduce subtle bugs, and create the burden of supporting two systems. - The team concluded that Lambda represented a fundamentally different scale and workload from the general-purpose Datadog Agent. - Rust was well suited because: - Memory safety reduces the risk of crashes and data races. - Extension crashes also terminate the Lambda function and trigger another cold start. - Rust produces small binaries with limited runtime overhead. - Lambda targets a narrow platform set: Amazon Linux on x86 and Arm. - Compile-time concurrency guarantees support reliable multithreaded code. - A hackathon prototype demonstrated enough potential to begin the full rewrite, named Project Bottlecap. ## Project Bottlecap’s Design Constraints - The extension had to minimize interference with the function handler, especially because many Lambda functions serve latency-sensitive APIs. - Telemetry work should occur after the handler returns whenever possible. - The team also minimized post-runtime duration—the CPU time added after normal function execution. - Performance was monitored from the start: - Dashboards and alerts tracked cold-start overhead. - Every pull request was benchmarked. - Regressions were investigated before merging. - The team accepted targeted tradeoffs for speed, including manually implementing AWS API calls and request signing instead of using SDKs that added too much overhead. - The design emphasized optionality because Lambda workloads range from small API functions to large asynchronous batch jobs. - Planned flush strategies included: - Flushing at the end of an invocation for infrequently called or CPU-constrained functions - Periodic or in-invocation flushing for workloads needing different latency and resource tradeoffs The practical lesson is that software optimized for large, long-running systems may be fundamentally unsuitable for serverless runtimes. When optimization reaches a hard performance floor, a focused rewrite—constrained by the target environment and measured continuously—can deliver major gains.

Read original(opens in new tab)
RustDatabase DesignDatadogObservability+3
datadog1 min readCurated summary

Squeezing every millisecond: How we rebuilt the Datadog Lambda Extension in Rust | Datadog

The provided text does not contain the blog post’s main article body. It mainly includes Datadog’s navigation links and a promotional announcement that Gartner named Datadog a Leader in the 2026 Magic Quadrant for Observability Platforms. The linked page URL suggests the intended post concerns Datadog’s AWS Lambda extension and Rust, but no technical details are included. ## Datadog’s Observability Platform Recognition - Datadog promotes its recognition as a Gartner Magic Quadrant Leader. - The surrounding navigation highlights products for: - Infrastructure and cloud monitoring - Application performance monitoring - Logs and data observability - Security - Digital experience monitoring - Software delivery - Service management - AI and platform capabilities ## Referenced Lambda and Rust Article - The navigation links to an engineering post titled around the Datadog Lambda Extension and Rust. - However, the supplied content contains no explanation of: - Why Rust was selected - The extension’s architecture - Performance or resource improvements - Deployment and compatibility considerations The article body is needed for a substantive technical summary.

Read original(opens in new tab)
RustDatadogObservabilityServerless+1