Database Design | Techlist.io

tossNov 3, 2025Original article

Working as a QA in a (opens in new tab)

Toss Place implements a dual-role QA structure where managers are embedded directly within product Silos from the initial planning stages to final deployment. This shift moves QA from a final-stage bottleneck to a proactive partner that enhances delivery speed and stability through deep historical context and early risk mitigation. Consequently, the organization has transitioned to a culture where quality is viewed as a shared team responsibility rather than a siloed functional task. ### Integrating QA into Product Silos * QA managers belong to both a central functional team and specific product units (Silos) to ensure they are involved in the entire product lifecycle. * Participation begins at the OKR design phase, allowing QA to align testing strategies with specific product intentions and business goals. * Early involvement enables accurate risk assessment and scope estimation, preventing the "shallow testing" that often occurs when QA only sees the final product. ### Optimizing Spec Reviews and Sanity Testing * The team introduced a structured flow consisting of Spec Reviews followed by Q&A sessions to reduce repetitive discussions and information gaps. * All specification changes are centralized in shared design tools (such as Deus) or messenger threads to ensure transparency across all roles. * "Sanity Test" criteria were established where developers and QA agree on "Happy Case" validations and minimum spec requirements before development begins, ensuring everyone starts from the same baseline. ### Collaborative Live Monitoring * Post-release checklists were developed to involve the entire Silo in live monitoring, overcoming the limitations of having a single QA manager per unit. * This collaborative approach encourages non-technical roles to interact with the live product, reinforcing the culture that quality is a collective team responsibility. ### Streamlining Issue Tracking and Communication * The team implemented a "Send to Notion" workflow to instantly capture messenger-based feedback and ideas into a structured, prioritized backlog. * To reduce communication fragmentation, they transitioned from Jira to integrated Messenger Lists and Canvases, which allowed for centralized discussions and faster issue resolution. * Backlogs are prioritized based on user experience impact and release urgency, ensuring that critical bugs are addressed while minor improvements are tracked for future cycles. The success of these initiatives demonstrates that QA effectiveness is driven by integration and autonomy rather than rigid adherence to specific tools. To achieve both high velocity and high quality, organizations should empower QA professionals to act as product peers who can flexibly adapt their processes to the unique needs and data-driven goals of their specific product teams.

Database Design Jira Notion Quality Assurance+4

datadogOct 15, 20253 min readCurated summary

Failure is inevitable: Learning from a large outage, and building for reliability in depth at Datadog

Datadog’s March 2023 outage exposed a fundamental weakness in its reliability strategy: although 40–50% of production Kubernetes nodes remained operational, customers experienced the platform as entirely unavailable. The incident showed that preventing every failure is impossible and that systems must instead continue delivering useful, accurate service when components fail. Datadog consequently began redesigning products around graceful degradation, prioritizing data preservation, fresh information, and partial results. ## Lessons from the March 2023 Incident - An unsupervised global update triggered a restart interaction that disconnected roughly 50–60% of production Kubernetes nodes. - The web interface recovered quickly, but logs, metrics, alerts, traces, and other core features became unavailable. - Pages loaded without displaying customer data, creating a nearly complete outage from the user’s perspective. ## Limits of Traditional Root-Cause Analysis - Datadog identified the legacy global security-update mechanism as the immediate trigger and disabled it. - Fixing that mechanism alone could not address the broader class of failures caused by certificates, configuration changes, overloads, date-handling bugs, or other unexpected events. - The company concluded that resilience requires reducing the impact of failures, not merely preventing one specific failure mode. ## Why Partial Infrastructure Became a Total User-Facing Failure - Datadog’s systems historically favored complete correctness over partial visibility. - For example, metric queries could wait until all relevant tags were processed to avoid showing misleading values or triggering false alerts. - During a large outage, this behavior created a “square-wave” failure: missing some data caused the system to show no data. - Ordered queues could stall fresh results behind stuck work, retries could overload already-strained services, and node-specific processing could make surviving capacity ineffective. - The underlying design assumption was that systems should either function fully or stop, rather than degrade while continuing to provide value. ## Prioritizing Graceful Degradation Datadog shifted from relying primarily on redundancy and “never-fail” architectures to explicitly designing for inevitable failures. - Customer data should never be lost, even if delivery is delayed. - Fresh, real-time data should take priority over stale backlog processing. - Systems should provide partial but accurate results whenever possible instead of returning nothing. ## Persistent Storage at the Start of Processing Pipelines - The outage caused a limited but non-zero amount of irreversible customer data loss. - Some pipelines acknowledged data before writing it to replicated storage, leaving unreplicated data only in memory or on a local disk. - When a node failed, that data disappeared and could not be recovered through agent retries. - After the node loss, surviving intake nodes also struggled to write to downstream replicated stores. - Their memory and local-disk buffers eventually filled, causing additional data loss as the outage continued. - Datadog therefore identified persistent intake storage as a key requirement for preserving data during large-scale failures. The broader recommendation is to design systems not only to prevent outages, but also to remain useful during them: preserve every accepted event, prioritize current information, and expose accurate partial results instead of failing completely.

Read original

Database Design Kubernetes Data Loss Prevention Graceful Degradation+4

datadogOct 1, 20253 min readCurated summary

Inside Husky’s query engine: Real-time access to 100 trillion events

Datadog’s Husky event store is designed not merely to store more than 100 trillion events, but to make them queryable interactively at massive scale. Its query engine handles schemaless data, highly variable tenant workloads, petabytes of object-store data, and both highly selective searches and broad analytics queries. The architecture combines distributed planning, metadata-driven pruning, coordinated execution, and reader-level optimizations to minimize the data and fragments that must be scanned. ## Husky’s Data and Query Workloads - Events contain a timestamp and flexible attributes. - Data shapes vary widely: - Logs, network events, and traces have different schemas. - Tenants may produce either a few large events or enormous numbers of small events. - Most queries fall into two categories: - **Needle-in-a-haystack searches**, such as finding a specific IP connection, error message, or trace. - **Analytics-style searches**, such as time series, grouped breakdowns, and distributions. - The engine must also support raw event retrieval and more complex queries such as joins. ## Distributed Query Path Husky divides query execution among four multi-tenant services deployed across regions and data centers. ### Query Planner - Serves as the main entry point for event-store queries. - Resolves query context, validates and throttles requests, and integrates with other Datadog data stores. - Applies optimizations and statistics to split queries into time-based steps. - Schedules those steps on orchestrators and merges their results into the final response. ### Query Orchestrator - Acts as the gateway to Husky’s stored data. - Fetches fragment metadata, including: - File paths and versions - Row counts - Timestamp boundaries - Zone maps for query matching - Dispatches only relevant fragments to reader nodes. - Uses zone-map pruning to reduce downstream work by up to 60% for structured events and about 30% on average. - Aggregates results after fragment processing, which can require more computation than query planning. ### Metadata Service - Provides an abstraction over FoundationDB clusters. - Preserves atomicity during operations such as compaction, preventing duplicate data from appearing in query results. - Separates FoundationDB implementation details from the rest of the query system. - Must work within FoundationDB’s five-second transaction limit. ### Reader Service - Receives a query and selected fragments, then returns results quickly. - Performs the direct scan and execution work over fragment data. - Contains multiple optimizations intended to keep queries interactive despite scanning data stored in blob storage at extreme scale. ## Minimizing Data Scans The reader service follows the principle that the fastest query is the one that avoids unnecessary work. - Scanning less data reduces both latency and storage costs. - Touching fewer fragments limits expensive object-store operations. - This is especially important because Husky stores millions of fragments daily and cannot afford multiple blob-storage GET requests for every fragment in every query. ## Row Groups and Reader Execution - Fragments can contain millions of rows, making fully in-memory processing risky and potentially causing memory pressure or failures. - To limit data retrieval, fragments are physically organized into **row groups**. - Row groups allow the reader to fetch only portions of a fragment needed for a query rather than loading the entire file. - The reader uses an iterator-based execution model inspired by the Volcano query-processing architecture. - The provided article ends as it begins explaining how this row-group layout supports efficient iterator-based query execution.

Read original

Database Design Data Architecture Foundationdb Husky+3

datadogSep 18, 20253 min readCurated summary

From hand-tuned Go to self-optimizing code: Building BitsEvolve

Datadog found that small Go-level optimizations can produce substantial infrastructure savings when applied to heavily used, autoscaled services. Manual work—such as removing bounds checks and prioritizing common input paths—delivered improvements ranging from 25% to over 90% in targeted functions. These successes also revealed the need to automate expert optimization techniques through systems like Datadog’s internal BitsEvolve. ## Finding Hotspots That Matter - Micro-optimizations are worthwhile when: - Functions run millions or billions of times. - Services are aggressively autoscaled, allowing CPU savings to reduce machine counts. - Resource usage drops measurably. - Datadog focused on high-throughput services processing timeseries tags and values. - Individual hotspots sometimes represented only 0.5% of compute, but repeated savings could add up to tens of thousands of dollars annually. - The broader goal was a 5–10% reduction in CPU usage across many improvements. ## Removing Bounds Checks from `NormalizeTag` - `NormalizeTag` called `isNormalizedASCIITag`, a frequently executed validator for ASCII tag strings. - AI coding tools suggested changes that were correct but produced no measurable performance gains. - Examining Go assembly with Compiler Explorer revealed two `runtime.panicBounds` calls per loop iteration. - Restructuring the loop eliminated unnecessary bounds checks and enabled further tuning. - The function became 25% faster, reducing service CPU usage by 0.75% and producing projected annual savings of tens of thousands of dollars. ## Using Observability to Optimize for Real Inputs - `NormalizeTagArbTagValue` handled arbitrary input, including invalid UTF-8 and binary data, and consumed 4.5% of CPU in its processing service. - Production data showed: - Nearly all inputs were ASCII. - UTF-8 appeared in fewer than 3% of cases. - Invalid UTF-8 represented less than 0.01% of inputs. - A fast path optimized for common ASCII data made the function more than 90% faster without reducing correctness or safety. - The change generated projected annual savings of hundreds of thousands of dollars. - The result demonstrated that observability is essential: optimization decisions should reflect actual workloads rather than hypothetical edge cases. ## From Manual Optimization to Automation - Deep performance tuning requires specialized knowledge of profiling, compiler behavior, assembly, and workload analysis. - Although the results can be valuable, the process is time-consuming and difficult to scale across a large organization. - Datadog wanted to move beyond isolated “heroic” optimizations toward a repeatable and automated process. - The manual techniques used by performance engineers became the foundation for heuristics in BitsEvolve, an internal agentic system intended to optimize code systematically. Datadog’s experience suggests that organizations should combine production observability with compiler-level analysis, prioritize high-impact hot paths, and automate proven optimization patterns so performance gains do not depend solely on a small group of experts.

Read original

Database Design Observability Go Ai Assisted Coding+4

datadogAug 12, 20253 min readCurated summary

Scaling down to speed up: How we improved efficiency of live process metrics by 100x

Datadog redesigned its real-time Processes and Containers pipeline to avoid collecting high-frequency metrics that users never see. By limiting 2-second collection to hosts actively viewed and using standard 10-second data for sorting, the company reduced real-time traffic by over 100x, cut infrastructure costs by 98%, and lowered Agent resource usage. The approach also improved scalability without sacrificing the live investigation experience. ## Original Real-Time Collection Model - Datadog Agents normally collect process and container metrics every 10 seconds. - When a user opened a live Processes or Containers view, all hosts in that tenant switched to 2-second collection. - This supported near-real-time monitoring similar to `htop`, but across distributed infrastructure. - As tenants grew, the pipeline had to process millions of processes per second, even though users typically viewed only around 50 processes or containers. - Live sorting required keeping all tenant data in memory on a single server, limiting horizontal scaling and forcing vertical scaling. ## Refocusing on User-Visible Data - Most collected metrics were never displayed to users. - Datadog determined that real-time collection only needed to be enabled for hosts running the processes or containers currently in view—up to roughly 50 hosts per user. - Internal telemetry suggested this could reduce traffic by more than 100x. - This required tracking active host subscriptions and updating them as users navigated the product. - Because sorting occurred every 10 seconds, it did not need 2-second data. Datadog switched live views to use the existing 10-second metrics, aligning live and historical sorting logic. ## Host Subscription Filtering - A proof of concept added host subscriptions to the live data servers. - Servers filtered Kafka payloads and discarded data for hosts without active subscriptions. - This immediately reduced: - Memory usage by 85% - CPU usage by 33% - The improvement came from storing fewer live metrics and processing fewer incoming payloads. - The prototype confirmed that filtering preserved product behavior while simplifying sorting. ## Moving Filtering Earlier in the Pipeline - Late filtering improved live data servers but still left unnecessary work for the rest of the system and customer-side Datadog Agents. - Datadog therefore planned to propagate subscription state to the intake service. - Live data servers publish users’ active host sets over Kafka once per second. - The intake service consumes this information and decides which hosts should activate 2-second process and container metric collection. - This allows real-time collection to be restricted to hosts users are actively investigating while maintaining responsive live views. Datadog’s redesign demonstrates that real-time systems scale more effectively when they prioritize data users can actually see. Filtering at intake, limiting high-frequency collection to subscribed hosts, and reusing standard-resolution data for sorting provide a simpler and more economical architecture without eliminating live functionality.

Read original

Database Design Datadog Apache Kafka Docker & Containers+4

datadogAug 4, 20254 min readCurated summary

Evolving our real-time timeseries storage again: Built in Rust for performance at scale

Datadog built a sixth-generation real-time timeseries database in Rust to keep pace with rapidly growing metric volume, cardinality, and query complexity. The new engine is designed for high throughput and low latency, reportedly achieving 60× higher ingestion performance and 5× faster peak-scale queries. Its development reflects a long evolution from general-purpose databases toward a purpose-built system with tighter control over storage, I/O, and execution. ## Datadog’s Metrics Storage Architecture - The metrics platform includes ingestion, enrichment, real-time and long-term storage, querying, and alerting. - This post focuses on real-time storage, which is split into two independently deployed services: - **RTDB:** Stores raw metric tuples of `<timeseries_id, timestamp, value>`, performs aggregations, and serves recent data. - **Index database:** Stores metric identifiers and their tags as `<timeseries_id, tags>`. - A storage router distributes incoming metrics across RTDB nodes based on load. - The query service contacts the relevant RTDB and index nodes, retrieves results, and combines them. - Each RTDB node includes: - An ingestion subsystem - A storage engine - A durability snapshot module - A gRPC query layer - Throttlers for resource management - A shared control plane coordinating these components ## Generation 1: Cassandra - Cassandra provided strong write scalability and a familiar operational model. - It was influenced by systems such as OpenTSDB and HBase. - Its main weaknesses were: - Limited flexibility for real-time queries - Difficulty supporting complex alerting and analytical workloads - Inefficient retrieval of large datasets - These limitations prompted Datadog to move to Redis. ## Generation 2: Redis - Redis improved read performance and offered a flexible, easy-to-understand storage model. - Datadog avoided Redis’s built-in clustering for reliability reasons, requiring the team to operate many independent instances. - Important drawbacks included: - Single-threaded execution limiting snapshotting during live traffic - Severe but uncommon memory-management and threading failures - Serialization and cross-process communication overhead - Inefficient memory layout, disk I/O, and CPU usage at scale - Redis nevertheless provided valuable operational insight and clarified the need for a purpose-built engine with direct control over I/O and system resources. ## Generation 3: MDBM and Memory-Mapped I/O - MDBM provided a memory-mapped key-value store based on `mmap`. - The operating system’s page cache loaded database pages on demand, making disk-backed data behave similarly to in-memory structures. - This simplified storage interactions initially, but performance degraded as workloads intensified. - Memory-mapped I/O introduced subtle performance and correctness concerns, leading Datadog to conclude that explicit I/O management would scale better. ## Generation 4: A Go-Based B+ Tree - Datadog replaced MDBM with a custom B+ tree written in Go. - The engine supported a thread-per-core-oriented design, with Go’s scheduler providing a useful foundation. - This change significantly improved throughput and latency. - It also created a platform that could be optimized more aggressively for Datadog’s workload. ## Generation 5: DDSketch and RocksDB - Datadog introduced DDSketch to support distribution metrics and accurate percentile estimation. - The existing Go engine was optimized for scalar floating-point values and was difficult to extend for sketches. - RocksDB was therefore integrated to store DDSketch data, offering flexibility and strong performance. - Over time, maintaining separate storage technologies created pressure to build a unified engine capable of handling multiple metric types efficiently. ## The Move Toward a New Engine - The progression from Cassandra to Redis, MDBM, a custom Go B+ tree, and RocksDB shows a pattern of replacing general-purpose components as scale and workload diversity increased. - Each generation solved important problems but introduced new operational or architectural trade-offs. - Datadog ultimately needed a unified, purpose-built storage system with: - High-throughput ingestion - Low-latency queries - Better support for high-cardinality data - Efficient handling of different metric types - More direct control over concurrency, memory, and I/O - The sixth generation addresses these requirements through a Rust-based real-time timeseries database. Datadog’s experience suggests that general-purpose storage systems can be effective early on, but sustained growth eventually favors a specialized engine. The practical lesson is to optimize existing infrastructure first while developing a purpose-built replacement before scale and workload complexity make incremental fixes insufficient.

Read original

Database Design Rust Redis Grpc+4

lineAug 1, 2025Original article

Replacing the Payment System DB Handling (opens in new tab)

The LINE Billing Platform successfully migrated its large-scale payment database from Nbase-T to Vitess to handle high-traffic global transactions. While initially exploring gRPC for its performance reputation, the team transitioned to the MySQL protocol to ensure stability and reduce CPU overhead within their Java-based environment. This implementation demonstrates how Vitess can manage complex sharding requirements while maintaining high availability through automated recovery tools. ### Protocol Selection and Implementation - The team initially attempted to use the gRPC protocol but encountered `http2: frame too large` errors and significant CPU overhead during performance testing. - Manual mapping of query results to Java objects proved cumbersome with the Vitess gRPC client, leading to a shift toward the more mature and recommended MySQL protocol. - Using the MySQL protocol allowed the team to leverage standard database drivers while benefiting from Vitess's routing capabilities via VTGate. ### Keyspace Architecture and Data Routing - The system utilizes a dual-keyspace strategy: a "Global Keyspace" for unsharded metadata and a "Service Keyspace" for sharded transaction data. - The Global Keyspace manages sharding keys using a "sequence" table type to ensure unique, auto-incrementing identifiers across the platform. - The Service Keyspace is partitioned into $N$ shards using a hash-based Vindex, which distributes coin balances and transaction history. - VTGate automatically routes queries to the correct shard by analyzing the sharding key in the `WHERE` clause or `INSERT` statement, minimizing cross-shard overhead. ### MySQL Compatibility and Transaction Logic - Vitess maintains `REPEATABLE READ` isolation for single-shard transactions, while multi-shard transactions default to `READ COMMITTED`. - Advanced features like Two-Phase Commit (2PC) are available for handling distributed transactions across multiple shards. - Query execution plans are analyzed using `VEXPLAIN` and `VTEXPLAIN`, often managed through the VTAdmin web interface for better visibility. - Certain limitations apply, such as temporary tables only being supported in unsharded keyspaces and specific unsupported SQL cases documented in the Vitess core. ### Automated Operations and Monitoring - The team employs VTOrc (based on Orchestrator) to automatically detect and repair database failures, such as unreachable primaries or replication stops. - Monitoring is centralized via Prometheus, which scrapes metrics from VTOrc, VTGate, and VTTablet components at dedicated ports (e.g., 16000). - Real-time alerts are routed through Slack and email, using `tablet_alias` to specifically identify which MySQL node or VTTablet is experiencing issues. - A web-based recovery dashboard provides a history of automated fixes, allowing operators to track the health of the cluster over time. For organizations migrating high-traffic legacy systems to a cloud-native sharding solution, prioritizing the MySQL protocol over gRPC is recommended for better compatibility with existing application frameworks and reduced operational complexity.

Database Design Grpc Mysql Database Migration+4

datadogJul 17, 20253 min readCurated summary

How we tracked down a Go 1.24 memory regression across hundreds of pods

Go 1.24 initially caused an unexpected ~20% increase in memory usage across several services, despite its Swiss Tables implementation being expected to reduce memory consumption. The increase appeared in system-level RSS metrics but not in Go’s runtime metrics or heap profiles. Investigation showed that a runtime allocator refactor likely caused more of the Go heap’s virtual memory to be committed to physical RAM. ## The Unexpected Go 1.24 Memory Increase - The issue emerged during an internal rollout of Go 1.24. - Multiple environments showed approximately 20% higher memory usage. - A staging bisect directly linked the increase to the Go 1.24 upgrade. - The behavior was surprising because Go 1.24’s headline Swiss Tables feature promised lower CPU and memory overhead. ## Ruling Out Swiss Tables and Mutex Changes - Swiss Tables were disabled with: ```bash GOEXPERIMENT=noswissmap ``` - Memory usage did not improve, ruling out the new map implementation as the cause. - The new spin-bit mutex implementation was disabled with: ```bash GOEXPERIMENT=nospinbitmutex ``` - The memory increase remained, eliminating this runtime change as the likely culprit. ## System Metrics vs. Go Runtime Metrics - Go runtime metrics showed almost no change after the upgrade. - System metrics reported a significant increase in resident set size (RSS). - RSS measures physical memory currently used in RAM, while Go’s runtime accounting primarily reflects allocated virtual memory. - This discrepancy matters operationally because systems such as Kubernetes and the Linux OOM Killer rely on physical-memory metrics. ## Examining the Go Heap with `/proc/[pid]/smaps` - Linux’s `/proc/[pid]/smaps` exposed memory usage for individual mappings. - In Go 1.24, the main Go heap mapping had roughly: - 1.28 GiB of virtual memory allocated - 1.26 GiB resident in physical RAM - In Go 1.23, a similarly sized heap mapping had about 300 MiB less RSS than its virtual size. - Other memory regions were not significantly affected, indicating that the increased RSS was isolated to the Go heap. - Upstream changes to label Go-allocated memory regions should make future `maps` and `smaps` investigations easier. ## The Suspected Allocator Regression - The evidence suggested Go 1.24 was not requesting substantially more virtual memory. - Instead, previously uncommitted virtual memory was being committed to physical RAM, increasing RSS without changing Go’s internal memory totals. - A major refactoring of the runtime’s `mallocgc` function stood out in the Go 1.24 changelog. - The investigation therefore focused on this allocator change as the likely source of the regression. Go 1.24’s memory increase was caused not by Swiss Tables or mutex changes, but likely by altered heap allocation behavior in the runtime. Comparing RSS with Go’s runtime metrics—and inspecting `/proc/[pid]/smaps`—was essential for identifying the allocator-related discrepancy.

Read original

Database Design Kubernetes Go Linux+4

datadogJul 17, 20253 min readCurated summary

How Go 1.24's Swiss Tables saved us hundreds of gigabytes

Go 1.24 initially caused a Go runtime regression that increased RSS across Datadog services, but some high-traffic workloads ultimately used substantially less memory. The reduction came from Go 1.24’s Swiss Tables map implementation, which made a large, mostly read-only routing cache more compact. Profiling also revealed opportunities to reduce memory further by removing redundant data from the cached values. ## The Unexpected Memory Reduction - Datadog observed roughly **500 MiB less live heap** in the `shardRoutingCache` map after upgrading to Go 1.24. - With `GOGC=100`, that translated to approximately **1 GiB less total memory usage**. - Even after accounting for an expected **400 MiB RSS increase** from the `mallocgc` regression, the service achieved a net reduction of about **600 MiB**. - The improvement was most visible in high-traffic environments because they contained larger routing caches. ## The `shardRoutingCache` Data Structure - The cache maps routing keys to shard information: ```go map[string]Response ``` - Each `Response` contains: - `ShardID int32` - `ShardType` - `RoutingKey string` - `LastModified *time.Time` - The map is populated mainly during service startup by querying a database. - It is rarely modified afterward, making its memory layout and initial allocation particularly important. - The routing key is stored both as the map key and again inside the value, creating potential redundancy. ## Estimating Memory per Entry - On a 64-bit system, a map key’s string header occupies **16 bytes**. - The value requires approximately: - 4 bytes for `ShardID` - 8 bytes for `ShardType` - 16 bytes for the `RoutingKey` string header - 8 bytes for the `LastModified` pointer - The value totals 36 bytes before alignment, or roughly **40 bytes with padding**. - Including the key header, each key-value pair requires about **56 bytes**, excluding the separately allocated string and `time.Time` data. ## Go 1.23 Bucket-Based Maps - Go 1.23 maps used hash tables organized into an array of buckets. - The number of buckets was always a power of two, and each bucket contained **eight slots**. - Reads and writes required scanning the slots in the selected bucket to find a matching key or an empty position. - When a bucket filled, Go added linked overflow buckets, which increased memory usage and made lookups more expensive. - Map growth occurred when the average load factor exceeded **13/16, or 6.5 of 8 slots**. - The map then allocated twice as many buckets. - To avoid a large latency spike, growth was incremental: old and new bucket arrays coexisted while entries were gradually moved during subsequent writes. ## Why Workload Shape Matters - The routing cache is populated in a startup-heavy phase and then primarily read. - Such a workload benefits from a compact map representation because it does not need frequent insertions or growth. - Differences in cache size and traffic patterns explain why the memory improvement was significant in some environments but not uniform across the fleet. Go 1.24’s Swiss Tables implementation can substantially reduce memory usage for large, stable maps, even when another runtime change causes RSS growth. Teams should profile real production heaps after Go upgrades and inspect large structs for duplicated strings, unnecessary pointers, and other avoidable per-entry overhead.

Read original

Database Design Go Garbage Collection Memory Optimization+4

datadogJun 24, 20253 min readCurated summary

How we built a real-time, client-side noise suppression library without server dependencies

Datadog’s CoScreen team needed high-quality noise suppression that could run in real time on client devices and integrate with WebRTC. Since existing solutions were either too slow, server-dependent, expensive, or difficult to embed, they built and open-sourced **dtln-rs**, a portable Rust library based on the DTLN model. It processes one second of audio in about 33 ms on an M1 MacBook Pro and supports WebAssembly, Node.js, and native clients. ## Introducing dtln-rs - dtln-rs is a lightweight, open-source noise reduction library based on the Dual-Signal Transformation LSTM Network (DTLN). - It can produce: - A WebAssembly module - A native Rust library - A Node.js native module - The library is designed to integrate with WebRTC-based applications. - Datadog also released a demo showing how to embed the filter in an application or webpage. ## Demonstrating Real-World Noise Suppression - The project was motivated by common remote-work disruptions, including lawn mowers and other background noise. - In one test, the filter removed a neighbor’s lawn mower so effectively that a colleague could not tell it was running. - The team used this result as evidence that the embedded library could provide meaningful value to CoScreen users. ## How DTLN Enables Real-Time Processing - AI noise suppression learns to distinguish desired speech from unwanted background sounds. - DTLN uses a short-time Fourier transform (STFT) to divide audio into smaller segments and analyze the magnitude of different frequencies. - It also uses phase information, which describes the starting position of each frequency in the sound wave. - A model analyzes magnitude and phase data to determine which parts are speech and which are noise. - Its LSTM-based architecture can adapt to different environments, such as: - Air-conditioner hum - Cafe conversations - Paper rustling - The combination of deep learning and efficient signal processing allows DTLN to operate with near-instantaneous latency. ## Why Existing Noise Suppression Solutions Were Insufficient - Many advanced machine-learning models require powerful backend servers, with processed audio sent back over the network. - This approach adds latency, infrastructure complexity, and operating costs. - WebRTC remains widely adopted but generally relies on older, built-in noise reduction techniques. - Earlier solutions such as RNNoise can reduce noise but often do not match the quality of newer commercial systems. - Although Web Audio and WebAssembly make custom client-side processing possible, implementation still requires substantial engineering effort. - Large companies can deploy specialized servers and models trained on enormous speech datasets, but smaller teams may not have the resources to do so. - CoScreen’s search for an alternative led to DTLN, which could run in real time on standard hardware and be embedded directly into client applications. ## Practical Recommendation For WebRTC applications needing client-side, real-time noise suppression, dtln-rs offers a portable alternative to expensive server-based services. Its Rust foundation and support for WebAssembly, Node.js, and native targets make it suitable for web, desktop, and embedded clients.

Read original

Database Design Rust Deep Learning Webassembly+4

datadogJun 23, 20253 min readCurated summary

How we built reliable log delivery to thousands of unpredictable endpoints

Datadog’s Log Forwarding system resembles a package delivery network: it must move large volumes of data efficiently and reliably to many unpredictable destinations. Kafka provides ordered transport, but its FIFO behavior creates difficult tradeoffs when endpoints are slow or unavailable. The central challenge is preserving delivery guarantees without losing logs, creating duplicates, blocking unrelated destinations, or overwhelming customer infrastructure. ## What Log Forwarding Does - Datadog forwards processed, enriched logs as schemaless JSON records. - Destinations can include: - Elasticsearch - Splunk - Generic HTTP endpoints accepting JSON `POST` requests - The system must support thousands of tenants and external endpoints with widely varying reliability and performance. ## Kafka as the Distribution Network - Logs move through Datadog on Kafka topics, analogous to packages traveling on conveyor belts. - Each Kafka partition provides strict FIFO ordering: - Records are read in the order they were written. - Kafka offsets must be committed in that same order. - Logs for different destinations are spread across multiple partitions, so records for a single destination may need to be regrouped during delivery. - Assigning a dedicated Kafka partition to every destination would be simple conceptually but infeasible at scale. ## Reliability Challenges - External endpoints may be: - Temporarily unavailable - Slow or unstable - Unreachable for hours or days - The system must avoid: - Losing customer logs - Sending duplicate logs - Delaying all destinations because one endpoint is unhealthy - Excessive resource usage - Overwhelming or effectively DDoSing a customer endpoint - Sending one HTTP request per log would be inefficient, so logs should be buffered and delivered in batches, much like packages going to the same address. ## Kafka Ordering and Blocked Progress - Waiting for each forwarding request to succeed before reading more Kafka data protects against data loss but can halt progress. - Continuing to read and acknowledge Kafka records before successful delivery risks losing logs. - Because offsets must be committed in order, one unavailable destination can block later records in the same partition—even if those records belong to healthy destinations. - This makes coordination between Kafka consumption, retries, batching, and concurrent delivery especially complex in a multi-tenant system. ## Lessons from Log Archives - Datadog had prior experience with similar delivery problems in its Log Archives feature. - Archiving was easier because: - Cloud object storage endpoints are generally more reliable. - Archiving has lower latency requirements. - Those lessons helped the team anticipate reliability and ordering pitfalls in Log Forwarding. ## Dedicated Kafka Topics per Destination - A possible solution would be to assign one or more Kafka partitions to each destination. - This would isolate destinations so that one slow endpoint could not block others. - However, the approach would require an impractically large number of Kafka topics or partitions as the number of customers and destinations grows.

Read original

Database Design Datadog Apache Kafka Json+4

datadogJun 17, 20251 min readCurated summary

Breaking up a monolith: How we’re unwinding a shared database at scale | Datadog

The provided text does not contain the blog post itself. It mainly includes Datadog’s navigation menu and a promotional link announcing its Gartner Magic Quadrant recognition, so the article’s argument and technical conclusions cannot be reliably summarized. ## Content Present in the Extract - A promotional banner links to Datadog’s recognition as a **Leader in the Gartner Magic Quadrant for Observability Platforms**. - The page navigation lists Datadog offerings across: - Infrastructure and application monitoring - Logs, databases, and data observability - Security - Digital experience monitoring - Software delivery - Service management - AI capabilities - The URL suggests the intended article is **“Unwinding a Shared Database”**, but its body text is missing. ## Practical Conclusion Please provide the article’s actual text or a complete page extract for a meaningful technical summary.

Read original

Database Design Datadog Observability Microservices

datadogJun 17, 20253 min readCurated summary

How we scaled fast, reliable configuration distribution to thousands of workload containers

Datadog’s seemingly simple tenant-configuration CRUD system must propagate updates rapidly and reliably to thousands of containers processing millions of logs per second. Loading configuration on every log is too expensive, while periodic caching introduces stale data and delayed updates. Datadog initially used database-backed caches invalidated through Kafka, but growing scale exposed reliability and resilience problems tied to repeated workload access to the central database. ## The Challenge of Propagating Context Data - Datadog calls tenant-specific settings—such as log parsing rules, Sensitive Data Scanner settings, and storage quotas—“context data.” - Configuration changes are expected to take effect almost immediately, including in Live Tail. - The same context data may be consumed by thousands of containers handling traffic for many tenants. - Because configuration directly affects customer-data processing, propagation must be both low-latency and highly reliable. - The system must assume that failures can occur anywhere in a large distributed environment. ## Why On-Demand Fetching and Simple Caching Fail - Fetching configuration from a database for every incoming log would create an impractical read load. - Large tenants can generate hundreds of thousands of logs per second. - Each processing instance could require thousands of database reads per second. - Multiplying this across many instances would require extensive, highly performant database replicas. - Caching configuration in each workload container reduces reads but does not eliminate the scaling problem. - Many workload instances still cache data for a high number of tenants. - Increasing the cache interval reduces database load but delays configuration updates. - With periodic invalidation, the average propagation delay is roughly half the cache interval. ## Context Loading v1: Database-Backed Caches and Kafka Datadog’s first successful architecture kept tenant configuration in a central durable database while allowing workload containers to cache entries indefinitely. - A user changes a log-processing configuration. - The central context database stores the update. - Kafka publishes an invalidation message after the database write. - Every workload container receives the notification. - Each container reloads the affected tenant’s configuration from the database. - This minimized routine database reads while preserving low-latency updates. ## Why the Initial Architecture Needed Reconsideration - The design required every workload instance to reach the central context database whenever a configuration changed. - As Datadog added more workloads and containers, update-related database traffic grew substantially. - Internal game days and production incidents showed that problems affecting the context database could spread to downstream processing workloads. - Database failures could prevent configuration updates from propagating and potentially make it impossible for new workload containers to initialize their context. - These reliability concerns demonstrated that Kafka-based invalidation alone did not sufficiently isolate workload processing from context-database failures. Datadog’s experience shows that configuration propagation at large scale requires more than a durable database and cache invalidation. The system must also reduce dependency on the central database during updates and startup, while continuing to provide near-immediate, reliable propagation.

Read original

Database Design Datadog Observability Log Management+4

datadogJun 17, 20253 min readCurated summary

Breaking up a monolith: How we’re unwinding a shared database at scale

Datadog is moving away from a large shared relational database because its benefits eventually give way to coordination costs, schema fragility, noisy-neighbor problems, and scaling limits. Splitting the database is difficult and expensive, but platform investments in service development and managed Postgres can make independently owned databases practical. The key is to establish functional boundaries, provide safe cross-domain access, and automate migrations. ## Why Shared Databases Persist - Shared databases reduce operational overhead for small or fast-moving organizations. - A single database enables simple, low-latency joins across all data. - Workload isolation and access management often matter less when systems are small. - Because the cost of splitting a database is high, organizations commonly keep the shared model longer than they should. ## Signs It Is Time to Split the Database - Data grows beyond the capacity of one machine, or replication becomes too slow. - Noisy-neighbor effects make performance unpredictable. - Schema changes by one team unexpectedly affect others. - Security requirements such as access-control lists are difficult to enforce. - These issues create engineering costs, incidents, and degraded user experiences across teams. ## What Database Decomposition Requires - Identify functional ownership boundaries. - Build services for cross-domain queries where necessary. - Require consumers to use those services instead of querying another domain’s tables directly. - Provision new database instances. - Migrate data and traffic carefully from the shared database to the new instances. Datadog had previously split off large portions of its database into only a few separate databases. The experience showed that finding boundaries, enforcing them, and migrating without incidents is difficult and highly manual. ## Why Teams Resist Leaving Shared Infrastructure - Building a service may jeopardize existing product goals. - Operating a service can introduce significant maintenance and on-call work. - Cross-domain data access may be unclear or cause unacceptable latency or user impact. - Owning a database creates additional operational responsibility. - Migrations are often handcrafted, risky, and difficult to repeat. - Forcing the transition can cost more than tolerating the existing problems and create organizational resistance. ## Platform Investments That Enable Change Datadog addressed these obstacles through two major initiatives: - **Rapid:** An opinionated framework for building and operating API and gRPC services. - **OrgStore:** A managed platform for Postgres databases. Rapid reduces the cost of creating and maintaining services by providing shared configuration, common data-access patterns, and operational support. OrgStore reduces the burden of owning separate database instances. Together, these platforms make it more attractive for new projects to avoid the legacy shared database and allow existing domains to migrate incrementally. The broader lesson is that database decomposition becomes realistic when platform engineering makes service ownership, database operations, cross-domain access, and migrations safe enough to fit into normal product development.

Read original

Database Design Observability PostgreSQL Grpc+4

datadogJun 3, 20253 min readCurated summary

Detecting faulty deployments: Our journey from unlabeled data to supervised learning

Deployments are a major source of software incidents, making rapid detection of faulty releases essential. Datadog developed Automatic Faulty Deployment Detection to identify releases associated with significant, deployment-related increases in error rates, despite having no reliable labeled dataset. Their solution evolved into an iterative, unsupervised ensemble of statistical checks designed to balance precision, recall, and the diverse behavior of customer applications. ## Challenges in Detecting Faulty Deployments - No universal ground truth exists because teams define “faulty” differently depending on their applications. - Faulty deployments are rare, creating severe class imbalance: - Random manual labeling would produce few useful examples. - Even a low false-positive rate could result in poor precision. - Applications have widely varying traffic and error patterns: - Seasonal applications naturally experience periodic changes. - Low-traffic services need longer observation periods. - Frequent deployments can make it difficult to identify which release caused an incident. ## Defining a Faulty Deployment Datadog focused on deployments that caused a significant and sustained increase in error rate. The definition relied on three attributes: - **Impact** - The total number of errors must be meaningfully higher than the baseline. - The increase must be significantly worse than in previous versions. - **Temporal correlation** - The error increase should align with the introduction of the new version. - **Persistence** - The elevated error rate must continue over time rather than reflecting temporary deployment noise. ## Building an Iterative Detection Framework - The initial system applied simple statistical rules to the first 60 minutes after each deployment. - Manual annotation was used to estimate precision, but this required substantial effort and did not reveal recall. - Datadog created an iterative framework composed of checks for different deployment requirements. - Checks included: - Comparing error rates before and after deployment. - Comparing a release with previous versions. - Accounting for periodic traffic and errors. - Handling sparse traffic patterns. - The checks were combined into a unanimous-voting ensemble: a deployment was flagged only when every check classified it as faulty. - The process began with a high-recall model, then: - Manually reviewed predicted faults. - Analyzed false positives. - Added new checks and adjusted thresholds to improve precision and recall. - Incident data and version rollbacks provided additional signals for finding faulty deployments the model had missed. ## Balancing Detection Speed and Recall - The model used the first hour after deployment to gather enough data to determine whether increased errors were persistent. - Increasing the observation period can improve confidence but delays detection. - The framework became progressively more sophisticated, adapting to: - Periodic error and traffic patterns. - Sparse traffic. - Multiple concurrent application versions. The practical recommendation is to begin with simple, high-recall statistical rules, then iteratively improve them through targeted manual review, false-positive analysis, and additional operational signals such as incidents and rollbacks. This approach can support other anomaly-detection problems where labels are scarce, failures are rare, and application behavior varies significantly.

Read original

Database Design Application Performance Monitoring Incident Management Statistical Analysis+4