techlist.io
KREN

garbage-collection

7 posts

dropbox

Improving storage efficiency in Magic Pocket, our immutable blob store (opens in new tab)

Magic Pocket’s immutable design protects data integrity but makes storage efficiency dependent on continuous reclamation. A new Live Coder service reduced write amplification while unintentionally creating severely under-filled volumes, driving fragmentation and storage overhead sharply upward. Dropbox responded by rethinking compaction, since its existing steady-state strategy was too slow to recover space from the resulting long tail of sparse volumes. ## The Cost of Immutability - Magic Pocket stores user files as immutable blobs distributed across its storage fleet. - Updates and deletions never modify data in place; obsolete blobs remain until compaction. - Garbage collection identifies unreferenced blobs, while compaction physically moves live blobs into new volumes and retires old ones. - Because closed volumes cannot be reopened, deleted data creates unused space unless it is actively consolidated. - Durability also increases storage requirements: - Replication stores multiple complete copies. - Erasure coding splits data into fragments and adds parity, providing fault tolerance with less overhead. - Fragmentation determines how efficiently that redundant capacity is used: - A volume with 50% live data effectively doubles required storage. - A volume with 10% live data uses roughly ten times the necessary space. ## The Live Coder Incident - A new on-the-fly erasure-coding service created severely under-filled volumes as it rolled out to new regions. - In the worst cases, less than 5% of a volume’s capacity contained live data. - Since volumes have fixed allocations, many mostly empty volumes consumed nearly as much raw capacity as full volumes. - Dropbox detected rising effective replication-factor signals, indicating more raw storage was being used per live byte. - The existing compaction system continued reclaiming space but was not designed for a long tail of extremely sparse volumes. - The incident demonstrated that compaction must adapt when the distribution of live data changes substantially. ## Steady-State L1 Compaction - Dropbox’s baseline strategy, L1, treats compaction as a packing problem. - It selects: - A highly filled host volume with available space. - Donor volumes whose live data fits into that space. - Live blobs from the donors are written into a new volume, eventually leaving the donors empty and removable. - L1 is simple, fast, and limits placement risk and metadata changes. - However, each run can read tens of GiB while typically producing only one densely packed volume. - Fewer than one complete volume is reclaimed on average because only donor volumes are fully drained. - This works well when volumes are already near full, but performs poorly when storage overhead is concentrated in many severely under-filled volumes.

garbage-collectiondatabase-designerasure-codingmagic-pocket+4
datadog

How we tracked down a Go 1.24 memory regression across hundreds of pods (opens in new tab)

Go 1.24 initially caused an unexpected ~20% increase in memory usage across several services, despite its Swiss Tables implementation being expected to reduce memory consumption. The increase appeared in system-level RSS metrics but not in Go’s runtime metrics or heap profiles. Investigation showed that a runtime allocator refactor likely caused more of the Go heap’s virtual memory to be committed to physical RAM. ## The Unexpected Go 1.24 Memory Increase - The issue emerged during an internal rollout of Go 1.24. - Multiple environments showed approximately 20% higher memory usage. - A staging bisect directly linked the increase to the Go 1.24 upgrade. - The behavior was surprising because Go 1.24’s headline Swiss Tables feature promised lower CPU and memory overhead. ## Ruling Out Swiss Tables and Mutex Changes - Swiss Tables were disabled with: ```bash GOEXPERIMENT=noswissmap ``` - Memory usage did not improve, ruling out the new map implementation as the cause. - The new spin-bit mutex implementation was disabled with: ```bash GOEXPERIMENT=nospinbitmutex ``` - The memory increase remained, eliminating this runtime change as the likely culprit. ## System Metrics vs. Go Runtime Metrics - Go runtime metrics showed almost no change after the upgrade. - System metrics reported a significant increase in resident set size (RSS). - RSS measures physical memory currently used in RAM, while Go’s runtime accounting primarily reflects allocated virtual memory. - This discrepancy matters operationally because systems such as Kubernetes and the Linux OOM Killer rely on physical-memory metrics. ## Examining the Go Heap with `/proc/[pid]/smaps` - Linux’s `/proc/[pid]/smaps` exposed memory usage for individual mappings. - In Go 1.24, the main Go heap mapping had roughly: - 1.28 GiB of virtual memory allocated - 1.26 GiB resident in physical RAM - In Go 1.23, a similarly sized heap mapping had about 300 MiB less RSS than its virtual size. - Other memory regions were not significantly affected, indicating that the increased RSS was isolated to the Go heap. - Upstream changes to label Go-allocated memory regions should make future `maps` and `smaps` investigations easier. ## The Suspected Allocator Regression - The evidence suggested Go 1.24 was not requesting substantially more virtual memory. - Instead, previously uncommitted virtual memory was being committed to physical RAM, increasing RSS without changing Go’s internal memory totals. - A major refactoring of the runtime’s `mallocgc` function stood out in the Go 1.24 changelog. - The investigation therefore focused on this allocator change as the likely source of the regression. Go 1.24’s memory increase was caused not by Swiss Tables or mutex changes, but likely by altered heap allocation behavior in the runtime. Comparing RSS with Go’s runtime metrics—and inspecting `/proc/[pid]/smaps`—was essential for identifying the allocator-related discrepancy.

garbage-collectiondatabase-designk8sgo+4
datadog

How Go 1.24's Swiss Tables saved us hundreds of gigabytes (opens in new tab)

Go 1.24 initially caused a Go runtime regression that increased RSS across Datadog services, but some high-traffic workloads ultimately used substantially less memory. The reduction came from Go 1.24’s Swiss Tables map implementation, which made a large, mostly read-only routing cache more compact. Profiling also revealed opportunities to reduce memory further by removing redundant data from the cached values. ## The Unexpected Memory Reduction - Datadog observed roughly **500 MiB less live heap** in the `shardRoutingCache` map after upgrading to Go 1.24. - With `GOGC=100`, that translated to approximately **1 GiB less total memory usage**. - Even after accounting for an expected **400 MiB RSS increase** from the `mallocgc` regression, the service achieved a net reduction of about **600 MiB**. - The improvement was most visible in high-traffic environments because they contained larger routing caches. ## The `shardRoutingCache` Data Structure - The cache maps routing keys to shard information: ```go map[string]Response ``` - Each `Response` contains: - `ShardID int32` - `ShardType` - `RoutingKey string` - `LastModified *time.Time` - The map is populated mainly during service startup by querying a database. - It is rarely modified afterward, making its memory layout and initial allocation particularly important. - The routing key is stored both as the map key and again inside the value, creating potential redundancy. ## Estimating Memory per Entry - On a 64-bit system, a map key’s string header occupies **16 bytes**. - The value requires approximately: - 4 bytes for `ShardID` - 8 bytes for `ShardType` - 16 bytes for the `RoutingKey` string header - 8 bytes for the `LastModified` pointer - The value totals 36 bytes before alignment, or roughly **40 bytes with padding**. - Including the key header, each key-value pair requires about **56 bytes**, excluding the separately allocated string and `time.Time` data. ## Go 1.23 Bucket-Based Maps - Go 1.23 maps used hash tables organized into an array of buckets. - The number of buckets was always a power of two, and each bucket contained **eight slots**. - Reads and writes required scanning the slots in the selected bucket to find a matching key or an empty position. - When a bucket filled, Go added linked overflow buckets, which increased memory usage and made lookups more expensive. - Map growth occurred when the average load factor exceeded **13/16, or 6.5 of 8 slots**. - The map then allocated twice as many buckets. - To avoid a large latency spike, growth was incremental: old and new bucket arrays coexisted while entries were gradually moved during subsequent writes. ## Why Workload Shape Matters - The routing cache is populated in a startup-heavy phase and then primarily read. - Such a workload benefits from a compact map representation because it does not need frequent insertions or growth. - Differences in cache size and traffic patterns explain why the memory improvement was significant in some environments but not uniform across the fleet. Go 1.24’s Swiss Tables implementation can substantially reduce memory usage for large, stable maps, even when another runtime change causes RSS growth. Teams should profile real production heaps after Go upgrades and inspect large structs for duplicated strings, unnecessary pointers, and other avoidable per-entry overhead.

garbage-collectiondatabase-designgomemory-optimization+4
datadog

.NET Continuous Profiler: Memory usage (opens in new tab)

Datadog’s .NET memory profiler helps identify excessive garbage collection, allocation hotspots, and objects that remain in memory after collection. It combines CLR events, operating-system thread metrics, sampled allocation data, stack traces, and weak handles to provide production-friendly memory insights. The approach favors low overhead, though some capabilities depend on the .NET version. ## Measuring Garbage Collector CPU Impact - The profiler uses CLR events to monitor garbage collection phases. - In server GC mode, the CLR creates two high-priority threads per heap/core to process collections in parallel. - Since .NET 5, these threads are named `.NET Server GC` and `.NET BGC`. - At each profile export, the profiler retrieves these threads’ CPU usage from the operating system. - It records the result as a sample with a native stack containing a `Garbage Collector` frame. - This uses a pull model: the exporter periodically requests the CPU measurement because no suitable event or dedicated profiler thread exists. - Before .NET 5, GC thread CPU usage could not reliably be identified because `GCCreateConcurrentThread` did not include thread IDs. ## Sampling Allocations - Per-allocation callbacks such as `ICorProfilerCallback::ObjectAllocated` provide detailed data but significantly slow allocation fast paths. - `GCSampledObjectAllocation` and `ObjectsAllocatedByClass` reduce some costs but do not provide call stacks for individual allocation sites. - Datadog instead listens to `AllocationTick`, emitted for roughly every 100 KB allocated. - Each event includes: - The object’s `ClassID` and type information. - The allocation address. - The object size and total allocation size since the previous tick. - The allocation kind: SOH (`0`), LOH (`1`), or POH (`2`). - Generic type names are reconstructed through the .NET profiling API. - Because allocation events are synchronous, the current thread is responsible for the allocation; the profiler walks that thread’s stack to capture the allocation call site. - This produces sampled allocation data for each heap category without imposing the cost of observing every allocation. ## Tracking Objects That Survive Garbage Collection - An allocation address alone cannot track an object indefinitely because compacting garbage collections can move objects. - Datadog uses weak handles, created through `GCHandle.Alloc`, which move with objects and do not keep them alive. - The profiler added this functionality through the .NET 7 `ICorProfilerInfo13` API and its `LiveObjectsProvider`. - For every sampled allocation, it creates a weak handle and records the object’s creation time. - After each garbage collection: - Handles for unreachable objects are removed and destroyed. - Handles for surviving objects remain and are included in the next profile. - This lets users inspect representative objects that persist after collection and investigate potential memory leaks. ## Practical Recommendation Use allocated-memory profiles to find endpoints and types responsible for excessive allocation, then examine surviving-object samples for retention or leak investigations. GC CPU data is especially useful for diagnosing applications whose high CPU usage is driven by frequent or expensive garbage collections.

garbage-collectiondatadogperformance-optimizationdotnet+4
datadog

.NET Continuous Profiler: Under the hood (opens in new tab)

Datadog’s .NET profiler is designed for continuous, low-overhead production monitoring rather than occasional diagnostic runs. It collects CPU, wall time, exceptions, lock contention, and allocation data, aggregates it into compact `.pprof` files, and links profiles to traces and services through runtime metadata. The post introduces the architecture and emphasizes preserving application performance as a central design requirement. ## What a Continuous Profiler Does - Profiling analyzes runtime performance and method call stacks. - It complements APM, which focuses on request latency, throughput, and errors. - The profiler also measures: - CPU usage - Wall time and method duration - Exceptions - Lock contention - Memory allocations and potential leaks - Unlike tools such as PerfView, dotTrace, dotMemory, and Visual Studio profilers, Datadog’s profiler is intended to run continuously in production with negligible overhead. - Continuous profiling avoids the need to recreate production traffic, security settings, hardware, and load in a separate environment. ## Datadog’s .NET Profiler Architecture - The profiler is composed of specialized profilers for different resource types. - Each profiler includes: - A sampler that collects raw data - A provider that exposes the collected samples - An aggregator combines samples from all profilers. - An exporter serializes the data into Google’s `.pprof` format and uploads it through the Datadog Agent. - Datadog’s backend processes the profiles for visualization and analysis. ## Sample Aggregation and Storage Each sample contains: - A call stack made up of method frames - Key-value labels, such as thread identifiers - A numeric value vector representing measurements like CPU consumption or wall time Samples with identical call stacks and labels are merged, and their numeric values are added together. This reduces duplication and produces smaller profile files—for example, repeated exceptions from the same code path and thread can be stored as one aggregated sample. The aggregation and `.pprof` serialization code is implemented in Rust and shared across Datadog’s Ruby, PHP, and other runtime profilers. ## Connecting Profiles to Traces and Services - Each uploaded profile includes process ID, host name, and runtime ID metadata. - The runtime ID uniquely identifies a .NET service running within a process. - This is important because a single .NET process can host multiple services, such as separate IIS applications running in different AppDomains. - The tracer communicates the mapping between runtime IDs, AppDomains, and service names. - Service names come from `DD_SERVICE`; if it is unset, the process name is used. - Datadog sends one profile per runtime ID every minute, so multiple profiles from one process may share a timestamp while representing different services. - Runtime IDs allow the backend to associate profiles with the correct traces and spans. ## Making .NET Call Stacks Easier to Read The .NET profiling API can expose compiler- and runtime-generated names that differ from the original source code. Datadog rewrites these frames to make visualized call stacks more understandable. - Constructors named `.ctor` are displayed using the class name. - Compiler-generated anonymous methods are rendered as the enclosing method followed by `_AnonymousMethod`. - Lambdas and local methods use an enclosing-method name with the `_Lambda` suffix. - Nested named methods such as `<DefiningMethodName>g__InnerMethodName|yyy_zzz` are displayed as `DefiningMethodName.InnerMethodName`. - Compiler-generated state-machine methods such as `MoveNext` are mapped back to the original source-level type and method names. ## Native and Managed Implementation Considerations - The team considered using Microsoft’s `TraceEvent` NuGet package to receive and parse CLR events in C#. - That approach would execute managed profiling code on the same CLR as the application being profiled. - Allocations made by the profiler could therefore increase garbage-collector pressure. - The post begins discussing how this performance concern influenced the implementation, but the provided excerpt ends before that design is explained. A production profiler must not only collect useful data but also minimize the memory and CPU costs of collecting it. Datadog’s architecture addresses this through specialized samplers, aggregation, compact serialization, runtime-aware trace association, and source-oriented call-stack cleanup.

garbage-collectiondatabase-designdatadogdotnet+4
figma

How Mozilla’s Rust dramatically improved our server-side performance | Figma Blog (opens in new tab)

Figma rewrote the performance-critical part of its multiplayer synchronization server from TypeScript to Rust to eliminate latency spikes and improve scalability. Rust’s low memory usage, lack of garbage collection, and high performance enabled Figma to isolate every document in its own process and make serialization more than ten times faster. However, Rust’s immaturity led Figma to abandon a full-server rewrite and use it selectively where performance mattered most. ## Scaling the Multiplayer Service - Figma’s server ran a fixed number of workers, with each document assigned exclusively to one worker. - The TypeScript server was single-threaded, so one slow operation could block synchronization for every document handled by that worker. - Encoding large documents was a frequent source of unpredictable delays. - Adding hardware or creating a Node.js process per document was impractical because of JavaScript VM memory overhead. - Figma temporarily isolated problematic “heavy” documents onto a separate worker pool, but this required manual monitoring and reassignment. ## Rust-Based Architecture - Figma moved performance-sensitive multiplayer logic into a separate Rust child process. - The Rust process communicates with its host through standard input and output. - Rust’s low memory usage made it feasible to run one process per document, fully parallelizing document operations. - Serialization became more than ten times faster, including for very large documents. - This architecture removed the worst-case blocking behavior of the original worker model. ## Server-Side Performance Improvements - Progressive rollout graphs showed a dramatic reduction in server performance problems after the Rust implementation reached full deployment. - The improvements primarily affected server stability and responsiveness, rather than directly making the client UI faster. - Users were less likely to experience synchronization hiccups caused by unusually large or expensive documents. - Figma reported substantial improvements in peak performance metrics compared with the old server. ## Benefits and Drawbacks of Rust - Rust provided: - Very low memory usage due to fine-grained memory control, no garbage collector, and a minimal standard library. - High performance comparable to lower-level languages such as C++. - Strong compile-time safety that prevents many classes of bugs common in C++. - The language was less mature than conventional server-side languages and still had significant rough edges. - Because of these limitations, Figma abandoned plans to rewrite the entire server in Rust. - Instead, it adopted Rust selectively for the most performance-sensitive components. Figma’s experience suggests that Rust can deliver major production benefits when applied to isolated, resource-intensive workloads, while a gradual or hybrid adoption strategy may be more practical than a complete rewrite.

garbage-collectiontypescriptrustnode.js+3