Go | Techlist.io

discordMar 4, 20253 min readCurated summary

Discord Patch Notes: February 3, 2025

Discord’s February 3, 2025 patch focuses on faster startup, more reliable infrastructure, improved notifications, and extensive usability fixes across desktop and mobile. Major changes include a roughly 20% improvement in median Android launch times, a Rust rewrite of the Media Proxy service that reduced aggregate latency by about 22%, and substantially improved push-notification delivery. The release also adds desktop DM Apps, redesigns Privacy & Safety settings, and addresses numerous platform-specific bugs. ## Performance and Reliability - Android app initialization improvements reduced launch times by approximately 20% on the median Android device. - Discord rewrote its Media Proxy service from Go to Rust and modernized its infrastructure: - Aggregate latency improved by about 22%. - Latency at the 90th percentile improved by more than 40%. - Service reliability also improved. - Push-notification reliability increased across server sizes: - Servers with fewer than 2,500 users now achieve a 99.98% delivery rate within 30 seconds. - Discord can notify approximately 20 million users on large servers such as Midjourney within 15 minutes of an `@everyone` mention. - Android 13 and earlier received fixes for layout failures caused by device rotation and incorrect screen-size calculations. ## New Features and Interface Updates - Desktop users can now launch Apps directly in DMs. - A quick-launch bar displays recently used Apps at the top of the DM list. - Privacy & Safety settings were redesigned with: - A new tab structure. - Better grouping and clearer descriptions. - Improvements intended to support future Trust settings. - macOS desktop notifications now use native operating-system APIs: - Message sounds respect Focus and Do Not Disturb modes. - Multiple notifications from the same channel can be grouped. - Desktop search now autocompletes `from:me` and `mentions:me`. - Linux now supports Soundshare and Wayland screen sharing, including video and audio support on Ubuntu 22 and later. - Mobile users can edit server invite links while inviting someone. - Push notifications now include user nicknames. ## General Bug Fixes - Fixed UI and layout problems involving: - Forum channel tag editing. - Overlapping channel-topic modals. - Incorrect friend-request layouts at narrow desktop widths. - Misaligned images, padding, borders, and animations across Shop, Quests, profiles, and reaction pickers. - Resolved Android issues involving duplicated channel-name letters, hidden confirmation dialogs, incorrect toast positioning, keyboard overlap, and navigation-bar padding. - Fixed mobile and iOS issues affecting: - Search-result counts and back-swipe navigation. - Text-entry scrolling during channel searches. - Excessive zooming. - Animated Emojis stopping after interaction. - Nitro Home back-swipe navigation. - Corrected problems with Privacy & Safety tabs being cropped or nonfunctional. - Improved reliability for the Claimed Quests interface, especially for users with many completed Quests. - Fixed profile customization issues related to Nitro Credit. - Corrected custom status duration behavior. - Resolved issues with mutual-friend and mutual-server views getting stuck loading. - Fixed Emoji picker server filters and enabled editing of Emoji forum tags whose associated Emoji had been deleted. - Corrected several Shop, tablet banner, profile-effect, and mobile DM layout issues. Overall, the patch combines measurable backend performance gains with a broad collection of cross-platform fixes. Users should see faster Android launches, more dependable media uploads and notifications, improved desktop and mobile navigation, and fewer layout and interface failures.

Read original

Go Android Rust Linux+3

datadogJun 28, 20243 min readCurated summary

Timeseries indexing at scale

Datadog’s metrics volume grew 30× from 2017 to 2022, while customers began running increasingly complex queries. This growth exposed limitations in the Timeseries Index service, whose original indexing approach became a performance and maintenance bottleneck. The post introduces Datadog’s metrics architecture and explains how its indexing strategy evolved to handle large-scale workloads more reliably. ## Metrics Platform Architecture - **Intake** - Datadog Agents send data points through a load balancer to metrics intake. - Each point contains a metric name, timestamp, numerical value, and optional tags. - Tags such as `env`, `host`, and `service` provide dimensions for filtering, aggregation, and comparison. - Data is written to Kafka, allowing multiple consumers to process it for storage, indexing, analysis, and archiving. - **Storage** - The short-term storage layer has two services: - The Timeseries Database stores tuples of `<timeseries_id, timestamp, float64>`. - The Timeseries Index stores `<timeseries_id, tags>` mappings. - The custom Timeseries Index database is built on RocksDB and supports filtering and grouping during queries. - **Query Processing** - The distributed query layer contacts index nodes, retrieves intermediate results from the timeseries database, and combines them. - Filters such as `env:prod AND service:event-consumer` restrict results to matching data points. - Grouping by tags, such as `service`, produces separate timeseries for each group. - Aggregators such as `avg` combine values within each group. ## Why Timeseries Indexing Matters - Indexes prevent queries from scanning every timeseries associated with a metric, much like database indexes avoid full table scans. - Poorly designed or insufficient indexes can make queries slow and consume excessive CPU and memory. - As Datadog’s data volume and query complexity increased, the indexing system became a critical scalability concern. ## Automatically Generated Indexes - The original system generated indexes from live query behavior. - Slow or resource-intensive queries were recorded in a query log and analyzed periodically. - Index selection considered: - Query frequency - Execution time - Number of input timeseries identifiers scanned - Number of output identifiers returned - Highly selective queries—with a high input-to-output ratio—received indexes. - Obsolete indexes that no longer received queries were removed. - These indexes acted as materialized views, replacing expensive scans with efficient key-value lookups. ## Original Indexing Service Design - The service was written in Go and used embedded SQLite and RocksDB databases. - SQLite stored metadata, including: - Index definitions - Query logs - Query counts and timestamps - Input and output cardinalities - Query durations - Index definitions were read frequently, updated rarely, and cached entirely in memory. - Query logs were bulk-written in the background, keeping them out of the ingestion and query paths. - SQLite’s SQL interface made the metadata easy to inspect and modify manually. - RocksDB handled the high-volume write workload required to index trillions of events per day. Datadog’s experience shows that indexing strategies that work at smaller scale can become bottlenecks as data volume and query sophistication grow. Effective timeseries systems therefore need adaptive indexing, careful separation of query and ingestion workloads, and storage technologies suited to extremely high write rates.

Read original

Go Database Design Datadog Apache Kafka+4

datadogJan 31, 2023Original article

Performance improvements in the Datadog Agent metrics pipeline | Datadog (opens in new tab)

Datadog engineers recently optimized the Datadog Agent's metric processing pipeline to achieve higher throughput and lower CPU overhead. By identifying that metric context generation—the process of creating unique keys for metrics—was a primary bottleneck, they implemented a series of algorithmic changes and Go runtime optimizations. These improvements allow the Agent to process significantly more metrics using the same computational resources. ### Identifying Bottlenecks via CPU Profiling * Developers utilized Go’s native profiling tools to capture CPU usage during high-volume metric ingestion via DogStatsD. * Flamegraph analysis revealed that the `addSample` and `trackContext` functions were the most CPU-intensive components of the pipeline. * The profiling data specifically pointed to tag sorting and deduplication as the underlying operations consuming the most processing time. ### The Challenges of Metric Context Generation * The Agent must generate a unique hash (context) for every metric received to address it within a hash table in RAM. * To ensure the same metric always generates the same key, the original algorithm required sorting all tags and ensuring their uniqueness. * The computational cost of sorting lists repeatedly for every incoming message created a performance ceiling for the entire metrics pipeline. ### Specialization and Runtime Optimization * **Algorithmic Specialization:** The team implemented specialized sorting logic that adjusts based on the number of tags, optimizing the "hot path" for the most common metric structures. * **Hashing Efficiency:** Micro-benchmarks identified Murmur3 as the most efficient hash implementation for balancing speed and collision resistance in this use case. * **Leveraging Go Runtime:** The team transitioned from 128-bit hashes to 64-bit metric contexts. This change allowed the Agent to utilize Go's internal `mapassign_fast64` and `mapaccess2_fast64` functions, which provide optimized map operations for 64-bit keys. ### Redesigning for Performance * The original design followed a rigid "hash metric name -> sort tags -> deduplicate tags -> iterative hash" workflow. * Recognizing that sorting was the primary architectural bottleneck, the team moved toward a new design intended to minimize or eliminate the overhead of traditional list sorting during context generation. To achieve similar performance gains in high-throughput Go applications, developers should profile their applications under realistic load and look for opportunities to leverage runtime-specific optimizations, such as using 64-bit map keys to trigger specialized compiler paths.

Go Database Design Cpu Profiling Datadog Agent+4

datadogJan 31, 20232 min readCurated summary

Performance improvements in the Datadog Agent metrics pipeline

The Datadog Agent needed to process more metrics without increasing CPU usage. Profiling showed that generating unique metric contexts—especially sorting and deduplicating tags—was a major bottleneck. Datadog improved throughput through specialized sorting paths, faster hashing, and a more efficient context-storage design. ## Identifying the Bottleneck - Datadog uses Go’s CPU and memory profiling tools to optimize the Agent’s metrics pipeline. - Profiles were captured while Agents processed large volumes of DogStatsD metrics, ensuring the results reflected real workload pressure. - Flamegraphs showed that `addSample` and `trackContext` consumed the most CPU. - Sorting-related functions, including `util.SortUniqInPlace` and `sort`, were significant contributors to that cost. ## How Metric Contexts Work - Each received metric is assigned a metric context that uniquely identifies it in an in-memory hash table. - The context must incorporate: - The metric name - Tags included in the DogStatsD message - Container-generated tags - The context is computed as a hash, so it must be fast while minimizing collisions. - Tags must be consistently ordered so the same metric always produces the same context. - The original implementation sorted tags and removed duplicates, making sorting a recurring CPU expense. ## Specialized Sorting - Performance varied according to the number of tags attached to a metric. - Datadog introduced specialized sorting paths based on tag count. - This allowed common cases to use more efficient algorithms while retaining correct ordering and deduplication. ## Faster Hashing and Map Access - Micro-benchmarks compared hash functions according to speed and uniqueness. - Murmur3 performed best for Datadog’s requirements. - Datadog also changed metric contexts from 128-bit to 64-bit hashes. - A 64-bit hash still provided sufficient collision resistance for the use case and enabled Go runtime optimizations: - `runtime.mapassign_fast64` - `runtime.mapaccess2_fast64` - These optimized map operations improved both context storage and metric sampling performance. ## Redesigning the Algorithm - Sorting served two purposes: producing an ordered tag list and helping deduplicate tags. - Because sorting was the largest bottleneck, Datadog began exploring a design that could address these responsibilities more efficiently rather than relying on a single general-purpose sort. The practical lesson is to profile under realistic load, optimize the hottest paths, and combine targeted specialization, benchmark-driven implementation choices, and data-structure redesign to increase throughput without adding CPU capacity.

Read original

Go Performance Optimization Cpu Profiling Datadog Agent+3

datadogFeb 28, 20223 min readCurated summary

Profiling improvements in Go 1.18

Go 1.18 introduced major profiling improvements alongside features such as generics and fuzzing. Its Linux CPU profiler became substantially more accurate on multicore systems by addressing dropped `SIGPROF` signals, and profiler labels received an important correctness fix. These changes strengthened Go’s ability to connect continuous profiling data with tracing systems such as Datadog. ## More Accurate Linux CPU Profiling - Earlier Go versions used `setitimer(2)` to request a `SIGPROF` signal every 10 ms of CPU time. - On busy multicore systems, Linux could generate multiple signals within a single kernel “jiffy” window, but standard POSIX signals do not queue. - As a result, many signals were dropped: - A service using 20 CPU cores might generate roughly 2,000 profiling signals per second. - Its Go profile could contain only about 240 samples per second. - Linux’s software clock, commonly operating at 250 Hz, could only measure CPU time in roughly 4 ms intervals. This caused signal bursts and undercounted CPU usage. - `setitimer(2)` also distributed process-directed signals unevenly across threads, creating additional profiling bias. ## Combining `timer_create` and `setitimer` - `timer_create(2)` provided more reliable per-thread signal accounting and avoided most of the signal-dropping and thread-bias problems. - Its drawback was that the profiler needed awareness of every thread, including threads created independently by cgo code. - The Go 1.18 fix combined both timer mechanisms: - The signal handler identifies the signal source. - Signals from inferior sources are discarded. - The implementation accounts for short-lived threads and cgo edge cases. - The work originated from investigations into Go issues GH 35057 and GH 14434 and was developed through collaboration between contributors and Go maintainers. ## Profiler Label Correctness - Profiler labels, also called pprof labels or tags, associate key/value metadata with goroutines. - Labels are inherited by child goroutines and appear in CPU and goroutine profiles, enabling profiles to be filtered by request, service, or trace metadata. - Testing at Datadog revealed that some stack samples were missing labels they should have carried. - The cause was a CPU profiler lookup using the wrong goroutine reference. - The fix changed the profiler to use `gp.m.curg`, the thread’s actual current goroutine, rather than relying on `gp`, which can differ in certain runtime situations. Go 1.18’s profiling changes made CPU measurements more trustworthy on Linux and improved the accuracy of metadata attached to profile samples. Together, they provided a stronger foundation for correlating Go profiling with distributed tracing.

Read original

Go Linux Cpu Profiling Profiling+4

datadogApr 16, 20182 min readCurated summary

Cgo and Python

Embedding Python in Go lets applications gradually migrate from Python, reuse existing libraries, and load scripts dynamically without recompiling. Datadog uses this approach in its Go-based Agent so checks can remain in Python while the core application moves to Go. The key is combining cgo with a Go-friendly wrapper around CPython’s C API. ## Why Embed Python in Go? - Supports incremental migration from an existing Python codebase. - Reuses mature Python libraries without reimplementing them in Go. - Enables runtime loading and execution of custom or updated Python scripts. - This dynamic extensibility is especially important for Datadog checks. ## Introducing cgo - CPython exposes a C API, while Go requires a Foreign Function Interface to call C code. - cgo provides that integration while preserving the normal `go build` workflow. - A C preamble placed immediately before `import "C"` can include headers and C code. - The pseudo-package `C` exposes C constants, functions, and types to Go. - `go build -x` reveals how cgo generates intermediate C and Go files, compiles them, and links the final binary. ## Initializing the CPython Interpreter - A Go program must initialize Python with `Py_Initialize()` before executing Python code. - It should shut down the interpreter with `Py_Finalize()` when finished. - `Py_GetVersion()` demonstrates retrieving Python information through the C API. - `#cgo` directives can use `pkg-config` to locate Python development headers and libraries, such as `python-2.7`. - The examples use Python 2, but the same approach largely applies to Python 3. ## Using a Go Wrapper - Direct cgo interaction is mostly boilerplate, so Datadog uses the `go-python` library. - The wrapper exposes operations such as: - `python.Initialize()` - `python.PyRun_SimpleString(...)` - `python.Finalize()` - This hides cgo details and makes embedded Python code look more idiomatic from Go. ## Importing and Calling Python Code - A Python module can be imported with `PyImport_ImportModule`. - Go retrieves a function using `GetAttrString`. - The function is invoked through the Python API, passing empty tuple and dictionary objects even when it accepts no arguments. - The Go code must check for failures when importing modules or locating functions. - A simple `foo.py` module containing a `hello()` function can therefore be loaded and executed from disk. Embedding CPython through cgo provides a practical bridge between Go and Python. A wrapper such as `go-python` makes the integration easier to maintain, while allowing applications like the Datadog Agent to combine a Go core with dynamically executed Python components.

Read original

Go Database Design Python Datadog Agent+4

datadogJul 11, 20162 min readCurated summary

Releasing czlib and zstd Go bindings

Datadog released Go bindings for two C compression libraries, czlib and zstd, to improve compression performance in production data pipelines. czlib addresses the standard library’s slower pure-Go zlib implementation, while zstd offers faster decompression, competitive compression, and modern features such as dictionaries and compression levels. The post concludes that performance depends heavily on payload size and interface, so applications should benchmark representative data. ## czlib: Faster zlib-Compatible Compression - czlib began as a fork of Vitess’s `cgzip` package. - Datadog modified it to use zlib wrapping instead of gzip headers, matching the format used by its primary data pipeline. - It provides: - Non-streaming compression and decompression - Streaming interfaces - Batch-oriented interfaces - The authors recommend benchmarking with realistic messages using: - `PAYLOAD=path_to_message go test -run=NONE -bench .` ## Benchmark Results - For a 2 KB plaintext message: - czlib compression: 44.42 MB/s - Streaming czlib compression: 34.11 MB/s - Standard `compress/zlib`: 9.27 MB/s - czlib decompression: 255.62 MB/s - Standard-library decompression: 66.72 MB/s - For a 1.7 MB message: - czlib and standard zlib had similar compression speeds, around 24 MB/s. - czlib decompression reached roughly 257 MB/s. - Standard-library decompression reached about 121 MB/s. - The results show that czlib’s advantage is especially significant for smaller messages and decompression workloads. ## zstd: A Modern Alternative - zstd, or Zstandard, was developed by Yann Collet, who also created lz4. - At the time of publication, its format had recently been finalized and version 1.0 was pending. - Compared with zlib at compression level 6, zstd: - Compresses slightly faster - Produces a slightly better compression ratio - Decompresses substantially faster - Its features include: - Streaming compression - Configurable compression levels - Precomputed dictionaries - Fixed-length batch compression similar to the lz4 interface ## Go Binding Design - The zstd binding intentionally mirrors the zlib API. - It is designed to be a functional drop-in replacement, aside from a few zstd functions that do not return errors. - Dictionary-building support is available in the upstream repositories. - The binding exposes both advanced streaming functionality and efficient batch compression. Use czlib when compatibility with zlib is required but the standard implementation is too slow; consider zstd for new systems that benefit from faster decompression and dictionary support. Always benchmark with data and interfaces representative of the target workload.

Read original

Go Database Design Benchmarking Compression Libraries+2