Scylladb | Techlist.io

discordMay 8, 20263 min readCurated summary

How Discord Automates ScyllaDB Clusters at Scale

Discord’s Persistence Infrastructure team replaced fragile, manually sequenced scripts with the Scylla Control Plane (SCP), a framework for safely automating large-scale database operations. The effort was driven by the difficulty of creating shadow clusters and managing hundreds of ScyllaDB nodes with a seven-person team. SCP emphasizes resumability, safety checks, configurable parallelism, and incremental development. ## The Scale of Discord’s Database Operations - Discord operates Elasticsearch, Postgres, and ScyllaDB infrastructure across dozens of clusters and hundreds of nodes. - ScyllaDB stores critical data, including messages, channels, servers, and much of Discord’s user data. - Routine work includes: - Rolling restarts after configuration changes - Cluster expansion as traffic grows - Operating-system upgrades without downtime - Creating test clusters for validating ScyllaDB releases - These operations require careful sequencing and continuous validation rather than simple, fire-and-forget automation. ## From Scripts to the Scylla Control Plane - Discord initially accumulated Python, Bash, and other scripts incrementally. - The scripts were useful but fragile and dependent on institutional knowledge. - As operational demands grew, Discord created the Scylla Control Plane, or SCP, to provide a more structured automation system. ## Shadow Clusters for Safer Upgrades - Shadow clusters are temporary, full replicas of production that receive the same reads and writes as live traffic. - They allow Discord to detect upgrade problems under realistic load before changing production. - Building one manually requires: - Provisioning and configuring nodes - Joining nodes to the cluster - Validating replication - Establishing dual-write pipelines - Eventually tearing the environment down - Repeating this process across every ScyllaDB cluster made automation essential, especially for testing operating-system, hardware, and ScyllaDB version changes. ## Lessons from the Previous Automation Discord identified three major weaknesses in its old scripts: - **Unsafe:** Scripts could be run against the wrong nodes or in the wrong order, often without precondition checks. - **Unrecoverable:** A failure late in a multi-step process required restarting from the beginning. - **Difficult to extend:** New operations often required copying and modifying existing scripts instead of composing reusable components. SCP was designed around four goals: - Provide an extensible task framework that hides orchestration complexity. - Support configurable parallelism, including constraints such as avoiding simultaneous work in different availability zones. - Make safety the default through preconditions, retries, and persisted state. - Deliver functionality incrementally and refine it through real-world use. ## SCP’s Task-Based Architecture - SCP is organized around **tasks, workflows, and jobs**. - A task represents one unit of work, such as draining a node, checking repair status, or running cleanup. - **Node tasks** operate on individual nodes. - **Cluster tasks** coordinate operations across an entire cluster and may run node tasks across many nodes. - SCP also uses **conditions**, which pause execution until a required state is reached. - Conditions poll ScyllaDB APIs or Prometheus metrics. - They either succeed when the criterion is met or fail after a timeout. - For example, after restarting a node, SCP can wait for compactions to settle before continuing. - This avoids unreliable fixed-duration sleeps and reduces the risk of creating cascading pressure during rolling operations. ## Practical Recommendation For large-scale database operations, automation should be built as a reusable, stateful orchestration framework rather than a collection of scripts. Explicit preconditions, observable conditions, retries, controlled parallelism, and resumable state make complex infrastructure changes safer and more repeatable.

Read original

Scylladb PostgreSQL Rust Elasticsearch+3

datadogNov 2, 20203 min readCurated summary

Introducing Glommio, a thread-per-core crate for Rust and Linux

Thread-per-core architecture can significantly improve performance and reduce cloud costs by avoiding lock contention and expensive context switches. However, adopting it directly can reduce developer productivity because it requires new programming patterns and careful data ownership. Datadog developed Glommio, a Rust framework intended to make thread-per-core applications easier to build and maintain. ## Why Traditional Threading Has Limits - Applications commonly use multiple threads to perform independent tasks in parallel. - Shared data requires locks, which introduce contention and waiting. - Thread context switches can cost around five microseconds—potentially more than modern storage I/O operations using technologies such as `io_uring`. - Asynchronous programming reduces blocking, but many runtimes still rely on thread pools or separate worker threads for operations such as file I/O. ## How Thread-per-Core Works - Each CPU core runs a single application thread, often pinned to that core. - Because the operating system does not move the thread between cores, ordinary thread context switches are eliminated. - Hardware interrupts and auxiliary tasks can still interrupt execution. - For maximum performance, operators may reserve certain CPUs for interrupts and system services rather than application work. ## Sharding Data Across Cores - Thread-per-core applications depend on sharding: each thread owns a distinct subset of the data or requests. - Examples include assigning Kafka partitions or database key ranges to individual threads. - Requests assigned to one thread execute there to completion unless the code explicitly yields. - This ownership model prevents multiple threads from handling the same request or data simultaneously. ## Eliminating Locks - Since one thread processes a shard at a time, operations on that shard are naturally serialized. - A conventional threaded cache requires locks because multiple threads may update the same data concurrently. - Sharding reduces contention by dividing a large cache into smaller sections, but locks may still be needed if the operating system switches between threads. - With thread-per-core, updates to keys in the same shard occur sequentially, so an update can complete without acquiring a lock. ## Glommio and Existing Precedents - Thread-per-core is not a new concept; the author previously worked with Seastar, a C++ framework used by ScyllaDB. - Datadog’s Glommio brings the model to Rust while aiming to make its programming challenges more manageable. - The framework is motivated by the need to preserve developer productivity while achieving the efficiency gains of thread-per-core systems. Thread-per-core is most suitable for highly parallel, high-throughput workloads with naturally shardable data. Its performance benefits depend on disciplined data ownership and cooperative execution, while frameworks such as Glommio can reduce the complexity of adopting the model.

Read original

Scylladb Database Design Datadog Rust+4