cloudflare·Jul 8, 2026·3 min readCurated summary

Introducing Meerkat- an experiment in global consensus

Cloudflare is building Meerkat, an experimental global consensus service for coordinating control-plane state across more than 330 data centers. It aims to provide linearizable reads and writes while remaining available despite machine failures, network degradation, and data-center outages. Meerkat uses QuePaxa rather than Raft because QuePaxa allows all replicas to write and does not halt progress while waiting for failure timeouts. ## The Challenge of Global Control-Plane State - Cloudflare services need to read and modify shared state from locations around the world. - Examples include: - Placement information for resources such as AI model instances. - Leadership information identifying which machine may write to a database. - The system must combine: - Strong consistency, so readers do not observe conflicting or stale state. - High availability, even when machines, links, queues, or data centers fail. - Wide-area networks are unpredictable, making replica synchronization difficult. ## Why Consensus Is Needed - Consensus algorithms allow machines to agree on a single ordered sequence of operations, such as key-value-store reads and writes. - A typical consensus system can continue safely as long as a majority of replicas remain alive and connected. - This provides a foundation for applications such as: - Transactional key-value stores. - Distributed leases and locks. - Database leadership management. ## Limitations of Raft in Wide-Area Networks - Raft depends on a leader, and only the leader can accept writes. - If the leader crashes or becomes unreachable, the system may become unavailable until a timeout triggers leader election. - Timeout configuration is especially difficult across global networks with unpredictable latency. - A single failed machine or degraded network link can therefore affect availability. - Cloudflare reports having experienced incidents caused by unavailable leaders in consensus-based systems. ## Strong Consistency and Linearizability - Consistency determines how concurrent reads and writes may be ordered or observed. - Weak consistency can allow writes to be reordered. - Stronger models may preserve write ordering while still allowing reads to observe different states. - Linearizability is the strongest model described: - Operations appear to occur in real-time order. - Every read after a completed write observes that write. - Linearizability lets developers reason about distributed state similarly to local memory on a single-threaded machine. - Meerkat’s planned key-value store also provides serializability, which Cloudflare says will be covered separately. ## Fault-Tolerance Requirements Meerkat is intended to remain available and correct under several classes of failure: - The system should support reads and writes from any data center when: - A majority of machines are alive and can communicate. - A client can reach a machine connected to that majority. - In a system of `2f + 1` machines, the design tolerates `f` faults. - Single-machine failures and individual network-link degradations should not interrupt availability. - The system must remain correct during: - Machine crashes and restarts. - Network failures and delays. - Data-center outages. - Up-to-date machines must never disagree about committed state. - Like Raft, Meerkat does not attempt to tolerate Byzantine faults or actively malicious participants. ## Introducing Meerkat and QuePaxa - Meerkat is being developed by Cloudflare Research as an internal, experimental consensus service. - It is powered by QuePaxa, a consensus algorithm published by EPFL researchers in 2023. - Unlike Raft: - Any replica can perform writes. - Progress does not stop because a timeout expires or a leader becomes unavailable. - Applications will be layered on Meerkat’s consensus log, initially focusing on small control-plane data. - The first use cases include database leadership and other coordination state. - Cloudflare describes this as the first planned industrial deployment of QuePaxa at global scale. - Meerkat will remain internal while it is still under development.

Read original(opens in new tab)

Fault ToleranceDistributed SystemsKey Value StoreQuepaxa+4

toss·Jun 19, 2026·5 min readCurated summary

Spark Connect on Kubernetes #1: Building a Robust Spark Connect

Toss Securities operates Spark Connect as a production service on Kubernetes so analysts and engineers can use Spark without complex setup. Spark Connect replaces per-application Drivers with long-running servers, making clients lighter and sessions faster, but it also introduces shared-failure and resource-contention problems. The post argues that production reliability requires both reducing server-wide failure triggers and distributing sessions across multiple replicas. ## How Classic Spark Works - Spark consists of: - A **Driver**, which plans jobs, schedules tasks, and collects results. - **Executors**, which perform the distributed computations. - In Classic Spark: - **Client mode** runs the Driver inside the client process. - **Cluster mode** launches the Driver in the cluster for each submitted application. - Both modes assume that one application has one Driver and one workload. - Clients also need Spark libraries, JVM support, and configuration. ## What Spark Connect Changes - Spark Connect turns the Driver into a pre-started, long-running server. - Clients send unresolved logical plans encoded with Protocol Buffers over gRPC. - The server handles analysis, optimization, scheduling, and execution. - Results are streamed back using Arrow. - This resembles a database accessed through JDBC. ### Benefits - **Thin clients:** Clients do not need the full Spark runtime or JVM. - **Language and platform independence:** Notebooks, BI tools, SQL clients, and different programming languages can use the same server. - **Fast session creation:** Sessions connect to an already-running server. - **Better client-failure tolerance:** A disconnected notebook does not necessarily terminate server-side work. ## Problems Created by Shared Long-Running Servers Spark’s internal design often assumes “one application equals one workload.” Sharing one application across many users breaks that assumption. ### A Shared Driver Becomes a Single Point of Failure - Multiple sessions share one `SparkContext` and Driver JVM. - A Driver failure terminates all sessions, jobs, and caches attached to it. - Spark’s global `spark.executor.maxNumFailures` counter can shut down the entire application after enough executor failures. - Because all sessions contribute to the same counter, one user’s unstable or memory-intensive query can terminate unrelated users’ workloads. - The counter is global, persists over time, and is separate from per-query task-level fault tolerance such as `spark.task.maxFailures`. ### Resource Contention and Scheduling Limits - `newSession()` isolates SQL state and namespaces, but not CPU, memory, or executors. - Heavy workloads can occupy all task slots and delay smaller queries. - FIFO scheduling favors earlier jobs, and Spark does not preempt tasks already using slots. - Fair Scheduler pools can influence task-slot ordering, but cannot provide true CPU or memory isolation. - Spark Connect does not automatically propagate `spark.scheduler.pool` to the server-side execution thread, causing queries to fall into the default pool unless the server explicitly assigns pools. - Actual resource isolation must therefore be implemented outside Spark’s task scheduler. ### Fixed Server Capacity - A server’s image, Driver and Executor resources, and Spark configuration are fixed when it starts. - Dynamic Resource Allocation can adjust executor counts, but cannot change the server’s basic specification. - Flexible scaling and team-level isolation require creating or replacing servers, which is addressed in a later part of the series. ## Reducing Server-Wide Failures Before adding replicas, Toss Securities reduces the chance that one bad query can kill the shared server. - Set `spark.executor.maxNumFailures` effectively high enough to disable the global shutdown mechanism. - Use `spark.executor.failuresValidityInterval` to periodically clear accumulated failure records. - Rely on query-scoped controls: - `spark.task.maxFailures` stops tasks that repeatedly fail due to OOMs or exceptions. - `spark.stage.maxConsecutiveAttempts` stops jobs whose stages repeatedly fail, such as from shuffle-fetch errors. - These limits must be tuned carefully: overly aggressive values can cause healthy queries to fail during temporary infrastructure problems. - With this approach, executor failures terminate the problematic query rather than the entire Spark Connect server. ## Protecting Driver Memory from Large Results - Spark Connect streams query results through the Driver, so a large `collect()` can threaten Driver memory. - `spark.driver.maxResultSize` aborts an action when accumulated task results exceed the configured limit. - The limit is checked before large executor-side results are fetched into Driver memory. - The default 1 GB value assumes a single workload; in a multi-session server, it should be reduced or tuned based on the number of concurrent queries. ## Replicating Spark Connect Servers - Configuration alone cannot prevent Driver OOMs, node failures, or other catastrophic events. - The stronger isolation boundary is a separate SparkContext. - Multiple identical Spark Connect replicas are deployed: - Each replica has its own Driver, SparkContext, and Executors. - A failure affects only the sessions assigned to that replica. - Other replicas can continue accepting sessions. - Replica-based deployment reduces the blast radius from the entire Spark Connect service to an individual server instance. ## Practical Recommendation For a multi-user Spark Connect service, disable global executor-failure shutdown, enforce query-level failure limits, protect Driver memory with `spark.driver.maxResultSize`, and use multiple replicas to contain unavoidable Driver or node failures. Scheduler pools can improve ordering, but they should not be treated as true resource isolation.

Read original(opens in new tab)

Fault ToleranceKubernetesApache SparkGrpc+4

meta·Jun 3, 2026·3 min readCurated summary

Lights Out, Systems On: Validating Instant Power Loss Readiness

Meta’s Instantaneous PowerLoss Storm is a disaster-readiness testing program designed to prepare data centers for sudden, zero-notice power loss. It extends existing fault-tolerance mechanisms across facilities, servers, storage, compute, and the Twine orchestrator, while addressing region-wide failures and autonomous recovery. Through incremental testing and carefully defined tradeoffs, Meta aims to make losing an entire region as manageable as losing a smaller fault domain. ## Defense-in-Depth for Instant Failures - Power-loss tolerance was built into the full data-center stack, including mechanical and electrical systems, server racks, storage, compute, and Twine. - Batteries and Power Loss Siren (PLS) preserve in-memory data when racks lose power. - Twine services use region-wide asynchronous unavailability events (UEs) to coordinate shutdown and recovery. - Existing mechanisms had been tested against smaller fault domains, but region-wide failures introduced new challenges involving scale, replica placement, and autonomous startup. ## Solving Region Bootstrap Problems - Restarting a region may require millions of services to start simultaneously and discover their dependencies. - Circular dependencies among Twine control-plane services—such as Scheduler, Allocator, Broker, and Zelos—could prevent the orchestrator from starting itself. - Belljar CI/CD tests continuously identify critical startup dependencies before deployment. - A Twine recovery kit, supported by Belljar and Twrko, provides a manual “jumpstart” mechanism for breaking unexpected dependency cycles. - Meta also encountered a “boomerang” problem in which UEs shut down the control-plane services responsible for generating and distributing those signals. - The simpler solution was to let control-plane services ignore power-related shutdown UEs, preventing orphaned services that could not be reaped or recovered. ## Balancing Reliability and Engineering Velocity - Absolute tolerance to instant power loss could require costly or overly complex infrastructure and might create false positives during normal operations. - Meta defined unacceptable impacts as: - Storage or database data loss - Permanent damage to data-center facilities - Sustained disruption beyond one region - The company accepted bounded risks such as transient service errors, limited rack failures, and temporary staleness in routing or region-availability information. - Issues were considered tolerable when they could be remediated after the incident within a reasonable mean time to respond (MTTR). ## Incremental Validation Through PowerLoss Storms - Because testing a full region carried significant risk, Meta validated readiness progressively: - Dependency tests in new and pre-production regions - Exercises in shadow regions that mirror production - Tests in small production regions - Full tests in large regions supporting storage, AI, and data-warehouse workloads - During a Storm, Meta injects a power-supply fault to immediately de-energize an entire region. - After a short, realistic MTTR, remedial drain actions isolate the region from global controllers and schedulers. - The tests avoid preemptive preparation so they accurately represent an unexpected power failure. - Repeated exercises train both systems and engineers to handle regional loss with the resilience normally expected from smaller fault domains. Meta’s approach is to expand disaster readiness gradually: define unacceptable consequences, build layered recovery mechanisms, test at increasing scale, and use each exercise to improve both architecture and operational practice.

Read original(opens in new tab)

Fault ToleranceCI/CDDistributed SystemsDisaster Recovery+3

aws·Dec 3, 2025Original article

Introducing checkpointless and elastic training on Amazon SageMaker HyperPod (opens in new tab)

Amazon SageMaker HyperPod has introduced checkpointless and elastic training features to accelerate AI model development by minimizing infrastructure-related downtime. These advancements replace traditional, slow checkpoint-restart cycles with peer-to-peer state recovery and enable training workloads to scale dynamically based on available compute capacity. By decoupling training progress from static hardware configurations, organizations can significantly reduce model time-to-market while maximizing cluster utilization. **Checkpointless Training and Rapid State Recovery** * Replaces the traditional five-stage recovery process—including job termination, network setup, and checkpoint retrieval—which can often take up to an hour on self-managed clusters. * Utilizes peer-to-peer state replication and in-process recovery to allow healthy nodes to restore the model state instantly without restarting the entire job. * Incorporates technical optimizations such as collective communications initialization and memory-mapped data loading to enable efficient data caching. * Reduces recovery downtime by over 80% based on internal studies of clusters with up to 2,000 GPUs, and was a core technology used in the development of Amazon Nova models. **Elastic Training and Automated Cluster Scaling** * Allows AI workloads to automatically expand to use idle cluster capacity as it becomes available and contract when resources are needed for higher-priority tasks. * Reduces the need for manual intervention, saving hours of engineering time previously spent reconfiguring training jobs to match fluctuating compute availability. * Optimizes total cost of ownership by ensuring that training momentum continues even as inference volumes peak and pull resources away from the training pool. * Orchestrates these transitions seamlessly through the HyperPod training operator, ensuring that model development is not disrupted by infrastructure changes. For teams managing large-scale AI workloads, adopting these features can reclaim significant development time and lower operational costs by preventing idle cluster periods. Organizations scaling to thousands of accelerators should prioritize checkpointless training to mitigate the impact of hardware faults and maintain continuous training momentum.

Fault ToleranceAIMachine LearningAWS+5

google·Jun 22, 2025Original article

A colorful quantum future (opens in new tab)

Google Quantum AI researchers have successfully implemented "color codes" for quantum error correction on the superconducting Willow chip, presenting a more efficient alternative to the standard surface code. This approach utilizes a unique triangular geometry to reduce the number of physical qubits required for a logical qubit while dramatically increasing the speed of logical operations. The results demonstrate that the system has crossed the performance threshold where increasing the code distance successfully suppresses logical error rates. ## Resource Efficiency through Triangular Geometry * Unlike the square-shaped surface code, the color code uses a hexagonal tiling arranged in a triangular patch to encode logical information. * This geometric configuration requires significantly fewer physical qubits to achieve the same "distance" (the number of physical errors needed to cause a logical error) compared to surface codes. * Experimental results comparing distance-3 and distance-5 color codes showed a 1.56× suppression in logical error rates at the higher distance, confirming the code's viability on current hardware. * While the color code requires more complex decoding algorithms and deeper physical circuits, recent advances in decoders like AlphaQubit have enabled the system to operate below the error correction threshold. ## Accelerating Logical Gates * Color codes allow for many single-qubit logical operations to be executed in a single step (transversal gates), whereas surface codes often require multiple error-correction cycles. * A logical Hadamard gate, for instance, can be executed in approximately 20ns using a color code, which is nearly 1,000 times faster than the same operation on a surface code. * Faster execution reduces the number of error-correction cycles an algorithm must endure, which indirectly lowers the physical qubit requirements for maintaining logical stability. * The research team verified these improvements through "logical randomized benchmarking," confirming high-fidelity execution of logical operations. ## Logical State Injection and Magic States * The researchers demonstrated a "state injection" technique, which is the process of preparing a physical qubit in a specific state and then expanding it into a protected logical state. * This process is essential for creating "magic states" (T-states), which are necessary for performing the arbitrary qubit rotations required for complex quantum algorithms. * By moving states from the physical to the logical level, the color code architecture provides a clear path toward executing the universal gate sets needed to outperform classical computers. While the color code currently exhibits a lower error suppression factor than the surface code, its advantages in hardware efficiency and gate speed suggest it may be the superior architecture for large-scale, fault-tolerant quantum computing as device hardware continues to improve.

Fault ToleranceSuperconducting QubitsQuantum Error CorrectionSurface Code+3

datadog·Apr 2, 2019·3 min readCurated summary

Building highly reliable data pipelines at Datadog

Datadog’s approach to reliable data pipelines focuses on delivering correct data on time, even when individual jobs fail. Reliability therefore requires fault tolerance, monitoring, and fast recovery rather than eliminating every failure. The company achieves this through isolated, short-lived clusters and pipelines designed to limit the impact of failures. ## Reliability Means Timely, Correct Results - A reliable pipeline is one that consistently produces correct outputs within the required time window. - Occasional crashes do not necessarily make a pipeline unreliable if automatic recovery still delivers the data on schedule. - Pipelines should be designed with the expectation that failures will eventually occur. - Monitoring must detect unexpected failures early, while operational processes should support rapid recovery. ## Architecture for Batch Pipelines - Datadog streams and analyzes live data in real time but uses batch pipelines for features such as optimized long-term storage. - Historical data is stored in object storage. - Cloud Hadoop/Spark services launch and configure processing clusters. - Luigi workers manage tasks and workflows, while Spark workers compile code and submit jobs. - Jobs can be launched through a web interface, command line, or scheduler. ## One Cluster per Pipeline Instead of placing all workloads on one large Hadoop cluster, Datadog gives each pipeline its own cluster. - **Isolation:** Jobs do not compete for resources or interfere with one another, simplifying monitoring and diagnosis. - **Workload-specific hardware:** Clusters can use CPU-optimized or memory-optimized instances depending on the job. - **Elastic scaling:** Clusters can be expanded to catch up with delays or handle growing data volumes without waiting for a shared cluster. - **Safer upgrades:** Hadoop and Spark versions can be upgraded gradually across separate clusters. - Clusters are typically short-lived, averaging about three hours, although dozens may run simultaneously. ## Using Spot Instances to Encourage Fault Tolerance - AWS spot instances can reduce infrastructure costs by as much as 80%, but their nodes may be terminated whenever capacity or demand changes. - Rather than avoiding this failure mode, Datadog designs pipelines to tolerate disappearing clusters. - Long-running jobs are risky because failures discard more work and make recovery slower. - Pipelines are split into smaller jobs: - **Vertically:** Separate transformations into multiple stages, persisting intermediate results in S3. - **Horizontally:** Partition input data so multiple jobs process different portions concurrently. ## Breaking Up the Rollup Pipeline - Datadog’s rollup pipeline generates aggregated time-series data for historical metrics queries. - A single job would take more than 14 hours, making failures costly and difficult to recover from. - The pipeline is divided into two stages: - Aggregate high-resolution data and checkpoint it to S3 as Parquet files. - Convert the intermediate data into a custom format optimized for queries. - As these jobs grew, they were partitioned further using Kafka’s partitioning scheme. - Kafka partitions are grouped into shards, allowing Datadog to: - Adjust how much data each job processes. - Run more or fewer jobs as needed. - Isolate unusually large or sensitive shards. - This decomposition adds overhead because launching jobs and checkpointing to S3 take extra time, but it substantially limits the work lost during failures. ## Practical Recommendation Design pipelines around failure rather than assuming uninterrupted execution. Use isolated, scalable clusters, short jobs, intermediate checkpoints, and partitioned processing so that failures affect only a small portion of the workload and recovery remains fast.

Read original(opens in new tab)

Fault ToleranceDatabase DesignPostgreSQLAWS+4