techlist.io
KREN

Distributed Systems

34 posts

dropbox3 min readCurated summary

How our universal content processing platform Riviera evolved for AI and beyond

Riviera evolved from Dropbox’s preview-generation service into a shared content-processing platform used by products including Search, Replay, Sign, and Dash. Its core insight was to compose reusable transformations rather than build separate pipelines for every file type and output. As AI increased demand for consistent document extraction and preparation, Dropbox expanded Riviera’s capabilities and began offering them through APIs and Model Context Protocol tools. ## The Preview Problem - Dropbox supports more than 300 file formats, each requiring outputs such as: - Thumbnails - Full previews - Extracted text - Streaming manifests - Metadata - Building a separate service for every format and output would duplicate logic, dependencies, and operational work. - Configurations and package versions could drift across services, making the system harder to maintain and scale. ## Reusable Transformations as the Foundation - Riviera treats previews as sequences of smaller, reusable transformations. - For example, a PowerPoint preview can be produced by: - Converting the presentation to PDF - Rendering each PDF page as an image - The same PDF-to-image transformation can support PDFs and other workflows requiring page images. - This approach enables new formats and products to reuse existing capabilities instead of starting from scratch. ## Separating Coordination from Execution - Riviera uses a central coordinator to: - Collect and validate requests - Compose transformation workflows - Cache responses - Dispatch jobs to backend workers - Each worker handles a specific transformation, creating a clear unit for maintenance and scaling. - The platform now includes more than 100 capabilities and performs hundreds of thousands of transformations per second. - New formats and transformations can generally be added as plugins without changing the core system. ## From Internal Service to Shared Platform - Other Dropbox teams quickly adopted Riviera when they discovered overlapping content-processing needs. - Machine learning teams reused preview thumbnails for image normalization, avoiding duplicate generation. - Search used Riviera to prepare documents for indexing, while Sign, DocSend, and Replay reused existing transformations. - Dropbox eventually opened the plugin model to product teams, allowing them to add capabilities while the Riviera team maintained the platform’s core architecture. - Replay particularly benefited from Riviera’s complex video transcoding and manipulation capabilities, accelerating product development from months to weeks. ## Supporting AI Workloads - Dash introduced greater demand for reliable document preparation before AI processing. - AI systems require content to be transformed into consistent, machine-readable representations, including: - Extracted text - Data from scanned pages - File metadata - Normalized versions of hundreds of file types - These are fundamentally content-transformation challenges rather than AI-model challenges. - Because Riviera already supported many formats and transformations, Dash could build on existing infrastructure instead of creating a separate document-processing system. ## Broader Availability - Dropbox is making Riviera’s capabilities available to external developers and design partners. - Access is provided through APIs and Model Context Protocol tools. - The platform is intended for applications such as content management, document automation, search indexing, and AI document processing. Riviera’s evolution demonstrates the value of a shared transformation platform: reusable workers reduce duplication, centralized coordination improves reliability, and each new capability benefits multiple products. For teams building content-heavy or AI-powered applications, using standardized transformation infrastructure can be more efficient than maintaining format-specific pipelines independently.

Read original(opens in new tab)
Distributed SystemsAIModel Context ProtocolPlugin Architecture+3
aws3 min readCurated summary

Amazon SQS turns 20: Two decades of reliable messaging at scale | Amazon Web Services

Amazon SQS has spent two decades helping distributed systems communicate asynchronously without tightly coupling services. While its core purpose remains unchanged—decoupling producers and consumers, buffering traffic, and isolating failures—its scale, security, integrations, and workload support have expanded significantly. Recent improvements also make SQS suitable for high-throughput, multi-tenant, and AI-driven architectures. ## SQS’s Core Role in Distributed Systems - Producers place messages in queues and continue processing without waiting for consumers. - Consumers process messages when they are ready, preventing slow or unavailable services from causing cascading failures. - Customers use SQS to: - Decouple application components - Absorb traffic bursts - Improve resilience when individual services fail - Coordinate independent services and AI agents ## Higher Throughput for FIFO Queues - High-throughput FIFO mode launched in 2021 at up to 3,000 transactions per second per API action. - Capacity increased progressively to: - 6,000 TPS in 2022 - 9,000 TPS in 2023 - 18,000 TPS later in 2023 - Up to 70,000 TPS per API action in select Regions - The FIFO in-flight message limit grew from 20,000 to 120,000 in 2024, enabling more concurrent processing. ## Stronger Security and Access Controls - SSE-SQS launched in 2021, providing server-side encryption with AWS-managed keys and eliminating customer key-management requirements. - Encryption became the default for newly created queues in 2022. - Attribute-based access control was introduced in 2022, allowing permissions to be based on queue tags rather than static resource policies. ## Improved Message Recovery and Integration - Dead-letter queue redrive became available in the SQS console in 2021. - SDK and CLI APIs—including `StartMessageMoveTask`, `CancelMessageMoveTask`, and `ListMessageMoveTasks`—followed in 2023. - FIFO queue redrive support was added later that year. - JSON protocol support reduced processing latency by up to 23% for 5 KB payloads while lowering client CPU and memory use. - SQS queues can connect directly to EventBridge Pipes, enabling routing to many AWS services without custom integration code. ## Larger Messages and Fairer Queuing - The Extended Client Library for Python allows payloads up to 2 GB by storing message data in Amazon S3 and sending a reference through SQS. - In 2025, the native maximum message size increased from 256 KiB to 1 MiB for standard and FIFO queues. - Fair queues help prevent one tenant in a shared standard queue from delaying others. Producers provide a message group ID, while consumers require no changes. ## SQS for AI Workloads - SQS can buffer requests to large language models and regulate inference throughput. - Queues also help coordinate autonomous AI agents that operate as separate services. - These use cases apply the same established messaging model to more complex, distributed AI systems. Amazon SQS’s recommendation remains straightforward: use asynchronous queues when systems need loose coupling, burst management, and resilience. Its newer throughput, security, recovery, integration, and fairness features extend that pattern to larger and more demanding applications.

Read original(opens in new tab)
Distributed SystemsAWSAmazon S3Amazon Sqs+4
cloudflare3 min readCurated summary

Introducing Meerkat- an experiment in global consensus

Cloudflare is building Meerkat, an experimental global consensus service for coordinating control-plane state across more than 330 data centers. It aims to provide linearizable reads and writes while remaining available despite machine failures, network degradation, and data-center outages. Meerkat uses QuePaxa rather than Raft because QuePaxa allows all replicas to write and does not halt progress while waiting for failure timeouts. ## The Challenge of Global Control-Plane State - Cloudflare services need to read and modify shared state from locations around the world. - Examples include: - Placement information for resources such as AI model instances. - Leadership information identifying which machine may write to a database. - The system must combine: - Strong consistency, so readers do not observe conflicting or stale state. - High availability, even when machines, links, queues, or data centers fail. - Wide-area networks are unpredictable, making replica synchronization difficult. ## Why Consensus Is Needed - Consensus algorithms allow machines to agree on a single ordered sequence of operations, such as key-value-store reads and writes. - A typical consensus system can continue safely as long as a majority of replicas remain alive and connected. - This provides a foundation for applications such as: - Transactional key-value stores. - Distributed leases and locks. - Database leadership management. ## Limitations of Raft in Wide-Area Networks - Raft depends on a leader, and only the leader can accept writes. - If the leader crashes or becomes unreachable, the system may become unavailable until a timeout triggers leader election. - Timeout configuration is especially difficult across global networks with unpredictable latency. - A single failed machine or degraded network link can therefore affect availability. - Cloudflare reports having experienced incidents caused by unavailable leaders in consensus-based systems. ## Strong Consistency and Linearizability - Consistency determines how concurrent reads and writes may be ordered or observed. - Weak consistency can allow writes to be reordered. - Stronger models may preserve write ordering while still allowing reads to observe different states. - Linearizability is the strongest model described: - Operations appear to occur in real-time order. - Every read after a completed write observes that write. - Linearizability lets developers reason about distributed state similarly to local memory on a single-threaded machine. - Meerkat’s planned key-value store also provides serializability, which Cloudflare says will be covered separately. ## Fault-Tolerance Requirements Meerkat is intended to remain available and correct under several classes of failure: - The system should support reads and writes from any data center when: - A majority of machines are alive and can communicate. - A client can reach a machine connected to that majority. - In a system of `2f + 1` machines, the design tolerates `f` faults. - Single-machine failures and individual network-link degradations should not interrupt availability. - The system must remain correct during: - Machine crashes and restarts. - Network failures and delays. - Data-center outages. - Up-to-date machines must never disagree about committed state. - Like Raft, Meerkat does not attempt to tolerate Byzantine faults or actively malicious participants. ## Introducing Meerkat and QuePaxa - Meerkat is being developed by Cloudflare Research as an internal, experimental consensus service. - It is powered by QuePaxa, a consensus algorithm published by EPFL researchers in 2023. - Unlike Raft: - Any replica can perform writes. - Progress does not stop because a timeout expires or a leader becomes unavailable. - Applications will be layered on Meerkat’s consensus log, initially focusing on small control-plane data. - The first use cases include database leadership and other coordination state. - Cloudflare describes this as the first planned industrial deployment of QuePaxa at global scale. - Meerkat will remain internal while it is still under development.

Read original(opens in new tab)
Distributed SystemsFault ToleranceKey Value StoreQuepaxa+4
datadog3 min readCurated summary

How we measure data completeness at scale

Datadog built a real-time data-completeness system to ensure that every customer’s telemetry is available for dashboards, alerts, queries, and AI-driven decisions. Because ingestion spans hundreds of distributed paths and customers may send delayed or retried data, global or watermark-based tracking is unreliable. The system instead tracks payloads segment by segment, using idempotent create and acknowledgment events to identify losses and calculate end-to-end completeness. ## Defining Completeness at Datadog’s Scale - Completeness means every ingested payload—metrics, logs, spans, or other telemetry—is ultimately available to customers. - The system must measure completeness: - Across hundreds of services and ingestion paths - For each individual customer - In real time - With enough detail to identify where degradation occurred - Customer traffic may take different routes because of partitioning, isolation, and traffic patterns. - Metrics and APM pipelines can each involve hundreds or tens of distinct paths, creating a large number of possible failure points. - The completeness system must remain independent of the services it monitors so it can provide trustworthy diagnostics during incidents. ## Tracking Completeness by Pipeline Segment - Datadog considered watermark-based tracking, but delayed customer data, replayed traffic, and pipeline loops made predictable watermarks impractical. - Pipelines are divided into segments representing steps within or between services. - For example, intake-in to intake-out is one segment. - Intake-out to processing-in is another. - Each segment is measured independently, allowing engineers to locate degradation within a service or between services. - Segment-level tracking also adapts to pipelines whose branches appear or disappear over time. ## Counting Creates and Acknowledgments - When a payload enters a segment, the system records a create event. - When it exits, the system records an acknowledgment using the payload’s unique identifier. - Comparing creates with acknowledgments reveals whether payloads were lost in that segment. - Events are organized into time buckets based on when the payload first entered Datadog, using a Datadog-controlled timestamp rather than the customer’s clock. - Each identifier has a state per segment: - Created - Acknowledged - Acknowledged before the create event arrived - Duplicate create or acknowledgment events are ignored, making the system idempotent despite retries and event reordering. ## Calculating End-to-End Completeness - Segment completeness is the ratio of payloads exiting a segment to those entering it. - For sequential services, overall completeness is calculated by multiplying segment ratios. - Parallel branches require a different approach: - Treating branches as one pipeline would make completeness wait for the slowest branch. - Instead, Datadog uses a weighted average, giving each branch influence proportional to the volume it processes. - In the example, one branch reaches 94% completeness by multiplying 98% and 96% across two sequential services, while another branch reaches 100%. - Combining these branch measurements produces a more accurate view of currently available data without incorrectly marking all data incomplete because one branch is slower. ## Practical Conclusion Segment-level, identifier-based tracking gives Datadog a real-time and customer-specific view of data completeness. It both supports reliable end-to-end calculations and helps humans or automated systems quickly determine where ingestion problems are occurring.

Read original(opens in new tab)
Distributed SystemsObservabilityApplication Performance MonitoringApache Kafka+3
cloudflare3 min readCurated summary

Scaling Security Insights: how we achieved a 10x increase in global scanning capacity

Security Insights needed a 10x throughput increase to scan all customers more frequently and detect risks sooner. The existing system was overwhelmed by Kafka backlogs, slow processing, database inefficiencies, and API timeouts. Cloudflare improved capacity by introducing parallel and lane-based processing, optimizing bulk database writes, and addressing regional latency between its API and database. ## Scaling Kafka Processing - Scans are scheduled and published to Apache Kafka. - Go-based checker services consume these messages, inspect accounts, zones, and DNS records, and send findings to an internal API. - Kafka’s partition ordering limits each consumer group to one active consumer per partition. - Slow messages could block all subsequent messages in the same partition. - Adding partitions was avoided because it would increase resource usage for shared Kafka brokers. ## Introducing Parallel Processing - Checkers were changed to consume messages in batches. - Each message in a batch is processed concurrently in its own goroutine. - This increased throughput without requiring additional Kafka partitions. - The trade-offs were higher memory usage and potentially more work to repeat after a process crash. ## Separating Slow and Fast Work - Some scans took seconds or milliseconds, while unusually large accounts or zones could take minutes or hours. - These slow messages caused head-of-line blocking for faster work. - Consumer groups and checkers were split into: - A fast lane for predictable, short-running scans - A slow lane for messages expected to require substantially more time - Fast-lane consumers skipped slow messages, allowing normal scans to continue without delay. ## Optimizing Postgres Writes - The API originally executed one insert/upsert transaction per insight. - A request containing up to 500,000 insights could therefore generate hundreds of thousands of database round trips. - Bulk insertion with `COPY` into a temporary table was tested but caused bloat in Postgres system tables. - The final hybrid approach used: - `UNNEST` for smaller batches - `COPY` for batches above a configured threshold - This delivered millisecond-level performance for small writes and completion within seconds for very large writes. ## Diagnosing API Timeouts - Client-side timeouts increased as scan volume grew. - Checkers sometimes spent 20–90% of their processing time waiting on a single API call. - Throughput initially rose but then deteriorated under heavy load. - The root cause was network latency: - Postgres was hosted in Portland, Oregon. - The API ran active-active in Portland and Amsterdam. - Requests routed to Amsterdam incurred roughly 50 milliseconds of network round-trip latency. - Amsterdam database queries held client connection-pool connections much longer—nearly three seconds on average versus about 10 milliseconds in Portland. - The connection pool became exhausted, causing requests to wait for available connections and creating uneven Kafka lag across partitions. Cloudflare’s results came from improving the full processing pipeline rather than relying on a single infrastructure change. Parallelize message handling, isolate slow workloads, batch database writes, and place latency-sensitive services close to their databases to achieve large throughput gains and more frequent security scanning.

Read original(opens in new tab)
Distributed SystemsPostgreSQLApache KafkaAPI+4
google3 min readCurated summary

A low-carbon computing platform from your retired phones

Retired smartphones can become low-carbon cloud infrastructure by reusing their still-capable motherboards instead of manufacturing new servers. Researchers at UC San Diego, with Google’s support, are developing clusters of stripped-down Pixel phones managed by Kubernetes. Their planned 2,000-phone datacenter aims to provide affordable computing for education and research while reducing hardware-related emissions. ## The Carbon Case for Reusing Smartphones - Computing emissions come from: - **Operational carbon**, produced by electricity consumed during use. - **Embodied carbon**, produced during hardware manufacturing and raw-material extraction. - Reusing phones primarily addresses embodied carbon by extending the life of components that remain functional. - Since people typically replace phones every four years, many retired devices still contain capable processors, accelerators, memory, and storage. ## Smartphone Performance and Limitations - Modern smartphone performance cores can match or exceed the per-core performance of some data-center servers. - Smartphones have significant limitations compared with servers: - Fewer, heterogeneous processor cores. - Only 8–12 GB of memory. - Less capacity for large, multithreaded workloads. - The platform therefore targets workloads that fit on a phone or can be distributed across multiple devices. ## Converting Phones into Datacenter Hardware - Unmodified phones are unsuitable for datacenters because they include unnecessary and potentially hazardous components such as: - Displays and cameras. - Batteries not designed for sustained datacenter operation. - Consumer-oriented chassis and peripherals. - Researchers remove everything except the motherboard, which accounts for roughly 50% of a phone’s embodied carbon. - Android’s mobile userspace is replaced with a general-purpose Linux distribution. - This removes mobile-specific restrictions such as Android’s “low memory killer” and enables broader server-style programmability. - Kubernetes manages containerized applications across clusters of approximately 25–50 phones, equivalent to roughly one conventional server. ## Applications for Education and Research - Many university workloads—including Jupyter notebooks, grading systems, and research applications—require modest resources that a single smartphone can provide. - Early tests showed that a 20-phone cluster could handle peak grading demand for a class of more than 75 students while achieving latency below a typical AWS backend. - The planned 2,000-phone cluster could support around 100 comparable classes simultaneously. - The deployment would provide approximately 50 server-equivalents at substantially lower cost. ## Testing Computing at Scale - The project will evaluate whether consumer smartphone hardware can operate reliably under sustained datacenter workloads. - It will also serve as a large-scale testbed for distributed smartphone computing. - The system is expected to launch at UC San Diego in fall 2026. Repurposing retired phones offers a practical way to reduce demand for newly manufactured computing hardware, especially for lightweight academic and cloud workloads. The approach is most promising when applications can tolerate distributed resources and the reliability challenges of consumer-grade components.

Read original(opens in new tab)
Distributed SystemsKubernetesAndroidCloud Computing+4
meta3 min readCurated summary

Lights Out, Systems On: Validating Instant Power Loss Readiness

Meta’s Instantaneous PowerLoss Storm is a disaster-readiness testing program designed to prepare data centers for sudden, zero-notice power loss. It extends existing fault-tolerance mechanisms across facilities, servers, storage, compute, and the Twine orchestrator, while addressing region-wide failures and autonomous recovery. Through incremental testing and carefully defined tradeoffs, Meta aims to make losing an entire region as manageable as losing a smaller fault domain. ## Defense-in-Depth for Instant Failures - Power-loss tolerance was built into the full data-center stack, including mechanical and electrical systems, server racks, storage, compute, and Twine. - Batteries and Power Loss Siren (PLS) preserve in-memory data when racks lose power. - Twine services use region-wide asynchronous unavailability events (UEs) to coordinate shutdown and recovery. - Existing mechanisms had been tested against smaller fault domains, but region-wide failures introduced new challenges involving scale, replica placement, and autonomous startup. ## Solving Region Bootstrap Problems - Restarting a region may require millions of services to start simultaneously and discover their dependencies. - Circular dependencies among Twine control-plane services—such as Scheduler, Allocator, Broker, and Zelos—could prevent the orchestrator from starting itself. - Belljar CI/CD tests continuously identify critical startup dependencies before deployment. - A Twine recovery kit, supported by Belljar and Twrko, provides a manual “jumpstart” mechanism for breaking unexpected dependency cycles. - Meta also encountered a “boomerang” problem in which UEs shut down the control-plane services responsible for generating and distributing those signals. - The simpler solution was to let control-plane services ignore power-related shutdown UEs, preventing orphaned services that could not be reaped or recovered. ## Balancing Reliability and Engineering Velocity - Absolute tolerance to instant power loss could require costly or overly complex infrastructure and might create false positives during normal operations. - Meta defined unacceptable impacts as: - Storage or database data loss - Permanent damage to data-center facilities - Sustained disruption beyond one region - The company accepted bounded risks such as transient service errors, limited rack failures, and temporary staleness in routing or region-availability information. - Issues were considered tolerable when they could be remediated after the incident within a reasonable mean time to respond (MTTR). ## Incremental Validation Through PowerLoss Storms - Because testing a full region carried significant risk, Meta validated readiness progressively: - Dependency tests in new and pre-production regions - Exercises in shadow regions that mirror production - Tests in small production regions - Full tests in large regions supporting storage, AI, and data-warehouse workloads - During a Storm, Meta injects a power-supply fault to immediately de-energize an entire region. - After a short, realistic MTTR, remedial drain actions isolate the region from global controllers and schedulers. - The tests avoid preemptive preparation so they accurately represent an unexpected power failure. - Repeated exercises train both systems and engineers to handle regional loss with the resilience normally expected from smaller fault domains. Meta’s approach is to expand disaster readiness gradually: define unacceptable consequences, build layered recovery mechanisms, test at increasing scale, and use each exercise to improve both architecture and operational practice.

Read original(opens in new tab)
Distributed SystemsCI/CDFault ToleranceDisaster Recovery+3
netflix3 min readCurated summary

From Silos to Service Topology: Why Netflix Built a Real-Time Service Map

Netflix built Service Topology to give engineers a real-time, unified view of dependencies across its thousands of microservices. Traditional metrics, logs, and traces provide isolated signals but do not reveal the broader service relationships needed to diagnose failures or assess blast radius. The system combines multiple dependency sources into a living map that supports fast, context-rich troubleshooting. ## The Observability Problem - Netflix’s distributed architecture involves thousands of services and complex chains of calls for actions such as playback, authentication, recommendations, and optimization. - During incidents, engineers need to determine: - Which services depend on one another - What the potential blast radius is - Whether a failure originates locally or upstream - Existing observability tools show symptoms, logs, or individual request paths, but not the complete steady-state topology. - Manually combining information from different tools is slow and error-prone, especially during urgent incidents. ## Why Real-Time Service Mapping Matters - Frequent deployments and changing traffic patterns make static architecture diagrams quickly obsolete. - Netflix’s Live programming and advertising-supported plans increase the need for rapid diagnosis and operational awareness. - Engineers repeatedly asked about dependencies, failures, maintenance impact, unknown metrics, and recent call-path changes. - These recurring questions demonstrated the need for accurate, near-real-time dependency information. ## Lessons from Earlier Approaches - Netflix evaluated vendor platforms, graph databases, and internal prototypes before developing Service Topology. - Key lessons included: - Dependency data must update in near real time. - Storage and query systems must operate at Netflix’s scale. - The solution should integrate with existing observability workflows. - Incorrect or incomplete topology data can mislead engineers during incidents. - No single data source captures every aspect of service relationships. ## Requirements for a Living Map Service Topology was designed to provide: - Real-time updates as services deploy and dependencies change - Sub-second queries for traversing service call graphs - Both network-level and application-level views - Context such as health, availability tiers, ownership, and business domains - A visual interface for engineers and programmatic APIs for automation, resilience systems, and blast-radius analysis ## Combining Multiple Sources of Truth Netflix separates dependency information into physically distinct graphs so each layer can evolve and be queried independently. When a unified view is requested, the system traverses the layers in parallel and merges the results to maintain fast response times. ### eBPF Network Flows - eBPF captures network activity at the kernel level, recording which services communicate over the network. - This provides broad coverage, including services that lack application instrumentation. - It supports both cluster-level and application-level topology. - Its limitation is that network traffic alone does not provide application-specific context, such as the APIs or endpoints involved. Netflix’s approach is to combine complementary perspectives rather than rely on a single imperfect dependency source, producing a more complete and actionable service map.

Read original(opens in new tab)
Distributed SystemsObservabilityMicroservicesEbpf+2
gitlab4 min readCurated summary

GitLab Act 2

GitLab is restructuring its organization and strategy to prepare for an agent-driven software industry. It expects AI agents to dramatically increase software production, making scalable infrastructure, orchestration, context, and governance more important than traditional developer tooling. The company is reducing geographic footprint and management layers while reorganizing R&D around smaller, autonomous teams, reaffirming its FY27 guidance pending final restructuring costs. ## Organizational Restructuring - GitLab is conducting the process openly, including a voluntary separation window. - The new organizational shape is expected to be finalized by June 1 where possible; local legal processes may extend timelines. - Planned operational changes include: - Reducing the number of countries with small GitLab teams by up to 30%, while relying on partners in affected markets. - Removing up to three management layers in some functions. - Reorganizing R&D into approximately 60 smaller teams with end-to-end ownership. - Automating internal reviews, approvals, and handoffs with AI agents, then adjusting roles accordingly. - The restructuring and strategic shift are related but independently justified. - GitLab will disclose the restructuring’s final scope and financial impact during its June 2 earnings call. ## Software Development in the Agentic Era - Software will increasingly be produced by machines under human direction. - Agents will plan, code, review, deploy, and repair software. - Engineers will remain responsible for architecture, customer understanding, judgment, and difficult tradeoffs. - Lower software-production costs are expected to expand demand for software and increase the value of developer platforms. - Deep engineering skills—such as system design, distributed systems, failure analysis, and integrating new capabilities safely—will become more important and scarce. - GitLab points to its Duo Agent Platform, released in January, as an initial investment in this future. ## Infrastructure for Machine-Scale Development - Agents can create merge requests, trigger pipelines, and push commits at volumes far beyond human teams. - Git and existing development platforms were not designed for this level of activity. - GitLab plans to: - Reengineer Git for machine-scale workloads. - Replace parts of its monolithic architecture with API-first, composable services. - Provide agent-specific APIs so agents can interact as first-class platform users. - The company argues that reliability, performance, and scalability at this level will become a major source of platform value. ## Orchestration Across the Software Lifecycle - Enterprises need more than individual agents that generate code or open merge requests; they need software that reaches production and delivers business value. - GitLab’s orchestration layer is intended to coordinate agents across the lifecycle by: - Assigning work and managing state. - Passing context between tasks. - Resolving conflicts. - Enforcing policies and guardrails. - Keeping humans involved where judgment is required. - CI/CD is being reconsidered as part of this shift, with orchestration serving as the runtime for validating and safely deploying machine-rate changes. ## Context as a Competitive Advantage - Code generation capabilities are increasingly similar across developer-tool vendors. - GitLab believes its advantage lies in the connected context accumulated across planning, code, review, security, deployment, and operations. - It plans to make this data model a first-class, API-accessible service. - More contextual information should allow agents to use fewer tokens and produce better results. ## Governance Built Into the Platform - As agents perform more work, enterprises need strong control over identity, permissions, policies, auditing, and data location. - GitLab intends to make governance core infrastructure rather than an add-on product. - Every agent, pipeline, and merge request should operate through platform services that can: - Control who or what may perform an action. - Record what happened and why. - Protect sensitive code and data. - Support flexible deployment models. ## One Platform, Three Modes - GitLab notes that most business software cannot realistically be rewritten for the agentic era. - Its platform strategy is therefore intended to support existing codebases alongside newer development models. - The provided text ends before explaining the three modes in detail. GitLab’s overall recommendation to itself is to reshape both its organization and platform around machine-scale software development, while preserving human control over architecture, judgment, and governance.

Read original(opens in new tab)
Distributed SystemsAI AgentsCI/CDGitlab+4
discord3 min readCurated summary

You’ve Got (Too Much) Mail: Behind the Scenes of the 3/25/26 Voice Outage

Discord’s March 25, 2026 voice outage began when a Kubernetes configuration change abruptly terminated 17% of session processes. The resulting reconnection storm propagated through Discord’s realtime systems and overloaded voice-routing infrastructure, preventing many users from starting or joining calls. The incident exposed how failures in one distributed subsystem can create cascading load several services away. ## The Infrastructure Background - Discord is migrating stateful Elixir services to Kubernetes. - Each host runs thousands of in-memory processes for guilds, presence, messaging, and calls. - Deployments normally wait for a server’s entity count to reach zero before shutting it down, allowing processes to hand off their state safely. - The sessions service maintains one process for every connected device and carries websocket traffic, messages, presence updates, and other realtime events. - To reduce weekend CPU utilization, Discord planned to increase pod CPU and memory while proportionally reducing the number of pods. ## The Session Loss - The resource change was deployed to the first availability zone at 12:13 PDT. - Kubernetes terminated half of that zone’s pods because of the reduced replica count. - A safety check delayed process handoffs until other events completed, but the Kubernetes termination grace period expired first. - Because the service operated across three balanced zones, approximately 17% of Discord’s sessions stopped without a graceful handoff. - The outage lasted from 12:13 to 15:30 PDT, with users commonly seeing “Awaiting Endpoint.” ## How Elixir Monitoring Amplified the Failure - Discord relies heavily on Elixir `GenServer` processes, which process one mailbox message at a time. - Process monitors notify dependent processes whenever a monitored process exits. - The sudden loss of sessions therefore generated a large number of `{:DOWN, …}` notifications throughout the realtime infrastructure. - Guild and other processes stopped attempting to deliver updates to disconnected users, while the gateway began driving those users to reconnect. ## Reconnecting Users - The gateway handles websocket ingress and egress, creating sessions and maintaining client connections. - Session disconnections are normally expected and recoverable, whether caused by hardware, network problems, software bugs, or temporary connectivity loss. - When a session disappears, the gateway immediately instructs the client to reconnect. - It optimistically tries to resume the session through a gateway instance in the same zone, but the mass failure created a much larger reconnection surge than the system was designed to absorb. The incident demonstrates that reducing pod count can be dangerous in stateful distributed systems: an apparently routine capacity adjustment can cause abrupt process loss, trigger widespread retries, and overload unrelated downstream services. Changes to stateful workloads should be evaluated not only for steady-state resource usage but also for graceful shutdown behavior and synchronized failure scenarios.

Read original(opens in new tab)
Distributed SystemsKubernetesWebsocketsElixir+2
cloudflare4 min readCurated summary

500 Tbps of capacity: 16 years of scaling our global network

Cloudflare’s network has grown from a single transit provider in 2010 to more than 500 Tbps of provisioned external capacity across 330+ cities. The company argues that this scale is not merely about bandwidth: it enables security decisions, application execution, and routing validation to happen locally on every server. Its distributed architecture can absorb massive attacks automatically while supporting edge computing and emerging Internet protocols. ## From Transit Provider to Global Network - Cloudflare began with nLayer Communications as its first transit provider. - Expansion required city-by-city work: colocation contracts, fiber installation, hardware deployment, and Internet exchange peering. - In 2018, Cloudflare opened 31 cities in 24 days, despite logistical challenges such as customs delays and missing equipment. - The network now spans more than 330 cities and protects over 20% of the web. - The 500 Tbps figure represents provisioned interconnection capacity across transit, private peering, Internet exchanges, and Cloudflare Network Interconnect ports—not peak traffic. ## Turning the Network into a Security Layer - Cloudflare expanded from caching websites to securing employees and enterprise networks. - Its systems establish secure tunnels to private subnets and advertise customer IP space through BGP. - In 2025, Cloudflare mitigated a 31.4 Tbps DDoS attack lasting 35 seconds. - The attack was part of more than 5,000 attacks blocked that day, without paging an engineer. - Distributed automation allows attacks that once required nation-state resources to be handled in seconds. ## Packet-Level DDoS Mitigation - Incoming packets enter an XDP program chain in driver mode immediately after reaching the network interface card. - The `l4drop` eBPF program applies mitigation rules generated by `dosd`, Cloudflare’s denial-of-service daemon. - Each server identifies heavy traffic sources and shares the information across its colocation facility. - Mitigation rules spread globally through Quicksilver, Cloudflare’s distributed key-value store. - Only legitimate traffic reaches Unimog, the Layer 4 load balancer; Magic Transit traffic receives additional stateful inspection through `flowtrackd`. - The 31.4 Tbps attack was stopped at line rate without centralized scrubbing or human intervention. - Sufficient physical port capacity remains essential: software defenses cannot work if the network cannot first absorb the traffic. ## A Developer Platform at the Edge - Because Cloudflare already runs software on every server for packet filtering, it extended the same infrastructure to customer code through Workers. - Workers, KV, and Durable Objects run across Cloudflare’s global footprint rather than in a small number of cloud regions. - Workers Containers, introduced in 2025, support heavier workloads at the edge. - V8 isolates and custom filesystem layers reduce cold-start times. - Applications run on the same servers that discard malicious traffic before it reaches the network stack. ## Securing Routing with RPKI and ASPA - Cloudflare uses IPv6 and RPKI to reduce the risk of BGP hijacks. - It signs Route Origin Authorizations and rejects routes that fail Route Origin Validation, even when misconfigured networks become temporarily unreachable. - ASPA will extend protection by validating the network path, not just the organization authorized to originate a prefix. - The post compares RPKI to checking a destination passport and ASPA to verifying the entire flight manifest. - Cloudflare says 867,000 prefixes now have valid RPKI certificates, compared with nearly none a decade ago. - The company promotes early adoption of routing security standards because delays leave the Internet exposed to hijacks and route leaks. ## AI Agents and Internet Traffic - AI crawlers, training systems, and autonomous agents now generate more than 4% of HTML requests on Cloudflare’s network. - Human-initiated “user action” crawling increased more than 15-fold in 2025. - Unlike browsers, crawlers may retrieve every linked resource at maximum speed, making legitimate activity difficult to distinguish from attacks. - Cloudflare uses verified bot IP ranges, TLS fingerprints, behavioral analysis, and robots.txt signals to classify AI crawlers. - These signals help site owners decide which automated agents to permit. Cloudflare’s central lesson is that a global network must combine abundant capacity with intelligence distributed across every server. Its continued investment in automated mitigation, edge execution, routing security, and traffic classification is intended to make the Internet faster, safer, and more resilient as traffic patterns evolve.

Read original(opens in new tab)
Distributed SystemsCloudflareNetwork SecurityEbpf+4
figma3 min readCurated summary

Figma's Next-Generation Data Caching Platform | Figma Blog

Figma built FigCache to address scalability, reliability, and operational weaknesses in its Redis-based caching infrastructure. The stateless proxy provides a unified Redis data plane, decouples Redis connections from volatile client fleets, centralizes routing and security, and standardizes observability. After rollout to Figma’s main API in 2025, the caching layer reached six nines of uptime. ## Growing pains in caching - Redis evolved from a secondary component into a critical dependency for site availability. - Redis clusters were nearing connection limits as Figma’s infrastructure grew. - Rapid client-service scaling caused thundering herds of new connections, creating I/O bottlenecks and reducing availability. - Decentralized traffic management allowed applications to pollute or corrupt data across clusters. - Client libraries provided inconsistent observability, complicating incident diagnosis and mitigation. - A fragmented client ecosystem made it difficult to guarantee correct client-side behavior during failovers and topology changes. - Figma initially reduced Redis dependency in core API subsystems and created service-specific connection pooling, but pursued a broader platform redesign for long-term scalability. ## Design goals for a durable platform Figma defined several objectives for a caching platform capable of supporting future growth: - **Decouple Redis from client volatility:** Redis connection volume should not rise directly with elastic application fleets. - **Provide built-in observability:** Service owners and platform operators should receive consistent, granular visibility across workloads in a multitenant environment. - **Hide Redis Cluster complexity:** Clients should not need to manage topology changes such as scaling, failovers, or shard loss. - **Offer a universal endpoint:** Applications should access multiple Redis clusters through a centralized routing layer rather than managing separate endpoints and clients. - **Enable alternative backends:** New storage technologies, including durable systems, should be usable behind the same protocol and API. - **Remain extensible:** Cross-cutting capabilities such as encryption, guardrails, and traffic backpressure should be implemented centrally rather than repeatedly in applications. ## FigCache’s foundational architecture - Figma identified the need for a caching proxy that would serve as: - A unified Redis data plane. - An ingress layer for applications. - A connection multiplexer shielding Redis from client connection spikes. - A language-agnostic interface that hides cluster routing and management. - The platform was designed to centralize traffic decisions and abstract the underlying Redis topology from application developers. - FigCache is stateless and communicates using the Redis RESP wire protocol, allowing existing Redis-compatible clients and first-party libraries to use it. - Its broader platform role includes centralized security, routing, and end-to-end observability across the caching stack. ## Results - FigCache was rolled out to Figma’s main API service during the second half of 2025. - The caching layer subsequently achieved six nines of uptime. - The system established a foundation for more reliable, scalable, and interchangeable ephemeral storage across Figma. Figma’s approach demonstrates that Redis reliability at large scale requires more than larger clusters: a dedicated platform layer can isolate connection volatility, simplify client behavior, and centralize operational controls.

Read original(opens in new tab)
Distributed SystemsObservabilityRedisCaching+2
github2 min readCurated summary

Multi-agent workflows often fail. Here’s how to engineer ones that don’t.

Multi-agent workflows often fail because agents make implicit assumptions about state, ordering, and intended actions. The post argues that these systems should be engineered like distributed software rather than treated as chat interfaces. Typed schemas, explicit action definitions, and MCP-enforced interfaces make agent behavior more predictable and failures easier to contain. ## Typed Schemas Prevent Data Drift - Natural-language exchanges and inconsistent JSON lead to changing field names, mismatched types, and ambiguous payloads. - Typed interfaces define machine-checkable contracts, such as a `UserProfile` with fixed fields and allowed plan values. - Schema violations can fail fast, triggering retries, repairs, or escalation before invalid state spreads. - Debugging becomes contract-based instead of dependent on inspecting logs and guessing. ## Action Schemas Clarify Intent - Agents cannot reliably infer what “take action” means; they may assign, close, escalate, or do nothing. - Action schemas restrict outcomes to explicit, valid choices such as: - Requesting more information - Assigning an issue - Closing an issue as a duplicate - Taking no action - A discriminated union or similar structure ensures every agent returns one recognized action. - Invalid or ambiguous actions can be rejected, retried, or escalated. ## MCP Enforces Agent Interfaces - Schemas and action definitions are only conventions unless consistently enforced. - Model Context Protocol (MCP) provides explicit input and output schemas for tools and resources. - Calls are validated before execution, preventing agents from inventing fields, omitting required inputs, or drifting between interfaces. - MCP therefore acts as the enforcement layer for both data structure and intended behavior. Reliable multi-agent systems require explicit contracts at every boundary. Engineers should treat agents like code components: define their data and actions precisely, enforce interfaces with mechanisms such as MCP, and prevent invalid state from propagating.

Read original(opens in new tab)
Distributed SystemsAI AgentsModel Context ProtocolGithub Copilot+2
airbnb3 min readCurated summary

My Journey to Airbnb — Anna Sulkina

Anna Sulkina’s career journey moved from hardware diagnostics and frontend development into backend infrastructure and engineering leadership. Her experiences at Twitter taught her to design distributed systems for failure and to build consensus around transformative technologies like GraphQL. She joined Airbnb in 2022 because it aligned her passion for travel with an opportunity to strengthen developer infrastructure, organizational strategy, and engineering collaboration. ## Discovering Technology in Post-Soviet Ukraine - Sulkina grew up in Eastern Ukraine as the Soviet Union collapsed. - Her older brother introduced her to computers by bringing home hardware components and assembling a machine that loaded programs from a cassette player. - Seeing how individual components formed a working system inspired her to pursue technology. ## Learning English While Building Technical Skills - She studied programming at a Ukrainian university before immigrating to the United States. - Although she understood written English and knew how to program, communicating in English was initially more difficult than learning programming languages. - She took ESL classes while studying C++ and Java through Berkeley Extension. - Her first job was in hardware diagnostics at a five-person company. - A language barrier caused her to run out of time on a technical interview, but an interviewer familiar with her Berkeley class gave her another opportunity. - She eventually transitioned from C++ to Java, which became her primary language for many years. ## Moving Down the Stack and Into Leadership - Sulkina’s career progressed from hardware diagnostics to frontend, backend, and infrastructure engineering. - At the same time, she increasingly took on leadership responsibilities. - At Caymas Systems, her manager recognized her leadership potential and showed her the difference effective leadership makes. - At Comcast, she moved from individual contributor to engineering manager. - Coaching engineers, building software collaboratively, and developing high-performing teams convinced her that leadership was the right path. ## Lessons from Twitter’s Distributed Systems - During nearly nine years at Twitter, Sulkina advanced from first-line manager to director. - She worked through major operational events, including the “fail whale” period and the tweetstorm surrounding Ellen DeGeneres’s viral selfie. - Twitter’s transition from a monolith to microservices taught her that failure is inevitable in complex systems. - Resilient distributed systems must be designed to handle failures rather than assuming failures can be prevented. - Her cultural lesson involved turning promising ideas into adopted technologies. - She helped bootstrap Twitter’s GraphQL API, replacing legacy REST services. - The effort required leadership support, cross-team consensus, and stakeholder alignment, but ultimately improved product teams’ development velocity. ## Choosing Airbnb - Airbnb contacted Sulkina in 2022, when she felt ready to move beyond a well-established organization at Twitter. - The company appealed to her because it combined her professional interests with her personal passion for travel; she had been an Airbnb guest since 2013. - Airbnb’s Developer Platform organization had strong work happening in separate silos but needed clearer strategy, direction, and trust across engineering. - Sulkina began by clarifying the organization’s purpose and future direction. - Her early priorities included strengthening the organization, coaching leaders, and creating alignment within the team and with the teams it supported. - Over the following years, this work produced a high-performing organization with clearer strategy, stronger execution, and a focus on delivering business value. Sulkina’s story emphasizes that technical growth, organizational leadership, and personal motivation can reinforce one another. Her experience suggests that successful engineering leaders design for failure, invest in alignment, and use clear strategy to turn fragmented efforts into meaningful platform-wide impact.

Read original(opens in new tab)
Distributed SystemsMicroservicesJavaGraphQL+3
kakao4 min readCurated summary

In Search of Lost Reports: Kakao

KIMS, Kakao’s internal SMS platform, experienced rare cases where vendors sent delivery reports successfully, yet messages remained stuck in `SENT` instead of becoming `REPORTED`. The cause was a race condition: a fast vendor’s report arrived before the API server had committed the message record. The investigation showed that an unnecessarily long transaction—especially for paid messages with billing-event processing—delayed persistence and allowed valid reports to be dropped. ## KIMS Message Processing Flow - KIMS processes roughly one million SMS messages per day across multiple IDC environments and external vendors. - The normal flow is: - Route the request to a suitable vendor. - Call the vendor and record the message as `SENT`. - Deliver the message to the recipient. - Receive the vendor’s delivery report. - Update the message to `REPORTED`. - These stages run asynchronously across separate services, so their execution order is not guaranteed. ## Discovering the Missing Reports - Some messages remained in `SENT` even though Report Server logs confirmed that delivery reports had arrived. - The issue affected only about `0.02%` of messages, making it difficult to reproduce in tests or local environments. - Two patterns emerged: - Missing reports were concentrated among messages sent through one particular vendor. - Paid messages were affected more often than free messages. ## The Race Condition - The problematic vendor returned reports unusually quickly: - Other vendors typically took more than one second. - This vendor averaged around 20 ms. - Missing-report cases averaged only about 8 ms. - The API server performed additional processing before committing the message record. - For paid messages, billing-event publication was included in the same `@Transactional` scope, making the transaction longer. - Consequently, the sequence could become: 1. API Server calls the vendor. 2. API Server performs billing-related processing. 3. The vendor delivers the message and immediately sends a report. 4. Report Server receives the report before the message row exists in the database. 5. Report Server treats the report as invalid and drops it. 6. API Server finally commits the message as `SENT`. - The report was not lost at the network or vendor level; it was discarded because the system’s write path had not completed. ## Reducing Transaction Scope - The first fix was to remove nonessential work from the main transaction. - Billing-event publication was moved to asynchronous processing using `@Async` and `@TransactionalEventListener`. - The transaction was reduced to the essential state change and database commit. - This advanced the average commit point by approximately 10 ms and significantly reduced report omissions. - It also avoided a dual-write anti-pattern in which an external Kafka event was published inside a database transaction that could later roll back. ## Reconsidering the Need for a Transaction The incident prompted a broader review of whether the transaction was needed at all. - **Atomicity:** The transaction contained only one database write, with no multi-table or cross-record operation requiring all-or-nothing rollback. - **Read isolation:** Metadata such as vendor quality metrics was updated only every few minutes, and using a slightly stale value was acceptable. The independently read tables did not require a single consistent snapshot. - **Write isolation:** JPA’s dirty checking kept the status change in the persistence context until transaction completion, delaying the actual database write. This delay was precisely what allowed the report to arrive first. The article therefore presents the transaction itself—not the vendor or report receiver—as a source of unnecessary latency and an architectural anti-pattern in this workflow. ## Practical Recommendation Use transactions only when their guarantees are required. Keep critical persistence paths short, move external events and nonessential processing after commit, and critically evaluate whether delayed commit semantics could allow asynchronous consumers to observe a missing record.

Read original(opens in new tab)
Distributed SystemsApache KafkaMysqlRace Condition+4