Horizontal Scalability

figmaMar 25, 20263 min readCurated summary

Figma's Next-Generation Data Caching Platform | Figma Blog

Figma built FigCache to address scalability, reliability, and operational weaknesses in its Redis-based caching infrastructure. The stateless proxy provides a unified Redis data plane, decouples Redis connections from volatile client fleets, centralizes routing and security, and standardizes observability. After rollout to Figma’s main API in 2025, the caching layer reached six nines of uptime. ## Growing pains in caching - Redis evolved from a secondary component into a critical dependency for site availability. - Redis clusters were nearing connection limits as Figma’s infrastructure grew. - Rapid client-service scaling caused thundering herds of new connections, creating I/O bottlenecks and reducing availability. - Decentralized traffic management allowed applications to pollute or corrupt data across clusters. - Client libraries provided inconsistent observability, complicating incident diagnosis and mitigation. - A fragmented client ecosystem made it difficult to guarantee correct client-side behavior during failovers and topology changes. - Figma initially reduced Redis dependency in core API subsystems and created service-specific connection pooling, but pursued a broader platform redesign for long-term scalability. ## Design goals for a durable platform Figma defined several objectives for a caching platform capable of supporting future growth: - **Decouple Redis from client volatility:** Redis connection volume should not rise directly with elastic application fleets. - **Provide built-in observability:** Service owners and platform operators should receive consistent, granular visibility across workloads in a multitenant environment. - **Hide Redis Cluster complexity:** Clients should not need to manage topology changes such as scaling, failovers, or shard loss. - **Offer a universal endpoint:** Applications should access multiple Redis clusters through a centralized routing layer rather than managing separate endpoints and clients. - **Enable alternative backends:** New storage technologies, including durable systems, should be usable behind the same protocol and API. - **Remain extensible:** Cross-cutting capabilities such as encryption, guardrails, and traffic backpressure should be implemented centrally rather than repeatedly in applications. ## FigCache’s foundational architecture - Figma identified the need for a caching proxy that would serve as: - A unified Redis data plane. - An ingress layer for applications. - A connection multiplexer shielding Redis from client connection spikes. - A language-agnostic interface that hides cluster routing and management. - The platform was designed to centralize traffic decisions and abstract the underlying Redis topology from application developers. - FigCache is stateless and communicates using the Redis RESP wire protocol, allowing existing Redis-compatible clients and first-party libraries to use it. - Its broader platform role includes centralized security, routing, and end-to-end observability across the caching stack. ## Results - FigCache was rolled out to Figma’s main API service during the second half of 2025. - The caching layer subsequently achieved six nines of uptime. - The system established a foundation for more reliable, scalable, and interchangeable ephemeral storage across Figma. Figma’s approach demonstrates that Redis reliability at large scale requires more than larger clusters: a dedicated platform layer can isolate connection volatility, simplify client behavior, and centralize operational controls.

Read original

Horizontal Scalability Observability Distributed Systems Redis+2

datadogNov 20, 20243 min readCurated summary

How we use formal modeling, lightweight simulations, and chaos testing to design reliable distributed systems

Courier’s design illustrates why distributed systems need more than unit, integration, and chaos testing. Datadog combined formal modeling, lightweight simulation, and conventional testing to uncover system-level risks before implementation. The approach was especially important after the March 8, 2023 outage, which showed how locally reasonable decisions can produce severe global failures. ## Why Distributed Systems Require Additional Analysis - Distributed systems provide greater scale and availability, but introduce concurrency, coordination, and failure modes that are difficult to reason about intuitively. - Traditional tests operate at relatively low levels of detail and may miss high-level design flaws. - Formal models and simulations allow teams to evaluate system behavior during the design phase, before implementation choices become expensive to change. - Model checking exhaustively explores all states permitted by a design and verifies defined correctness properties. ## Formal Modeling and Lightweight Simulation - Formal modeling uses a high-level specification language to describe: - System components and their interactions - Allowed system states - Properties the system must satisfy - Lightweight simulation builds a replica that runs under controlled conditions to study statistical characteristics such as: - Latency - Cost - Scalability - Behavior under realistic workloads - Modeling verifies correctness but cannot fully assess performance-related concerns. - Neither technique validates the final implementation directly. - Keeping models and simulations synchronized with the production design adds maintenance overhead. - Datadog considered the additional effort worthwhile because Courier was foundational, needed strong reliability guarantees, and incorporated lessons from the 2023 outage. ## Courier’s Requirements Courier was created to replace a decade-old Redis-backed queuing system that had begun to face throughput, scaling, and durability limitations. Its main requirements were: - **Multi-tenancy:** Isolate teams and products so one tenant cannot significantly disrupt others. - **At-least-once delivery:** Messages must not be lost; they must be delivered and acknowledged or sent to a dead-letter queue. - **Graceful degradation and high availability:** Throughput should decline roughly linearly as compute capacity is lost, rather than collapsing entirely. - **Horizontal scalability:** Throughput should increase linearly as compute capacity is added. The graceful-degradation requirement directly addressed the March 8 outage, when lost compute capacity caused a disproportionate throughput failure. ## FoundationDB Sharding for Tenant Isolation - Courier uses multiple FoundationDB clusters. - Each tenant is assigned to a subset of clusters. - No two tenants share the exact same cluster subset. - The initial design used: - Eight FoundationDB clusters - Four clusters per tenant - A theoretical maximum of `8 choose 4 = 70` tenant assignments - If one tenant saturated or disabled its four clusters, other tenants would still retain access to at least 25% of the total cluster capacity. - This arrangement provided sufficient isolation for the intended workloads. ## Broker Layer and High Availability - A broker layer exposes gRPC APIs for: - Sending messages - Receiving messages - Deleting messages - Clients connect only to the brokers, which apply the tenant-sharding logic. - Brokers health-check FoundationDB clusters and remove unhealthy clusters from consideration. - Both brokers and FoundationDB clusters are deployed across three availability zones to improve resilience. Courier demonstrates that formal verification and simulation are valuable complements to implementation testing. For mission-critical distributed services, teams should validate both correctness and operational behavior early, while also using unit, integration, and chaos testing to verify the final system.

Read original

Horizontal Scalability Distributed Systems Grpc High Availability+4