Write Ahead Log | Techlist.io

netflixSep 26, 2025Original article

Building a Resilient Data Platform with Write-Ahead Log at Netflix | by Netflix Technology Blog | Netflix TechBlog (opens in new tab)

Netflix has developed a distributed Write-Ahead Log (WAL) abstraction to address critical data challenges such as accidental corruption, system entropy, and the complexities of cross-region replication. By decoupling data mutation from immediate persistence and providing a unified API, this system ensures strong durability and eventual consistency across diverse storage engines. The WAL acts as a resilient buffer that powers high-leverage features like secondary indexing and delayed retry queues while maintaining the massive scale required for global operations. ### The Role of the WAL Abstraction * The system serves as a centralized mechanism to capture data changes and reliably deliver them to downstream consumers, mitigating the risk of data loss during administrative errors or database corruption. * It provides a simplified `WriteToLog` gRPC endpoint that abstracts underlying infrastructure, allowing developers to focus on data logic rather than the specifics of the storage layer. * By acting as a durable intermediary, it prevents permanent data loss during incidents where primary datastores fail or require schema changes that might otherwise lead to corruption. ### Flexible Personas and Namespaces * The architecture utilizes "namespaces" to define logical separation, allowing different services to configure specific storage backends like Kafka or SQS based on their needs. * The "Delayed Queues" persona leverages SQS to provide a scalable way to retry failed messages in real-time pipelines without sacrificing overall system throughput. * The system can be configured for "Cross-Region Replication," enabling high availability and disaster recovery for storage engines that do not natively support multi-region data transfer. ### Solving System Entropy and Consistency * The WAL addresses the "dual-write" problem, where updates to primary stores (such as Cassandra) and search indices (such as Elasticsearch) can diverge over time, leading to data inconsistency. * It facilitates reliable secondary indexing for NoSQL databases by managing updates to multiple partitions as a coordinated sequence of events. * The platform mitigates operational risks, such as Out-of-Memory (OOM) errors on Key-Value nodes caused by bulk deletes, by staging and throttling mutations through the log. Organizations operating at scale should adopt a WAL-centric architecture to simplify the management of heterogeneous data stores and enhance system resilience. By centralizing the mutation log, teams can implement complex features like Change Data Capture (CDC) and cross-region failover through a single, consistent interface rather than building bespoke solutions for every service.

Write Ahead Log Apache Kafka Distributed Systems Data Replication+4

figmaMay 16, 20242 min readCurated summary

Keeping It 100(x) With Real-time Data At Scale | Figma Blog

Figma’s LiveGraph powers real-time collaboration by subscribing to GraphQL-like queries and updating clients automatically. Rapid growth—tripled sessions since 2021 and fivefold view-request growth in one year—exposed limits in its single-server, mutation-based architecture. Figma launched “LiveGraph 100x,” a redesign focused on scaling reads and database updates while preserving performance and enabling a safe migration. ## LiveGraph’s Role in Figma - LiveGraph keeps data synchronized across collaborative features such as: - File editing - Comments - FigJam voting - It exposes a web API for subscribing to GraphQL-like queries. - Results are returned as JSON trees based on a schema of entities, relationships, and views. - A custom React Hook automatically re-renders interfaces when subscribed data changes. ## The 100x Scaling Initiative Figma’s growing user base increased both the number and cost of LiveGraph client sessions. At the same time, the underlying database evolved from one PostgreSQL instance into vertically and horizontally sharded infrastructure. The redesigned system needed to: - Preserve or improve service-level objectives for initial loads and updates. - Support more database shards reliably and efficiently. - Scale reads and database-update processing independently. - Allow incremental, transparent migrations without disrupting users. ## Limitations of the Original Architecture Originally, LiveGraph consisted of: - A single LiveGraph server. - An in-memory query cache. - One PostgreSQL instance. - A cache that tailed PostgreSQL’s logical replication stream. PostgreSQL writes row mutations to its write-ahead log, including pre- and post-row images and a monotonically increasing sequence number. LiveGraph used these mutations to update cached query results directly rather than recomputing them. This design worked well at smaller scale because: - All updates came from one primary database. - The replication stream provided a global ordering. - Each row mutation could be applied directly to the relevant cached results. ## Sharding Breaks Global Ordering As the original PostgreSQL instance reached capacity, Figma introduced vertical shards and began moving toward broader horizontal scaling. This invalidated the assumption that all database updates arrive in one globally ordered stream. - Multiple shards can generate updates simultaneously. - Their updates have no guaranteed global order. - LiveGraph therefore needed an architecture that could process distributed database changes while maintaining reliable, timely query updates. The growing load made it necessary to rethink LiveGraph fundamentally rather than continue extending its single-database design.

Read original

Write Ahead Log PostgreSQL Distributed Systems GraphQL+4

figmaOct 19, 20223 min readCurated summary

Making multiplayer more reliable | Figma Blog

Figma improved multiplayer reliability by adding a durable write-ahead journal alongside its existing checkpoint system. Instead of relying on full-file snapshots every 30–60 seconds, Figma now records incremental changes frequently, allowing crashed servers to recover nearly to the latest state and reducing deployment-related database spikes. The goal was to reduce potential data loss from up to 60 seconds to less than one second. ## How Figma’s Multiplayer System Worked - Browsers connect to Figma’s multiplayer service over WebSockets. - The service authoritatively handles: - Validation - Ordering - Conflict resolution - Broadcasting updates to connected clients - File state is held in memory for speed. - Every 30–60 seconds, the entire file is: - Encoded into a binary format - Compressed - Uploaded to Amazon S3 as a checkpoint ## Problems with Checkpoint-Only Persistence - A multiplayer crash could lose up to 60 seconds of server-side work. - Checkpoints become increasingly expensive as files grow in size and complexity. - Redeployments caused large write spikes: - All in-memory files had to be closed. - Each file needed a final checkpoint. - The resulting burst increased database load. ## Introducing the Journal - Figma added a durable transaction log, or journal, backed by DynamoDB. - Each accepted change receives an incrementing sequence number. - Checkpoints store the latest sequence number they include. - During recovery: - Multiplayer loads the latest checkpoint. - It queries the journal for entries with higher sequence numbers. - It replays those incremental changes to reconstruct the latest file state. - Journal entries are much smaller than full-file checkpoints, since they contain only user edits. - Figma writes journal data roughly every 0.5 seconds rather than waiting 60 seconds between full snapshots. - The target was less than one second of data loss in rare failure scenarios. ## Smoother Deployments - During deployment, Figma can close connections and wait for unsaved changes to reach the journal. - The 99th percentile persistence time is under one second. - Since journal writes happen continuously during normal operation, deployments no longer create a sudden checkpoint-writing surge. - Database write load is therefore steadier and more predictable. ## Datastore and Batching Decisions - Figma selected DynamoDB because the journal requires a horizontally scalable datastore with high write capacity. - Postgres was considered but rejected because the anticipated write volume exceeded Figma’s current horizontal-scaling approach for Postgres. - Clients send updates at approximately 30 frames per second, or every 33 milliseconds. - The journal does not need that granularity, so multiple changes can be batched before being persisted, improving performance. Figma’s approach combines inexpensive, frequent incremental journal writes with larger periodic checkpoints. This provides faster recovery, minimizes data loss, and avoids deployment-related load spikes while retaining checkpoints for efficient long-term storage and features such as version history.

Read original

Write Ahead Log PostgreSQL Real Time Collaboration Websockets+2