PostgreSQL

60 posts

datadogJan 15, 20252 min readCurated summary

Unraveling a Postgres segfault that uncovered an Arm64 JIT compiler bug

Postgres was crashing with segmentation faults when executing certain expensive queries on an Arm64 Kubernetes cluster. Investigators reduced the failure to a simple table scan and discovered that disabling JIT compilation prevented the crash. Assembly-level debugging ultimately traced the problem to a bug in LLVM’s Arm64 JIT support. ## Isolating the Crash - The failures occurred across multiple EC2 nodes, ruling out faulty hardware. - Query logs showed that the crashes consistently followed a small number of query patterns. - The simplest reproducer was: ```sql SELECT repo_id FROM repository; ``` - Core dumps had badly corrupted stacks, but surviving frames pointed to `ExecRunCompiledExpr`, suggesting a failure during JIT execution. - The unusually short backtraces reinforced the suspicion that the stack itself had been corrupted. ## How PostgreSQL JIT Works - PostgreSQL normally evaluates SQL expressions through a general-purpose interpreter. - JIT compilation converts expressions such as `1+1` into native machine code, reducing interpreter overhead for large workloads. - JIT can also optimize tuple deforming by converting disk tuples into in-memory values more efficiently. - PostgreSQL uses LLVM to generate the compiled code. - Because compilation adds overhead and compiled functions are not reused between queries, PostgreSQL enables JIT primarily for expensive queries based on cost thresholds. ## The Query of Death - The affected query scanned a partitioned `repository` table with: - 64 partitions - More than 1.6 million rows - 128 JIT-generated functions - Its query plan enabled expression compilation and tuple deforming: ```text JIT: Functions: 128 Expressions: true Deforming: true Inlining: false Optimization: false ``` - Running the query with: ```sql SET jit = off; ``` completed successfully. - Disabling JIT cluster-wide immediately stopped the crashes without noticeable query-latency effects. ## Root Cause Direction - The release build of PostgreSQL offered limited debugging flexibility, so the team planned to reproduce the failure in a dedicated test environment. - Further investigation eventually isolated the defect to JIT compilation on Arm64 systems. - The underlying issue was identified as an LLVM bug rather than a PostgreSQL query or hardware problem. - The investigation continued down to generated assembly and resulted in an upstream fix. The immediate mitigation was to disable PostgreSQL JIT, while the durable solution was to adopt the LLVM fix addressing the Arm64 code-generation bug.

PostgreSQL Database Design Kubernetes Cloud Computing+3

datadogMay 21, 20243 min readCurated summary

Engineering VP spotlight: Ivo Dimitrov

Ivo Dimitrov’s career evolved from low-level systems programming into engineering leadership focused on large-scale distributed storage. His experience at Microsoft and LinkedIn shaped his approach to building scalable data platforms, while Datadog attracted him with its talented people, modern technology, and culture of experimentation. Today, he leads Datadog’s Distributed Data Systems organization, supporting the company’s metrics, events, query, alerting, and analytics infrastructure. ## From Electrical Engineering to Systems Programming - Dimitrov initially studied electrical engineering and became interested in software while working on digital control systems. - His early work included contributing to a real-time operating system kernel. - He spent roughly a decade as an individual contributor, specializing in: - High-performance systems - Low-level programming - C and C++ - System software ## Transition from Individual Contributor to Manager - At Microsoft, Dimitrov worked on an early version of Azure Blob Storage. - Following a reorganization, he accepted an opportunity to lead his team despite having no prior management experience. - Microsoft supported the transition through: - Leadership mentorship - Formal management training - Guidance on communication, conflict resolution, and interpersonal leadership - He discovered that management allowed him to expand his ownership beyond individual projects and influence broader organizational outcomes. - The role combined his technical background with responsibilities such as cross-functional coordination, team development, and engineering strategy. ## Building Internet-Scale Storage at Microsoft and LinkedIn - At Microsoft, Dimitrov worked on storage systems supporting Hotmail. - After joining LinkedIn in 2014, he adapted to a technology environment centered on open source tools such as MySQL and Java. - He led development of Espresso, LinkedIn’s proprietary key-value storage platform. - The platform matured into a core system supporting approximately 95 percent of LinkedIn’s data sets. - He also helped oversee several other large-scale storage projects: - **Venice**, an open source platform for serving derived data - **Ambry**, an open source blob storage system - **Helix**, an open source cluster manager - These systems supported critical parts of LinkedIn’s internet-scale infrastructure. ## Why Datadog Was Appealing - Dimitrov was drawn to Datadog by three main factors: - Highly capable engineers and leaders - Interesting, modern technology - The opportunity to contribute to a rapidly growing company - Compared with the legacy systems and processes that had accumulated at LinkedIn, Datadog offered less bureaucracy and more freedom to: - Take thoughtful risks - Experiment - Deliver quickly - Fail fast and learn - Iterate and innovate - He was particularly interested in Datadog’s Kubernetes-based Metrics and Events platforms and the challenge of building best-in-class infrastructure during the company’s growth. ## Distributed Data Systems at Datadog - Dimitrov leads the Distributed Data Systems organization, which owns a portfolio of storage and data technologies. - Its responsibilities include: - **Metrics**, supporting metrics and time-series data - **Events**, handling semi-structured data such as logs, profiles, and traces - **Driveline**, a main-memory database optimized for online analytics - The **Cross-Platform Queries** team provides a unified query interface across systems that historically exposed separate, domain-specific APIs. - This reduces the learning curve for engineers and customers. - It abstracts the underlying data stores behind a common API. - The organization also operates Datadog’s Alerts platform, which generates a large share of the queries sent to the Metrics and Events systems. Dimitrov’s experience demonstrates how deep systems expertise can translate into effective engineering leadership. His recommendation by example is to remain technically engaged while expanding one’s scope—from writing individual components to shaping teams, platforms, and long-term engineering direction.

PostgreSQL Database Design Kubernetes Distributed Systems+4

figmaMay 16, 20242 min readCurated summary

Keeping It 100(x) With Real-time Data At Scale | Figma Blog

Figma’s LiveGraph powers real-time collaboration by subscribing to GraphQL-like queries and updating clients automatically. Rapid growth—tripled sessions since 2021 and fivefold view-request growth in one year—exposed limits in its single-server, mutation-based architecture. Figma launched “LiveGraph 100x,” a redesign focused on scaling reads and database updates while preserving performance and enabling a safe migration. ## LiveGraph’s Role in Figma - LiveGraph keeps data synchronized across collaborative features such as: - File editing - Comments - FigJam voting - It exposes a web API for subscribing to GraphQL-like queries. - Results are returned as JSON trees based on a schema of entities, relationships, and views. - A custom React Hook automatically re-renders interfaces when subscribed data changes. ## The 100x Scaling Initiative Figma’s growing user base increased both the number and cost of LiveGraph client sessions. At the same time, the underlying database evolved from one PostgreSQL instance into vertically and horizontally sharded infrastructure. The redesigned system needed to: - Preserve or improve service-level objectives for initial loads and updates. - Support more database shards reliably and efficiently. - Scale reads and database-update processing independently. - Allow incremental, transparent migrations without disrupting users. ## Limitations of the Original Architecture Originally, LiveGraph consisted of: - A single LiveGraph server. - An in-memory query cache. - One PostgreSQL instance. - A cache that tailed PostgreSQL’s logical replication stream. PostgreSQL writes row mutations to its write-ahead log, including pre- and post-row images and a monotonically increasing sequence number. LiveGraph used these mutations to update cached query results directly rather than recomputing them. This design worked well at smaller scale because: - All updates came from one primary database. - The replication stream provided a global ordering. - Each row mutation could be applied directly to the relevant cached results. ## Sharding Breaks Global Ordering As the original PostgreSQL instance reached capacity, Figma introduced vertical shards and began moving toward broader horizontal scaling. This invalidated the assumption that all database updates arrive in one globally ordered stream. - Multiple shards can generate updates simultaneously. - Their updates have no guaranteed global order. - LiveGraph therefore needed an architecture that could process distributed database changes while maintaining reliable, timely query updates. The growing load made it necessary to rethink LiveGraph fundamentally rather than continue extending its single-database design.

PostgreSQL Distributed Systems GraphQL Real Time Data+4

figmaMar 13, 20243 min readCurated summary

How Figma's Databases Team Lived to Tell the Scale | Figma Blog

Figma’s database stack grew nearly 100× from 2020, pushing its single-Postgres architecture beyond the limits of vertical partitioning. After adding caching, read replicas, and vertically partitioned databases, the team found that individual tables were reaching terabyte and billion-row scales, creating vacuum reliability issues and approaching AWS RDS IOPS limits. The solution was to pursue horizontal sharding while preserving Postgres, minimizing application changes, avoiding massive backfills, and maintaining consistency and rollback options. ## Scaling from One Postgres Database - In 2020, Figma ran on one large Postgres instance. - By the end of 2022, it had introduced: - Caching - Read replicas - Around a dozen vertically partitioned databases - Related tables, such as those for Figma files and organizations, were grouped into separate database partitions. - Vertical partitioning reduced pressure on the system and provided valuable short-term runway. ## Why Vertical Partitioning Was No Longer Enough - The team monitored multiple scaling constraints, including: - CPU and I/O utilization - Table size - Rows written - Database IOPS - Some tables grew to several terabytes and billions of rows. - Large tables began affecting reliability during PostgreSQL vacuum operations, which prevent transaction ID exhaustion. - High-write tables were on track to exceed the maximum IOPS supported by Amazon RDS. - Because a table is the smallest unit of vertical partitioning, splitting databases by table group could not solve these limits. ## Requirements for the Next Scaling Strategy Figma established several design goals for horizontal scaling: - Minimize developer changes and preserve the existing relational data model. - Make future scale-outs transparent to application teams after initial compatibility work. - Avoid months-long backfills of large tables. - Roll out changes incrementally to reduce outage risk. - Preserve rollback capability after physical sharding. - Maintain strong consistency without relying on difficult double-write schemes. - Support near-zero-downtime scale-outs. - Favor technologies and techniques the database team already understood, given the limited runway. ## Evaluating Alternatives - The team considered CockroachDB, TiDB, Spanner, and Vitess. - Moving to another database would have required a risky migration between storage systems while preserving consistency and reliability. - Figma already had substantial operational expertise running Postgres on RDS; replacing it would mean rebuilding that expertise under severe time pressure. - NoSQL systems were also unsuitable because Figma’s application depends on a complex relational data model and requires the flexibility of relational queries. - The team therefore favored a lower-risk approach that retained Postgres and offered greater control over the migration. ## Practical Direction Figma’s experience shows that vertical partitioning can be an effective intermediate step, but it cannot solve limits imposed by individual tables. For systems with rapidly growing relational workloads, horizontal sharding within a familiar database ecosystem can provide a safer path to scale when it is introduced incrementally and designed around consistency, rollback, and minimal application disruption.

PostgreSQL Distributed Systems Database Sharding Database Replication+2

datadogDec 21, 20233 min readCurated summary

Scaling self-serve analytics: The tools empowering 5,000 employees

Datadog scaled self-serve analytics from 200 to 5,000 employees by building an open-source-based platform around three pillars: trusted data, accessible tools, and organizational knowledge. The goal is to let employees answer routine questions and make informed decisions without relying on a centralized Data & Analytics team. This approach combines a single source of truth, self-service data pipelines and transformations, data discovery, quality monitoring, and training. ## The Purpose of Self-Serve Analytics - Datadog’s mission is to “empower everyone at Datadog to make data-informed decisions on their own.” - Self-service allows Data & Analytics teams to focus on higher-value initiatives instead of handling every request. - The organization identified three primary user profiles: - **Analytics Explorers:** Need discoverable data and ready-made reports. - **Analytics Builders:** Create reports and run advanced queries. - **Analytics Experts:** Expose new data, maintain business logic, and manage quality. ## Data as a Single Source of Truth - Datadog centralizes product, operational, and business data so consumers work from the same version of reality. - Its “Bring Your Own Data” (BYOD) tool lets teams expose their own data for analytics. - The shared data layer supports BI tools, notebooks, data discovery, programmatic access, and machine-learning models. - Trust depends on: - Consistent naming and modeling conventions. - Comprehensive documentation. - Continuous data-quality monitoring. ## Self-Serve Data Intake - Teams can connect internal and third-party data sources through integrations and BYOD. - The platform provides scheduling and a user interface for exposing or requesting datasets. - Pipeline observability covers: - Pipeline execution. - Data quality. - Actionable alerts when failures occur. ## Self-Serve Transformation - Analysts manage their departments’ business logic using SQL and dbt. - The development environment integrates with workflow management, metadata, and pipeline-run systems. - Enforced conventions keep the shared modeling layer consistent and understandable as more analysts contribute. - Analysts can inspect lineage, pipeline runs, quality checks, and alerts. ## Data Discovery and Metadata - Every employee can browse datasets and fields in the central data platform. - Search capabilities help users identify which data can answer a particular question. - Metadata explains: - The dataset’s origin and owner. - Definitions and intended meaning. - Where the data is used. - Sensitivity and reliability. - This context helps employees determine whether data is both relevant and trustworthy. ## Supporting Adoption - Tools alone are insufficient; Datadog also provides data knowledge, support, and training. - The Data & Analytics organization acknowledges that self-service has limits and works to mitigate risks such as misunderstanding data or applying incorrect business logic. - Success is tracked through adoption and the effectiveness of the overall self-service strategy. Datadog’s experience suggests that self-serve analytics scales best when data is treated as a product: centralized, documented, observable, and accessible through tools designed for users with different levels of expertise.

PostgreSQL Sql Dbt Open Source Tools+4

figmaApr 3, 20233 min readCurated summary

The growing pains of database architecture | Figma Blog

Figma outgrew its single Amazon RDS PostgreSQL database as traffic increased roughly threefold annually, pushing peak CPU utilization above 65% and making latency unpredictable. Initial fixes—larger hardware, read replicas, new databases, and PgBouncer—provided temporary relief but could not adequately reduce write load or handle replication-sensitive reads. Figma ultimately chose vertical partitioning, moving groups of related tables into separate databases as a lower-risk, incremental path to scalability. ## The Limits of a Single Database - Figma stored metadata such as permissions, file information, and comments in one large RDS instance. - Increasing users, new features, and preparation for a second product drove database traffic sharply upward. - Peak CPU utilization reached more than 65%, with latency becoming less predictable as the database approached its limits. - Full saturation would have made Figma unavailable, so the infrastructure team addressed the risk before it became an outage. ## Tactical Measures for More Headroom Figma introduced several short-term improvements: - Upgraded the database from an `r5.12xlarge` to an `r5.24xlarge` instance. - Added multiple read replicas to distribute read traffic. - Created separate databases for new use cases to prevent further growth of the original database. - Added PgBouncer to pool connections and reduce the impact of thousands of application connections. - These changes provided approximately another year of runway, but writes still consumed substantial resources. - Some reads could not be moved to replicas because the application was sensitive to replication lag. ## Evaluating Horizontal Scaling Figma considered horizontally sharding the database but found substantial technical and operational risks: - Many managed horizontally scalable databases were not natively compatible with PostgreSQL. - Migrating to NoSQL or Vitess would require complex double-read and double-write migration strategies. - NoSQL would also require significant application changes. - A managed distributed PostgreSQL system could make Figma an unusually large customer, exposing it to untested scaling limits. - Self-hosting would require new expertise, training, and considerable operational investment, diverting attention from the core scalability problem. ## Choosing Vertical Partitioning Instead of splitting individual tables across many database nodes, Figma chose vertical partitioning: - Groups of related tables would be moved to separate databases. - This approach immediately reduced load on the original database. - It preserved a future path toward horizontal sharding for particularly large or demanding table groups. - The strategy was considered more incremental and operationally manageable than replacing PostgreSQL or adopting a self-hosted distributed system. ## Selecting Tables to Move Figma evaluated candidate tables using two criteria: - **Impact:** Moving the tables should remove a meaningful portion of the database workload. - **Isolation:** The tables should have limited dependency on tables that remained in the original database. - To measure impact, the team analyzed average active sessions (AAS), which estimates the average number of active threads handling a query. - They gathered query activity from PostgreSQL’s `pg_stat_activity` view at 10-millisecond intervals to identify CPU waits associated with individual queries. Figma’s experience shows that database scaling does not always require an immediate move to distributed infrastructure. Carefully selected vertical partitioning can reduce pressure on a primary database while limiting migration risk and preserving more ambitious scaling options for the future.

PostgreSQL Amazon Rds Database Sharding Pgbouncer+3

figmaOct 19, 20223 min readCurated summary

Making multiplayer more reliable | Figma Blog

Figma improved multiplayer reliability by adding a durable write-ahead journal alongside its existing checkpoint system. Instead of relying on full-file snapshots every 30–60 seconds, Figma now records incremental changes frequently, allowing crashed servers to recover nearly to the latest state and reducing deployment-related database spikes. The goal was to reduce potential data loss from up to 60 seconds to less than one second. ## How Figma’s Multiplayer System Worked - Browsers connect to Figma’s multiplayer service over WebSockets. - The service authoritatively handles: - Validation - Ordering - Conflict resolution - Broadcasting updates to connected clients - File state is held in memory for speed. - Every 30–60 seconds, the entire file is: - Encoded into a binary format - Compressed - Uploaded to Amazon S3 as a checkpoint ## Problems with Checkpoint-Only Persistence - A multiplayer crash could lose up to 60 seconds of server-side work. - Checkpoints become increasingly expensive as files grow in size and complexity. - Redeployments caused large write spikes: - All in-memory files had to be closed. - Each file needed a final checkpoint. - The resulting burst increased database load. ## Introducing the Journal - Figma added a durable transaction log, or journal, backed by DynamoDB. - Each accepted change receives an incrementing sequence number. - Checkpoints store the latest sequence number they include. - During recovery: - Multiplayer loads the latest checkpoint. - It queries the journal for entries with higher sequence numbers. - It replays those incremental changes to reconstruct the latest file state. - Journal entries are much smaller than full-file checkpoints, since they contain only user edits. - Figma writes journal data roughly every 0.5 seconds rather than waiting 60 seconds between full snapshots. - The target was less than one second of data loss in rare failure scenarios. ## Smoother Deployments - During deployment, Figma can close connections and wait for unsaved changes to reach the journal. - The 99th percentile persistence time is under one second. - Since journal writes happen continuously during normal operation, deployments no longer create a sudden checkpoint-writing surge. - Database write load is therefore steadier and more predictable. ## Datastore and Batching Decisions - Figma selected DynamoDB because the journal requires a horizontally scalable datastore with high write capacity. - Postgres was considered but rejected because the anticipated write volume exceeded Figma’s current horizontal-scaling approach for Postgres. - Clients send updates at approximately 30 frames per second, or every 33 milliseconds. - The journal does not need that granularity, so multiple changes can be batched before being persisted, improving performance. Figma’s approach combines inexpensive, frequent incremental journal writes with larger periodic checkpoints. This provides faster recovery, minimizes data loss, and avoids deployment-related load spikes while retaining checkpoints for efficient long-term storage and features such as version history.

PostgreSQL Real Time Collaboration Websockets Dynamodb+2

datadogMay 12, 20223 min readCurated summary

How Datadog's IT team automated account inactivity and SaaS spend management

Datadog expanded its Clarity auditing tool into Clarity License Manager (CLM), a system that tracks SaaS usage, reduces licensing costs, and improves security. CLM identifies inactive accounts, notifies employees, automatically deactivates unused access, and restores it quickly when needed. Its microservice architecture and application-specific adapters allow the system to scale across many SaaS products. ## The SaaS License Management Problem - Datadog used many commercial SaaS tools with substantial per-user costs. - License usage data was outdated and collected through quarterly manual audits. - IT Support had to contact employees individually, creating administrative overhead and a poor user experience. - Unused accounts also created security risks, including stale credentials that could be compromised. ## Goals of Clarity License Manager - Monitor and automatically deactivate inactive accounts, especially in sensitive services such as cloud providers. - Reduce the risk of leaked or abused stale credentials. - Limit the potential impact of security incidents. - Lower SaaS spending and support data-driven licensing decisions. - Preserve employee productivity through an easy account restoration process. ## Usage Monitoring and Automated Workflows - CLM gathers activity data through: - Direct integrations with individual SaaS APIs. - Google Workspace SAML audit logs for indirect integrations. - Employee activity is stored per application in an Amazon RDS-backed PostgreSQL database. - Employees receive email and Slack notifications after a configurable period of inactivity, with 90 days as the default. - Notifications explain the specific login or application action required to remain active. - If the employee does not respond after multiple reminders, CLM deactivates the account automatically. - Employees can reopen access by submitting a Freshservice ticket. - Accounts are restored within seconds, including their previous roles and permissions. ## Microservice Architecture - CLM consists of Python microservices running on AWS Lambda. - The services share a central PostgreSQL database. - Microservices provide: - Easier scaling as Datadog adds more SaaS applications. - Greater resilience and flexibility. - A modular foundation for future development. - The architecture introduced complexity because services required different APIs and libraries with overlapping functionality. ## Application-Specific Adapters - Each SaaS product is represented by an adapter shared across CLM microservices. - Adapters isolate application-specific API logic from the core workflows. - A typical adapter supports operations such as: - Retrieving users. - Fetching login activity. - Activating and deactivating accounts. - Onboarding and offboarding users. - This design provides: - Clear separation of responsibilities. - Reusable and flexible integration code. - Simpler microservices that do not need to handle each application’s unique behavior. CLM demonstrates how automated usage monitoring can simultaneously improve SaaS security, reduce unnecessary spending, and minimize disruption for employees. A modular adapter-based architecture is particularly useful when managing a growing portfolio of third-party applications.

PostgreSQL Database Design Python Microservices+3

datadogApr 13, 20223 min readCurated summary

It's always DNS . . . except when it's not: A deep dive through gRPC, Kubernetes, and AWS networking

A routine update to a critical metrics query service caused intermittent errors and increased latency. Although logs initially pointed to DNS failures, the investigation revealed a deeper networking problem involving dropped packets and saturated AWS VPC connection tracking. The incident highlighted how Kubernetes, Cilium, AWS networking, and DNS behavior can interact in ways that obscure the true cause. ## Initial Symptoms and Apparent DNS Failures - Errors increased whenever the metrics query service was rolled out. - The service retrieves data from metric stores for dashboards and monitor evaluations. - Automatic retries reduced user-facing failures but increased latency. - Service logs showed DNS errors when connecting to dependencies inside Kubernetes. - The investigation therefore began with the cluster’s DNS infrastructure. ## NodeLocal DNSCache Reaches Its Limits - NodeLocal DNSCache runs as a `node-local-dns` DaemonSet on every Kubernetes node. - DNS pods had: - A 64 MB memory limit - A `max_concurrent` limit of 1,000 requests - The pods experienced out-of-memory errors and rejected requests during rollouts. - Increasing memory to 256 MB stopped the OOM errors, but DNS failures continued. - Request volume was far below the expected capacity: - Normally about 400 queries per second - Nearly 2,000 queries per second during rollouts - Expected capacity of at least 200,000 queries per second - Upstream resolvers were marked unhealthy, suggesting that NodeLocal DNSCache could not establish or maintain connections. - Because upstream requests could wait up to five seconds, connection failures consumed concurrency slots and made the cache appear overloaded. ## Evidence of a Network Problem - The instances were below their 5-Gbps sustained throughput limits. - TCP retransmits increased in correlation with service rollouts. - Engineers suspected brief traffic spikes, or microbursts, that were not visible in aggregate throughput metrics. - This shifted the investigation from DNS configuration toward lower-level AWS networking behavior. ## AWS VPC Connection Tracking - ENA metrics revealed a significant increase in `conntrack_allowance_exceeded`. - This metric counts packets dropped when VPC connection tracking becomes saturated. - Connection tracking maintains state for network flows and supports features such as stateful EC2 security groups. - The infrastructure used two tracking layers: - VPC conntrack maintained at the hypervisor level - Linux conntrack inside each instance - VPC conntrack appeared saturated even though Linux conntrack contained fewer than 60,000 entries—well within the observed capacity of similar instances. - AWS Support confirmed that conntrack capacity varies by instance type and that VPC conntrack limits could differ substantially from Linux conntrack limits. - Scaling to larger instances resolved the symptoms, but the engineers wanted to understand the traffic pattern and find a more efficient long-term solution. ## VPC Flow Logs as the Next Investigation Tool - The team turned to Amazon VPC Flow Logs to examine the service’s low-level network behavior. - These logs were expected to clarify why connection tracking filled up and how rollout traffic contributed to the saturation. - The investigation was still ongoing at the point where the provided article excerpt ends.

PostgreSQL Database Design Kubernetes AWS+4

figmaOct 13, 20213 min readCurated summary

LiveGraph: real-time data fetching at Figma | Figma Blog

LiveGraph is Figma’s in-house real-time data-fetching layer built on PostgreSQL. It lets frontend developers declare live data views with GraphQL-like queries, while LiveGraph reads PostgreSQL’s replication stream to deliver updates within milliseconds. Figma built it to replace fragile, manually maintained client events and to support real-time subscriptions at large scale without relying on polling or a new database technology. ## Problems with Figma’s Earlier Real-Time Architecture - React clients initially loaded large data sets through Ruby HTTP endpoints and stored them in Redux. - Backend code manually emitted events whenever database records changed. - Frontends subscribed over WebSockets and applied those events to client state. - As data volumes grew, Figma split requests into incremental loads, making data ownership and availability harder to reason about. - Complex changes—such as permission updates affecting many resources—were difficult to represent with individual events. - Events could arrive out of order or fail to correspond reliably with database writes, causing client state to diverge from server state. ## Why Figma Chose Live Queries - Figma wanted developers to define data subscriptions declaratively rather than manually coordinate fetches and update events. - GraphQL provided a natural interface for describing the relevant portion of the object graph. - LiveGraph uses “live queries,” which keep query results synchronized, rather than GraphQL subscriptions in the narrower sense of consuming event streams. - The system is a query and data-fetching layer over existing PostgreSQL infrastructure, not a replacement persistence layer. ## In-House System Versus Existing Tools - Figma’s multiplayer service handles collaborative writes and conflict resolution within individual files, whereas LiveGraph focuses on reading application data. - Systems such as Hasura, Prisma, and PostGraphile offered GraphQL subscription features but were not designed primarily for Figma’s scale of concurrent live subscriptions. - Polling was rejected because it increases database load and requires developers to choose polling intervals for each query. - Figma’s collaborative product made real-time data central enough to justify building and operating a specialized internal system. - The company did not claim LiveGraph was universally superior; its value came from matching Figma’s specific scale and requirements. ## Replication-Stream-Based Updates - LiveGraph executes queries directly against PostgreSQL. - It tails the database replication log to detect changes instead of repeatedly polling tables. - Reading the replication stream enables update latency measured in milliseconds. - Because the system must process the complete volume of database changes, its architecture needs to distribute updates across machines and database shards. - This approach separates the complexity of detecting database changes from product code, allowing frontend engineers to work with declarative JSON data views. ## Frontend API - Product developers send GraphQL-like queries and receive results as JSON trees. - A schema defines server-side entities and relationships, while views expose queryable subsets of that graph. - The frontend can therefore request the data it needs and rely on LiveGraph to keep the result synchronized as the underlying PostgreSQL data changes. LiveGraph’s central recommendation is architectural: derive live client views from the database’s authoritative change stream rather than maintaining a parallel network of hand-written events. For organizations with similar scale and real-time requirements, this can improve consistency and simplify product development, though Figma’s in-house approach was justified by its unusually collaborative workload.

PostgreSQL React TypeScript GraphQL+4

figmaJan 28, 20203 min readCurated summary

Postmortem: Service disruption on January 21-22, 2020 | Figma Blog

Figma’s January 21–22, 2020 outages were caused by separate database failures that compounded one another. A long-running query triggered the first incident and created a vacuuming backlog; the next day, PostgreSQL 9 produced a severely mis-planned query after database statistics changed, while aggressive autovacuuming increased write and lock pressure. Upgrading to PostgreSQL 11 restored stability and addressed both the query-planning and autovacuum performance issues. ## Incident Timeline ### January 21: Long-Running Query - Automated alerts reported elevated error rates at 6:11 AM PST. - Engineers found an expensive, long-running database query driving CPU usage. - Canceling the query at 6:54 AM restored normal performance. - The terminated query left behind a backlog of data requiring vacuuming. ### January 22: Database Saturation - Increased write IOPS and lock contention appeared, despite database CPU being below normal. - Queued API requests eventually made Figma unavailable to some users. - Engineers canceled nonessential queries and increased allocated IOPS, providing only temporary relief. - Performance deteriorated again in the afternoon. - Restarting the database temporarily disabled a suspected background process. - Figma performed an emergency upgrade from PostgreSQL 9 to PostgreSQL 11. - The service returned online at 8:15 PM, with metrics back to normal. ## Aggressive Autovacuuming - The vacuuming backlog crossed the threshold for PostgreSQL’s more aggressive transaction-ID wraparound protection. - This mode generated substantial locking and write activity, particularly in the PostgreSQL version Figma was using. - Canceling autovacuum operations on large tables temporarily improved metrics, but the operations resumed. - Fully suppressing the aggressive behavior required changing `autovacuum_freeze_max_age` and rebooting the database. - Autovacuum was a significant contributor, but disabling it did not eliminate all performance problems. ## PostgreSQL Query Planner Failure - A complex query repeatedly appeared in lock-contention reports. - PostgreSQL estimated that the query would return more than 20 million rows, while the actual result contained only three. - The incorrect plan used full table scans instead of expected indexes. - It also wrote large amounts of data to temporary buffers, matching the observed increases in write IOPS and temporary-byte metrics. - The issue was likely caused by inaccurate statistics or a PostgreSQL planner defect or limitation following a routine statistics change. ## Upgrade and Preventive Measures - PostgreSQL 11 generated a substantially better plan for the problematic query. - Newer PostgreSQL versions improve autovacuum performance and query planning. - PostgreSQL 10+ also provides more advanced performance-analysis tools through Amazon RDS. - Figma had already tested the PostgreSQL 11 upgrade in staging and prepared a detailed production rollout plan, allowing the emergency upgrade to succeed safely. - The company planned to improve monitoring for expensive queries and impose stricter limits on query execution time. Figma concluded that upgrading PostgreSQL, improving query monitoring, and enforcing tighter runtime limits were necessary to prevent similar database-driven outages.

PostgreSQL Amazon Rds Database Monitoring Query Optimization+2

figmaOct 15, 20193 min readCurated summary

How Figma’s multiplayer technology works | Figma Blog

Figma built a custom multiplayer system because traditional operational transformation (OT) was too complex for its document-editing needs. Its client/server architecture synchronizes document changes over WebSockets, supports offline editing, and separates document collaboration from other data such as comments and users. The system began as a prototype that enabled rapid experimentation before being integrated into production. ## Why Figma Built Its Own Multiplayer System - In 2015, no major design tool offered real-time collaborative editing. - Figma avoided OT, the algorithm used by tools such as Google Docs, because it considered OT unnecessarily complex for its problem space. - The custom approach was designed to be simpler and faster to implement. - Multiplayer eliminated the need to export, email, or manually synchronize design files. - It also allowed non-designers—such as copywriters and developers—to participate or view work without interrupting the designer. ## Figma’s Client/Server Architecture - Figma clients are web pages connected to a server cluster through WebSockets. - Each multiplayer document runs in a separate server process, with all editors connected to that process. - When a document opens, the client downloads an initial copy of the file. - Subsequent changes are synchronized in both directions over the WebSocket connection. - Server performance and scaling were important considerations, later addressed in part through the use of Rust. ## Offline Editing and Reconnection - Clients can continue editing while offline for an arbitrary period. - When reconnecting, the client: - Downloads a fresh version of the document. - Reapplies its locally stored offline edits to that latest state. - Resumes synchronization through a new WebSocket connection. - This keeps connection and reconnection logic relatively simple by concentrating multiplayer complexity on already-connected clients. ## Separate Systems for Different Data - Figma’s multiplayer system is used only for syncing document changes. - Comments, users, teams, projects, and similar information are stored in Postgres. - That data is synchronized through a separate system because it has different requirements involving: - Performance - Offline availability - Security ## Prototyping Before Production - Figma first created a standalone browser-based prototype rather than experimenting directly in the production codebase. - The prototype simulated three clients connected to a server and visualized the complete system state. - Engineers could test: - Offline clients - Bandwidth-limited connections - Different collaborative algorithms - Alternative data structures - Once the design was validated, the ideas were transferred into the main codebase. Figma’s experience demonstrates that collaborative systems do not always require the most established algorithm. A focused, custom protocol—validated through fast prototyping—can provide a simpler solution when its data model and product requirements differ from tools like document editors.

PostgreSQL Real Time Collaboration Rust Websockets+2

datadogApr 2, 20193 min readCurated summary

Building highly reliable data pipelines at Datadog

Datadog’s approach to reliable data pipelines focuses on delivering correct data on time, even when individual jobs fail. Reliability therefore requires fault tolerance, monitoring, and fast recovery rather than eliminating every failure. The company achieves this through isolated, short-lived clusters and pipelines designed to limit the impact of failures. ## Reliability Means Timely, Correct Results - A reliable pipeline is one that consistently produces correct outputs within the required time window. - Occasional crashes do not necessarily make a pipeline unreliable if automatic recovery still delivers the data on schedule. - Pipelines should be designed with the expectation that failures will eventually occur. - Monitoring must detect unexpected failures early, while operational processes should support rapid recovery. ## Architecture for Batch Pipelines - Datadog streams and analyzes live data in real time but uses batch pipelines for features such as optimized long-term storage. - Historical data is stored in object storage. - Cloud Hadoop/Spark services launch and configure processing clusters. - Luigi workers manage tasks and workflows, while Spark workers compile code and submit jobs. - Jobs can be launched through a web interface, command line, or scheduler. ## One Cluster per Pipeline Instead of placing all workloads on one large Hadoop cluster, Datadog gives each pipeline its own cluster. - **Isolation:** Jobs do not compete for resources or interfere with one another, simplifying monitoring and diagnosis. - **Workload-specific hardware:** Clusters can use CPU-optimized or memory-optimized instances depending on the job. - **Elastic scaling:** Clusters can be expanded to catch up with delays or handle growing data volumes without waiting for a shared cluster. - **Safer upgrades:** Hadoop and Spark versions can be upgraded gradually across separate clusters. - Clusters are typically short-lived, averaging about three hours, although dozens may run simultaneously. ## Using Spot Instances to Encourage Fault Tolerance - AWS spot instances can reduce infrastructure costs by as much as 80%, but their nodes may be terminated whenever capacity or demand changes. - Rather than avoiding this failure mode, Datadog designs pipelines to tolerate disappearing clusters. - Long-running jobs are risky because failures discard more work and make recovery slower. - Pipelines are split into smaller jobs: - **Vertically:** Separate transformations into multiple stages, persisting intermediate results in S3. - **Horizontally:** Partition input data so multiple jobs process different portions concurrently. ## Breaking Up the Rollup Pipeline - Datadog’s rollup pipeline generates aggregated time-series data for historical metrics queries. - A single job would take more than 14 hours, making failures costly and difficult to recover from. - The pipeline is divided into two stages: - Aggregate high-resolution data and checkpoint it to S3 as Parquet files. - Convert the intermediate data into a custom format optimized for queries. - As these jobs grew, they were partitioned further using Kafka’s partitioning scheme. - Kafka partitions are grouped into shards, allowing Datadog to: - Adjust how much data each job processes. - Run more or fewer jobs as needed. - Isolate unusually large or sensitive shards. - This decomposition adds overhead because launching jobs and checkpointing to S3 take extra time, but it substantially limits the work lost during failures. ## Practical Recommendation Design pipelines around failure rather than assuming uninterrupted execution. Use isolated, scalable clusters, short jobs, intermediate checkpoints, and partitioned processing so that failures affect only a small portion of the workload and recovery remains fast.

PostgreSQL Database Design AWS Monitoring+4

datadogAug 23, 20173 min readCurated summary

Scaling support with Vagrant and Terraform

Datadog’s Solutions Team uses reproducible virtual environments to investigate customer issues across diverse operating systems, kernels, and integrations. Vagrant simplifies local VM creation, while provisioning scripts eliminate repeated installation and configuration work. Terraform extends the same approach to shared AWS environments, enabling teams to provision, preserve, and collaborate on sandboxes quickly. ## Reproducing Customer Environments with Vagrant - Containers are useful, but virtual machines are better when reproducing specific operating systems, kernels, orchestrators, or complex infrastructure. - Vagrant provides a simple workflow: - `vagrant init` - `vagrant up` - `vagrant ssh` - The main challenge is not creating a VM, but installing and configuring the technologies needed to match a customer’s environment. - With more than 200 integrations, engineers cannot be experts in every technology they may need to troubleshoot. ## Standardizing Setup with Provisioning Scripts - Vagrant provisioning supports tools such as Chef, Puppet, Ansible, and ordinary shell scripts. - Datadog stores reusable reproduction environments in a shared GitHub repository. - Each sandbox includes: - A `Vagrantfile` - A `setup.sh` provisioning script - A `data` directory for configuration files and supporting scripts - A `README.md` with usage information - Engineer-specific values, such as hostnames and tags, are kept in a local `.sandbox.conf.sh` file. - Once a sandbox exists, an engineer can run `vagrant up` and begin reproducing the customer issue within minutes. - The directory hierarchy organizes sandboxes by operating system, version or provider, and technology—for example, Ubuntu Xenial with Kafka. ## Sharing Remote Environments with Terraform - Terraform provides similar infrastructure management for remote cloud instances, including AWS EC2. - The team reuses the same `setup.sh` and `data` files for both Vagrant and Terraform, avoiding duplicate configuration work. - Each sandbox adds a `.tf` file that: - Creates an EC2 instance - Copies required data files - Executes the provisioning script remotely - A shared Terraform module handles common infrastructure tasks, while a `tf.example` file helps engineers create new configurations. - This preserves the same repository structure and workflow while extending sandboxes from local VMs to remote environments. ## Benefits for Team Collaboration - Remote sandboxes can remain available without consuming engineers’ local RAM. - Proper network security allows teammates to access and share environments. - Engineers can reproduce previously configured integrations during live customer interactions. - Investigations can continue across time zones, allowing teams to hand off urgent issues without rebuilding the environment. The overall recommendation is to treat reproduction environments as reusable infrastructure: encode installation and configuration steps once, store them in version control, and use Vagrant for local testing and Terraform for persistent, shared cloud sandboxes.

PostgreSQL Docker & Containers Terraform Virtual Machines+4

figmaApr 11, 20173 min readCurated summary

An alternative approach to rate limiting | Figma Blog

Figma built a Redis-backed rate limiter to protect its web application from excessive traffic and spam. The system needed to work across multiple servers, add minimal latency, remove stale data efficiently, remain accurate, and use little memory. Common algorithms each met some of these goals, but introduced trade-offs involving atomicity, burst behavior, or memory consumption. ## Requirements and Redis - Rate limits cap requests from a user or IP within a time period. - Figma needed shared state because its application ran on multiple machines. - Redis was preferred over PostgreSQL because it provides: - Faster in-memory reads and writes - Built-in expiration for stale tracking data - Efficient storage for rate-limit state ## Token Bucket - Stores each user’s last-request timestamp and remaining token count in a Redis hash. - Tokens refill over time; a request is rejected when no tokens remain. - It is memory-efficient and conceptually elegant. - Its read-then-write operations are not atomic: - Two servers can read the same remaining token count. - Both may accept a request even though only one token was available. - Redis locks could prevent this race but would slow concurrent requests and add complexity. - Lua scripting could make the operations atomic, but Figma avoided introducing that complexity. ## Fixed Window Counters - Stores a request count for each user and fixed time interval, such as one Redis key per minute. - Each request atomically increments its interval’s counter. - Keys expire after the interval, preventing stale data from accumulating. - The approach is simple, memory-efficient, and avoids the token bucket’s race condition. - Its major flaw is boundary bursts: - With a five-request-per-minute limit, a user could send five requests at the end of one minute and five more immediately afterward. - This allows up to twice the intended traffic over a short sliding period. ## Sliding Window Logs - A sliding window log records the timestamps of individual requests. - Older timestamps can be removed as the window advances, and the remaining entries provide an accurate count. - This avoids the boundary problem of fixed windows. - The trade-off is memory usage: users making many requests require many timestamps to be stored. ## The Central Trade-off - Token buckets use little memory but require careful handling of distributed atomicity. - Fixed counters are atomic and efficient but can permit bursts at window boundaries. - Sliding logs are accurate but consume more memory. - Figma’s rate limiter was designed around balancing these competing concerns rather than choosing the theoretically simplest algorithm. The practical lesson is to select a rate-limiting strategy based on the required accuracy, concurrency model, storage system, and memory budget. Redis is a strong fit for distributed rate limiting, but the algorithm must account for both race conditions and burst behavior.

PostgreSQL Distributed Systems Redis Rate Limiting+2