From Multi-Day Latency to Near Real-Time Insights: Figma’s Data Pipeline Upgrade | Figma Blog

Figma replaced a daily full-table export system that could take hours or days with an incremental synchronization pipeline designed for near real-time analytics. The new architecture combines database snapshots, change data capture (CDC), and Snowflake merge logic to transfer only recent changes. By building the system in-house, Figma gained greater flexibility, lower projected costs, and a design that can scale with continued growth.

Why the Legacy Pipeline Failed

• Since 2020, a daily cron job ran SELECT * FROM <TABLE>, exported results to S3, and loaded them into Snowflake.
• As Figma’s tables and insert volume grew:
  • Daily syncs reached roughly six hours by 2023.
  • The largest tables took several days or longer.
  • Additional database replicas were required for exports.
  • Replica maintenance cost millions of dollars annually.
• The delays limited access to timely company KPIs and analytical insights.

Choosing Incremental Synchronization

Figma evaluated three options:

• Continue using the legacy process, which was increasingly expensive and too slow.
• Add parallelism, which might improve throughput temporarily but would not scale sustainably.
• Rebuild the synchronization system around incremental updates.

Incremental synchronization transfers only new and changed records instead of repeatedly copying entire tables, reducing data movement, processing time, and infrastructure usage.

Buy vs. Build

Figma decided to build the pipeline internally because available proprietary tools did not meet its requirements.

• Flexibility: Generic SQL tools did not take advantage of capabilities such as Amazon RDS for PostgreSQL snapshot exports.
• Cost: Commercial solutions were projected to cost five to ten times more than an in-house implementation.
• Scale: Building internally allowed Figma to optimize the system for its infrastructure and adapt it as the company grows.

Pipeline Components

The bespoke system combines several lower-level technologies:

• Snapshots: Amazon RDS exports initial table copies to S3.
• Change data capture: Kafka Connect streams database changes through Amazon MSK.
• Warehouse ingestion: A Snowflake Connector loads CDC events into Snowflake.
• Incremental merging: Custom Snowflake stored procedures and scheduled tasks merge changes into base tables.

Architecture Principles

The redesign was guided by four goals:

• Reduce end-to-end synchronization latency.
• Control costs as data volume increases.
• Meet regulatory and compliance requirements.
• Preserve data accuracy, completeness, consistency, and trustworthiness.

The resulting architecture uses two workflows: a bootstrap workflow for onboarding tables and a validation workflow for checking data correctness.

Bootstrap Workflow

The automated onboarding process includes:

• The CDC service begins capturing the new Postgres table and publishes events to a per-table Kafka topic.
• Amazon RDS exports the latest database snapshot to S3.
• Snowflake’s COPY INTO <table> loads the snapshot into a per-entity base table.
• An MSK Connect Snowflake Sink Connector streams Kafka events into a separate CDC table, with offsets arranged so changes before the snapshot timestamp are retained.
• A scheduled Snowflake task runs a custom MERGE procedure to combine the snapshot and CDC data.
• Once the process catches up with current changes, Figma creates a lightweight user-facing view over the base table.

Zero-Downtime Re-Bootstrapping

• Bootstrap artifacts are versioned, while the final user-facing view remains stable.
• New versions can be built in parallel without interrupting queries.
• Promotion is completed through an atomic view update.
• This supports schema evolution and other situations requiring a fresh bootstrap without downtime.

Data Validation

• Even well-designed pipelines can suffer corruption from partial failures, configuration errors, software bugs, or unexpected source-data anomalies.
• Figma therefore added a validation workflow to verify correctness as data moves through snapshot exports, CDC capture, and incremental merging.

Figma’s experience shows that incremental synchronization is a more sustainable alternative to repeated full-table exports. Combining managed infrastructure with custom orchestration can deliver lower latency, better cost control, and stronger operational flexibility than a one-size-fits-all commercial pipeline.