techlist.io
KREN
figma

Postmortem: Service disruptions on June 6 & 7 2022 | Figma Blog (opens in new tab)

redisincident-responseaws-elasticachepublish-subscribecluster-mode

Figma experienced four service disruptions between June 6–7, 2022, lasting from seven minutes to 1 hour 20 minutes. Existing files remained usable, but users could not open new files or collaborate; no data was lost. The root cause was a rare AWS ElastiCache/Redis bug that saturated the Engine CPU of a Cluster-Mode Enabled node under heavy Publish/Subscribe load.

Impact and Timeline

  • Incidents occurred between 11:34 PM PDT on June 6 and 10:43 AM PDT on June 7.
  • The web application became fully functional at approximately 10:41 AM; some API features were restored later.
  • Users could continue working in already-open files.
  • Opening files and real-time collaboration were unavailable during disruptions.
  • Local changes were preserved and synchronized when service connectivity returned.

Redis and ElastiCache Architecture

  • Figma uses AWS ElastiCache, a managed Redis service, for:
    • Caching frequently accessed data
    • Routing messages between services
  • Its setup included:
    • A Cluster-Mode Disabled instance
    • A Cluster-Mode Enabled instance, which supports horizontal scaling by adding Redis nodes
  • Several weeks before the outage, Figma moved its Redis Publish/Subscribe traffic from the CMD instance to the CME instance.
  • The workload had operated normally for weeks, with no obvious traffic or usage changes immediately before the incidents.

Root Cause: Engine CPU Saturation

  • Monitoring alerted Figma within seconds that one CME ElastiCache node had reached 100% Engine CPU.
  • AWS later identified a rare Redis bug triggered by high Publish/Subscribe traffic on CME clusters.
  • Figma could not obtain CPU profiles from the underlying ElastiCache machines, making it difficult to identify the exact operation consuming CPU.
  • Increasing cluster capacity did not solve the problem and ultimately made the behavior worse.

Mitigation and Investigation

  • Figma initially suspected insufficient capacity or faulty hardware.
  • Engineers:
    • Initiated a failover of the affected node
    • Created a larger ElastiCache cluster with more nodes
    • Redirected traffic to the replacement clusters when failover took too long
  • The first traffic redirection restored service, but the new cluster later experienced the same CPU saturation.
  • A subsequent failover completed successfully and restored service more quickly.
  • Engineers investigated and ruled out:
    • Routine backups or unexpected snapshots
    • Sudden increases in Redis command volume
    • Slow commands, large keys, or other obvious misuse
    • Scheduled background jobs
  • Blocking requests that appeared to be waiting longest for Redis commands did not resolve the issue.

Figma’s outage was ultimately caused by an AWS Redis/ElastiCache defect rather than a data-loss event or an observable change in application traffic. The incident underscores the need to validate Redis Publish/Subscribe workloads on Cluster-Mode Enabled deployments and maintain mitigation strategies that do not rely solely on adding capacity.