Measuring Time Savings From Figma Make | Figma Blog
Figma’s Data Science team found that Figma Make reduced design-task completion time by 20% and made work 16% easier. Product managers benefited most, completing tasks 23% faster and reporting a 37% improvement in ease. Because ordinary A/B tests and observational analyses could not adequately control for task complexity and user experience, Figma used a randomized controlled trial (RCT) with 100 participants. ## Why Measuring AI Time Savings Is Difficult - Productivity is influenced by confounders such as: - Job tenure and career experience - Individual design ability - Task complexity - Without controlling for these factors, it is difficult to determine whether improvements come from AI or from differences among users and tasks. ## Limitations of Common Research Methods - **Online A/B testing** - Randomly assigning users to treatment and control groups helps balance user characteristics. - However, users may perform different tasks, making it difficult to ensure that task complexity is comparable. - **Causal inference using product logs** - Methods such as propensity score matching require all relevant confounders to be present in the data. - Anonymized logs cannot capture subjective factors such as a user’s design experience. - Instrumental-variable analysis requires a valid factor that influences AI usage without independently affecting task speed; Figma could not identify one. ## The Randomized Controlled Trial - RCTs were selected because they can control confounders before data collection begins. - The study combined: - Random assignment to Figma Make and control groups - Identical tasks for all participants - Moderation by trained researchers - The study focused only on Figma Make to avoid introducing variables from multiple AI tools. - Participants included 100 people: - 50 product designers - 50 product managers - The sample size was based on effect sizes from prior industry research, including GitHub Copilot RCTs, followed by a statistical power analysis. ## Findings - Overall, Figma Make: - Made design work **20% faster** - Made work **16% easier** - Product managers experienced the largest gains: - Tasks were **23% faster** - Tasks were **37% easier** The study suggests that a carefully controlled RCT is a more reliable way to measure AI’s productivity impact when task differences and user characteristics are difficult to capture in product data. Teams evaluating similar tools should standardize tasks, randomize participants, and moderate the study to separate genuine AI benefits from other sources of variation.
Read original(opens in new tab)