Causal Inference | Techlist.io

figmaAug 11, 20262 min readCurated summary

Measuring Time Savings From Figma Make | Figma Blog

Figma’s Data Science team found that Figma Make reduced design-task completion time by 20% and made work 16% easier. Product managers benefited most, completing tasks 23% faster and reporting a 37% improvement in ease. Because ordinary A/B tests and observational analyses could not adequately control for task complexity and user experience, Figma used a randomized controlled trial (RCT) with 100 participants. ## Why Measuring AI Time Savings Is Difficult - Productivity is influenced by confounders such as: - Job tenure and career experience - Individual design ability - Task complexity - Without controlling for these factors, it is difficult to determine whether improvements come from AI or from differences among users and tasks. ## Limitations of Common Research Methods - **Online A/B testing** - Randomly assigning users to treatment and control groups helps balance user characteristics. - However, users may perform different tasks, making it difficult to ensure that task complexity is comparable. - **Causal inference using product logs** - Methods such as propensity score matching require all relevant confounders to be present in the data. - Anonymized logs cannot capture subjective factors such as a user’s design experience. - Instrumental-variable analysis requires a valid factor that influences AI usage without independently affecting task speed; Figma could not identify one. ## The Randomized Controlled Trial - RCTs were selected because they can control confounders before data collection begins. - The study combined: - Random assignment to Figma Make and control groups - Identical tasks for all participants - Moderation by trained researchers - The study focused only on Figma Make to avoid introducing variables from multiple AI tools. - Participants included 100 people: - 50 product designers - 50 product managers - The sample size was based on effect sizes from prior industry research, including GitHub Copilot RCTs, followed by a statistical power analysis. ## Findings - Overall, Figma Make: - Made design work **20% faster** - Made work **16% easier** - Product managers experienced the largest gains: - Tasks were **23% faster** - Tasks were **37% easier** The study suggests that a carefully controlled RCT is a more reliable way to measure AI’s productivity impact when task differences and user characteristics are difficult to capture in product data. Teams evaluating similar tools should standardize tasks, randomize participants, and moderate the study to separate genuine AI benefits from other sources of variation.

Read original

Causal Inference AI Figma Make Data Science+2

netflixJun 19, 20264 min readCurated summary

A Human-Augmenting Agentic Workflow for Causal Inference

The post presents a human-augmenting agentic workflow for observational causal inference (OCI), designed to automate repetitive analysis while preserving expert oversight. It combines an actor agent that executes analyses with a critic agent that evaluates assumptions, diagnostics, and credibility. The authors argue that transparent artifacts and process audits are essential because observational analyses rarely have definitive ground truth. ## Why Causal Inference Requires Oversight - Data agents can quickly query data and run regressions, but may overlook confounding, selection bias, or differences between average users and specific subgroups. - OCI requires substantial domain judgment, particularly when estimating effects from observational data under an unconfoundedness assumption. - Automation is best used to reduce repetitive work—such as repeated balance checks, sensitivity analyses, and tracking iterations—so practitioners can focus on framing questions and scrutinizing assumptions. ## Target Trial Emulation and Design Diagnostics Netflix’s OCI toolkit frames each analysis around the ideal randomized controlled trial that would answer the question. - This “target trial” clarifies the treatment, outcome, population, timing, and assumptions required for a credible estimate. - The workflow evaluates: - **Covariate balance:** weighted standardized mean differences should generally be below 0.2. - **Overlap:** propensity scores should remain between 0.1 and 0.9. - **Placebo outcomes:** treatment should not appear to affect variables measured before treatment. - **Sensitivity to hidden confounding:** estimated effects should be assessed against hypothetical omitted variables. - These diagnostics help identify whether treated and untreated groups are sufficiently comparable. ## Human-Augmenting Agent Design The workflow uses three personas: - **Principal:** The human data scientist who defines the research question, context, threats to validity, tools, and data. - **Actor:** The software agent that turns the plan into an analysis specification, executes the analysis, runs diagnostics, and produces reproducible artifacts. - **Critic:** The software agent that reviews the plan and results, identifies omissions, assesses credibility, and recommends improvements. The actor and critic operate in an iterative loop. Actors must use only approved tools, create human- and machine-checkable outputs, and report how they address failed diagnostics. Critics check for missing confounders, inconsistencies between the plan and execution, differences between the estimated estimand and the ATE, and gaps relative to the ideal randomized trial. ## Transparent Evaluation Through Artifacts Because observational data generally lacks ground truth, evaluating an agent solely by comparing its answer to a known result is insufficient. - Agents produce plans, specifications, plots, reports, and executed notebooks. - Reports are version-controlled and notebooks are stored so principals can download and re-run them. - Human reviewers can inspect every analytical step rather than trusting only the final estimate. - The workflow also supports conventional evaluations using simulated datasets, while emphasizing process audits for real-world analyses. ## Empowering Practitioners The system provides a templated notebook built on Netflix’s vetted, non-agentic OCI toolkit. - The toolkit uses doubly robust learning for causal effect estimation. - Humans remain responsible for writing the initial analysis plan and reviewing the executed notebook and critic’s report. - The design is intended to extend beyond unconfoundedness-based OCI to methods with different assumptions, such as panel methods requiring parallel trends. ## Case Study: New Entertainment Types Netflix applies the workflow to questions about whether newer entertainment offerings affect member satisfaction and subscription retention. - The case study focuses on an entertainment category referred to as **Type X**. - The broader goal is to estimate effects that could inform business strategy and understanding of member behavior. - The workflow is positioned as a way to combine automated analysis with human judgment in this setting. The recommended approach is not to let an agent make an unaudited causal claim. Instead, practitioners should use agents to execute standardized analyses and diagnostics, then inspect reproducible artifacts and critically assess the assumptions behind the result.

Read original

Causal Inference Agentic Workflow Observational Causal Inference Doubly Robust Learning+3

spotifyMay 18, 20263 min readCurated summary

When Can LLMs Replace Humans in A/B Tests? | Spotify Engineering

LLMs can approximate human outcomes in A/B tests only when strong assumptions hold; unlike randomized user experiments, those assumptions are not guaranteed by design. In the Upworthy headline dataset, raw GPT-4o-mini predictions recovered just 39% of the human treatment effect, but appropriate calibration and repeated sampling substantially improved accuracy. However, the assumptions are hardest to justify for genuinely new products or interventions—the cases where replacing human tests would be most valuable. ## Raw LLM Predictions Underestimate Treatment Effects - Researchers used the Upworthy Research Archive, containing click-through rates from thousands of headline experiments. - GPT-4o-mini predicted click-through rates for treatment and control headlines. - Treating those predictions as human outcomes recovered only 39% of the observed human treatment effect. - The error was systematic rather than random: LLM predictions attenuated effects toward zero. - This could lead companies to underestimate product improvements and make poor shipping decisions. ## Conditions for Valid LLM Surrogates Two assumptions are required: - **Surrogacy:** LLM predictions must capture everything about a treatment that affects human behavior. Once predictions and relevant pre-treatment characteristics are accounted for, treatment assignment should provide no additional information about the human outcome. - **Comparability:** The relationship between LLM predictions and human behavior—the calibration function—must remain stable between historical experiments and the new experiment. - If either condition fails, more LLM samples will not solve the problem; the procedure estimates an effect on the model rather than the effect on users. ## Calibration Methods Matter - Linear calibration with ordinary least squares failed a falsification test, producing estimates 3.8 standard errors away from the human benchmark. - Random forests and gradient-boosted trees performed better because they could model nonlinear relationships between LLM predictions and human click behavior. - Repeatedly sampling the LLM and averaging its outputs reduces temperature-driven noise, lowering bias toward zero and reducing variance. ## Limits for New Interventions - Historical data can partially test surrogacy and comparability, but neither assumption can be verified for a treatment never previously tested. - Trust declines as a new treatment becomes more different from past experiments. - The Upworthy setting is unusually favorable: binary outcomes, text-only treatments, similar headline variants, and an LLM trained on extensive textual data. - These assumptions are much less plausible for changes to layouts, recommendation algorithms, pricing, or entirely new product concepts. Human A/B tests remain essential for genuine product innovation. LLM-based testing is most defensible for incremental changes that resemble well-understood historical treatments, with human experiments used to validate the approach and calibrate its predictions.

Read original

Causal Inference Large Language Models Machine Learning Ab Testing+4

stripeDec 17, 20253 min readCurated summary

Businesses grow revenue on Stripe 27 percentage points faster after accepting financing through Stripe Capital

Stripe’s two-year randomized trials found that businesses accepting Stripe Capital financing grew faster than comparable businesses without financing. The 2023–2025 study showed an average 27-percentage-point growth advantage, while the fastest-improving 10% saw an average boost of 211 percentage points. The results suggest that embedded, data-driven financing can help small businesses overcome traditional lending barriers and invest in growth. ## Proving Financing Causes Growth - Stripe compared businesses that accepted Capital with similar businesses matched on credit, revenue, and longevity. - The study was conducted across two periods: - **2020–2021:** financing was associated with a 114-percentage-point average growth boost, though pandemic-era economic conditions may have influenced results. - **2023–2025:** financing still produced a strong 27-percentage-point average boost in a different economic environment. - Stripe conducted the trial at scale, serving 76,000 financed businesses in 2025 alone. ## Strongest Effects Among Small Businesses - Businesses processing **$3,000–$76,000 annually** saw average growth-rate improvements of **33–43 percentage points**. - Businesses processing less than **$52,000 annually** with top-tier credit scores saw even larger improvements of **94–106 percentage points**. - Even businesses with low or unavailable credit scores experienced **11–18 percentage-point** growth improvements. - Stripe says its data-driven process delivers financing in **1–2 days**, compared with roughly **14–40 days** at traditional banks. - Traditional bank applications are often time-consuming, and rejection rates can approach 50%, including for established businesses. ## Growth-Oriented Spending Produces Better Results - A survey of approximately 900 participating businesses found that financing use strongly correlated with outcomes. - Among businesses with top-tier credit, those using funds to launch products, start projects, or scale operations saw average growth boosts of **70–95 percentage points**. - Examples included: - MyPark used financing to deploy additional revenue-generating machines. - Xirsys expanded server infrastructure into China, India, and Japan, more than doubling annual revenue. ## Expanding Access Through Embedded Finance - The World Bank estimates a **$5.7 trillion** funding gap for SMBs in developing economies. - Platforms that already manage payments or business operations can use transaction data to make proactive financing offers. - This model broadens access beyond traditional credit scoring and may encourage owners to pursue investments they would otherwise avoid. - Marketplaces and software platforms are positioned to become important channels for closing the global SMB funding gap. Stripe’s research supports using embedded, data-based financing to provide faster access to capital, particularly for small businesses and owners pursuing concrete expansion plans. However, financing remains subject to approval and may take the form of loans or merchant cash advances depending on the market.

Read original

Causal Inference Data Science Fintech Experiment Design+3

tossNov 19, 2025Original article

Beyond LTV: MTV (opens in new tab)

Toss has developed MTVi (Mid-term Value - incremental) to quantify the financial impact of specific services within its platform, moving beyond the limitations of traditional LifeTime Value (LTV). By focusing on the incremental value generated over a one-year period, the metric allows the company to justify services that may lose money individually but drive significant ecosystem-wide growth. This framework provides a data-driven standard for prioritizing features and setting marketing budgets based on actual financial contributions. ### Limitations of Traditional LTV * **Time Horizon Mismatch:** Traditional LTV projects value over 3 to 5 years, which is too slow for Toss’s rapid iteration cycles and fails to reflect the immediate impact of service improvements. * **Investment Recovery Gaps:** Standard LTV models often benchmark marketing costs (CAC) against long-term projections, making it difficult to evaluate the efficiency of short-term experiments. * **Lack of Incrementality:** LTV measures average user value but cannot isolate the specific "extra" value created by a single service, making it impossible to distinguish between a service's impact and natural user growth. ### Defining MTVi and DID Methodology * **Incremental Focus:** MTVi is defined as the net financial value generated over one year specifically because a user experienced a new service, rather than just the average revenue of a user. * **Quasi-Experimental Design:** Since A/B testing every service combination is impossible, Toss uses the Difference-in-Difference (DID) method to compare "Newly Activated Users" (NAU) against "Never" users. * **Segment-Based Analysis:** To prevent bias—such as highly active users naturally gravitating toward more services—Toss segments users by age and historical activity (e.g., app open frequency) to ensure "apples-to-apples" comparisons within identical cohorts. ### Organizational Impact and Strategy * **Unified Decision Metric:** MTVi provides a "common language" for different product teams (silos), allowing them to compare the value of disparate services—like pedometers versus remittances—on a single financial scale. * **Efficiency Benchmarking:** The metric establishes a hard ceiling for investment; for example, Customer Acquisition Cost (CAC) is strictly managed so it does not exceed the calculated MTVi. * **Platform-Wide Valuation:** By calculating both direct revenue and indirect spillover effects, Toss can prove the financial viability of "loss-leader" services that provide user benefits but increase overall app engagement and cross-service usage. For organizations operating complex multi-service platforms, adopting an incremental value metric like MTVi is essential for moving beyond isolated P&L statements. Data teams should prioritize quasi-experimental methods like DID and rigorous user segmentation to accurately map how individual features influence the broader financial health of the ecosystem.

Causal Inference Data Analytics Difference In Difference Unit Economics+4

discordFeb 27, 20253 min readCurated summary

Measuring Product Impact Without A/B Testing: How Discord Used the Synthetic Control Method for Voice Messages

Discord used the Synthetic Control Method to measure the impact of Voice Messages when network effects made traditional A/B testing unreliable. Because users’ behavior is interconnected, randomizing individuals could contaminate treatment and control groups, while country-level comparisons could introduce geographic bias. Synthetic controls offered a stronger alternative by constructing a weighted “synthetic” comparison region from multiple untreated countries. ## Why Traditional A/B Testing Was Difficult - Discord launched Voice Messages in 2023 for text channels, DMs, and Group DMs on mobile. - The feature inherently involves networks: one user sends a message and another receives it. - Network effects violate the assumption that treatment and control users behave independently, known as SUTVA. - Randomizing entire networks would be ideal, but Discord’s testing platform did not support cluster randomization. - User-level A/B testing risked cross-group interactions. - Country-level testing could reduce network contamination, but comparing countries directly would conflate the treatment with differences in language, culture, history, and user behavior. ## How Synthetic Controls Work - Synthetic controls compare one treated unit, such as Brazil, with a weighted combination of untreated units. - Instead of comparing Brazil only with Argentina, Discord might construct a synthetic Brazil from: - 50% Argentina - 30% Uruguay - 20% Chile - The weighted combination is designed to better reproduce the treated country’s pre-treatment outcomes. - This approach addresses omitted-variable bias more effectively than selecting a single “similar” control country. - It also produces a result that may be more representative than learning only how users in one specific country respond. ## Benefits and Evaluation - Synthetic controls can account for both observable and unobservable differences between regions. - They require: - Outcome data for the treated unit before and after treatment - Data from multiple untreated control units over the same periods - An analytical library, such as `Synth` in R or `SyntheticControlMethods` in Python - Discord evaluates the fit using Mean Squared Prediction Error (MSPE). - A close pre-treatment fit indicates that the synthetic control is a credible counterfactual. - A substantial increase in MSPE after rollout suggests that the feature changed outcomes in the treated region. - Additional placebo checks can test whether the method tracks outcomes accurately during periods without an intervention. Synthetic controls are a practical choice when network effects prevent conventional experimentation. For geographically distributed products like Discord, constructing a weighted counterfactual from multiple untreated regions can provide a more credible and generalizable estimate than either user-level A/B tests or simple geo-tests.

Read original

Causal Inference Python Ab Testing Statistical Analysis+3