Ab Testing | Techlist.io

githubJun 12, 20263 min readCurated summary

How we made GitHub Copilot CLI more selective about delegation

GitHub improved Copilot CLI by making subagent delegation more selective rather than treating delegation as inherently beneficial. The new orchestration policy keeps narrow tasks with the main agent, delegates broad or independent work, and encourages parallel execution instead of waiting. After full production rollout, it reduced tool failures by 23% and improved high-percentile wait times without reducing quality. ## The Cost of Unnecessary Delegation - Subagents help with complex investigations, large repositories, and parallel work, but every handoff adds tool calls, coordination, and latency. - Copilot sometimes delegated simple, well-scoped tasks that the main agent could complete directly. - Common problems included: - Repeated or overlapping repository searches. - Subagents re-discovering context already available to the main agent. - Sequential delegation that left the main agent idle. - Stale paths, incorrect relative paths, and workspace mismatches. - The result was slower execution and more tool failures for tasks that should have required only a few steps. ## How the Problem Was Identified - GitHub used LLMs to analyze complete agent trajectories rather than manually reviewing sessions. - The analysis found that delegation was frequently used for narrow, obvious, or fully described tasks. - This led to a clear target: - Keep focused discovery-and-edit work with the main agent. - Reserve subagents for broad exploration, cross-cutting tasks, or genuinely independent work. ## A More Selective Orchestration Policy - Copilot now starts with the narrowest effective workflow: - Find and read the relevant file. - Make the targeted change. - Verify the result. - Delegation becomes appropriate when additional context, uncertainty, or parallel execution creates real value. - Subagents are treated as a parallelism mechanism, not a reason for the main agent to pause. - Handoffs should clearly specify: - The user’s request. - What the main agent already knows. - Which work the subagent owns. - What result the subagent should return. ## Evaluation and Production Results - GitHub tested the change with generated regression cases and existing benchmarks before rollout. - Staff and public A/B tests measured reliability, responsiveness, subagent workload, and quality. - Production results showed: - 23% fewer tool failures per session. - 27% fewer search-tool failures. - 18% fewer edit-tool failures. - 5% lower P95 wait time. - 3% lower P75 wait time. - No quality regression. - The improvements came mainly from avoiding unnecessary subagent paths and reducing orchestration overhead, not from making individual model calls faster. Copilot CLI users can access the improvement by running `/update` and upgrading to version 1.0.42 or later. The broader recommendation is to delegate selectively: use the main agent for focused tasks and subagents only when independent context or parallel work provides meaningful leverage.

Read original

Ab Testing Large Language Models Github Copilot Cli Orchestration+2

spotifyMay 18, 20263 min readCurated summary

When Can LLMs Replace Humans in A/B Tests? | Spotify Engineering

LLMs can approximate human outcomes in A/B tests only when strong assumptions hold; unlike randomized user experiments, those assumptions are not guaranteed by design. In the Upworthy headline dataset, raw GPT-4o-mini predictions recovered just 39% of the human treatment effect, but appropriate calibration and repeated sampling substantially improved accuracy. However, the assumptions are hardest to justify for genuinely new products or interventions—the cases where replacing human tests would be most valuable. ## Raw LLM Predictions Underestimate Treatment Effects - Researchers used the Upworthy Research Archive, containing click-through rates from thousands of headline experiments. - GPT-4o-mini predicted click-through rates for treatment and control headlines. - Treating those predictions as human outcomes recovered only 39% of the observed human treatment effect. - The error was systematic rather than random: LLM predictions attenuated effects toward zero. - This could lead companies to underestimate product improvements and make poor shipping decisions. ## Conditions for Valid LLM Surrogates Two assumptions are required: - **Surrogacy:** LLM predictions must capture everything about a treatment that affects human behavior. Once predictions and relevant pre-treatment characteristics are accounted for, treatment assignment should provide no additional information about the human outcome. - **Comparability:** The relationship between LLM predictions and human behavior—the calibration function—must remain stable between historical experiments and the new experiment. - If either condition fails, more LLM samples will not solve the problem; the procedure estimates an effect on the model rather than the effect on users. ## Calibration Methods Matter - Linear calibration with ordinary least squares failed a falsification test, producing estimates 3.8 standard errors away from the human benchmark. - Random forests and gradient-boosted trees performed better because they could model nonlinear relationships between LLM predictions and human click behavior. - Repeatedly sampling the LLM and averaging its outputs reduces temperature-driven noise, lowering bias toward zero and reducing variance. ## Limits for New Interventions - Historical data can partially test surrogacy and comparability, but neither assumption can be verified for a treatment never previously tested. - Trust declines as a new treatment becomes more different from past experiments. - The Upworthy setting is unusually favorable: binary outcomes, text-only treatments, similar headline variants, and an LLM trained on extensive textual data. - These assumptions are much less plausible for changes to layouts, recommendation algorithms, pricing, or entirely new product concepts. Human A/B tests remain essential for genuine product innovation. LLM-based testing is most defensible for incremental changes that resemble well-understood historical treatments, with human experiments used to validate the approach and calibrate its predictions.

Read original

Ab Testing Large Language Models Machine Learning Causal Inference+4

metaApr 9, 20263 min readCurated summary

Escaping the Fork: How Meta Modernized WebRTC Across 50+ Use Cases

Meta escaped the “forking trap” by replacing its divergent WebRTC fork with a modular architecture based on the latest upstream release. The system builds legacy and current WebRTC versions side by side, enabling runtime A/B testing across more than 50 use cases before rollout. This improved performance, binary size, and security while establishing a repeatable process for continuous upstream upgrades. ## Why the WebRTC Fork Became a Problem - Meta’s RTC stack supports Messenger, Instagram video calls, Cloud Gaming, and Meta Quest casting. - Internal optimizations and bug fixes gradually caused its WebRTC fork to diverge from upstream. - As the fork accumulated custom changes, merging community improvements became increasingly expensive and risky. - A one-time upgrade was impractical because WebRTC serves billions of users across diverse devices and environments. ## Requirements for a Sustainable Upgrade Strategy - Meta needed to: - Run legacy and upstream-based WebRTC implementations simultaneously. - Dynamically assign users to either version for safe A/B testing. - Statically link both versions into the same application. - Maintain custom patches in a monorepo without repeatedly rebuilding the migration process. - Standard patch-file workflows were considered difficult to scale for Meta’s large codebase. ## Shim Layer and Dual-Stack Architecture - A shim library was placed between application code and WebRTC. - Applications call a unified, version-neutral API rather than calling either WebRTC implementation directly. - A runtime “flavor” configuration routes each call to either the legacy or latest implementation. - Shimming at the lowest practical layer avoided duplicating the higher-level call orchestration library: - Full duplication would have added about 38 MB uncompressed. - The shim-based design added roughly 5 MB, an 87% reduction. ## Resolving C++ Symbol Collisions - Linking two WebRTC copies normally violates the C++ One Definition Rule and creates thousands of duplicate symbols. - Meta automated namespace rewriting: - `webrtc::` in the current version became `webrtc_latest::`. - The legacy version became `webrtc_legacy::`. - Global functions, variables, and classes outside namespaces were moved into namespaces where possible or assigned flavor-specific names. - Macro conflicts, including `RTC_CHECK` and `RTC_LOG`, were addressed by: - Removing unnecessary includes. - Renaming infrequently used macros. - Sharing modules such as `rtc_base` between versions to reduce duplication and shimming work. ## Preserving Backward Compatibility - Renaming symbols could have broken existing call sites, especially code built for only one WebRTC flavor. - An initial solution forward-declared every required symbol, but this created a large and fragile maintenance burden. - The improved approach used C++ `using` declarations to bulk-import a flavor namespace into the familiar `webrtc::` namespace. - This preserved existing source-level APIs without adding binary overhead, while allowing Meta to migrate selected call sites incrementally. ## Runtime Flavor Dispatch - Shim adapters and converters must instantiate objects from either the legacy or current namespace. - A template-based helper library keeps shared adapter logic in one place. - Template specializations handle version-specific behavior. - A global flavor enum, initialized during application startup, determines which WebRTC implementation is used. - The design also supports single-flavor builds during the transition. Meta’s approach demonstrates that large internal modifications do not have to require a permanent fork. A low-level shim, automated renamespacing, compatibility imports, and template-based dispatch provide a practical foundation for continuously rebasing custom functionality onto upstream WebRTC while safely validating each release through A/B testing.

Read original

Ab Testing Monorepo C++Webrtc+4

lineMar 18, 20264 min readCurated summary

Large-scale iOS Settings System Unraveled through AttributedString Structure

LINE’s “Service Configuration” system lets teams deploy features dynamically without waiting for LINE’s two-week app release cycle. As the iOS app grew to roughly 700 configuration keys across 60 modules, its monolithic design created dependency, usability, concurrency, testing, and QA problems. The article argues that the original design was reasonable at small scale but needed to evolve, beginning with lessons from Foundation’s type-safe `AttributedString` design. ## What Service Configuration Provides - Service operators modify values through an administration page. - The server notifies LINE clients, which fetch updated values. - Values are selected based on factors such as: - User region - Device - OS version - The system supports: - Feature flags - Rollbacks - A/B tests - Error-reporting sample rates - UI behavior policies - Configuration is delivered as a string-to-string dictionary, for example: - `"function.media.image_medium": "1280,70"` - `"function.media.message.flow.v2.image": "Y"` ## Problems Caused by the Monolithic Design The original implementation required every key to be declared in one roughly 7,000-line file. Although this was simple initially, growth in teams and modules made the structure increasingly costly. ### Circular Dependencies and Weak Typing - Configuration values were exposed as raw strings because the configuration module could not depend on feature-specific modules. - For example, `"1280,70"` represented image dimensions and JPEG quality, but callers had to parse it into an `ImageTransferQuality` value themselves. - Defining `ImageTransferQuality` in the configuration module avoided repeated parsing but polluted unrelated modules with photo-specific types. - Defining it in the photo module preserved separation of concerns but created an impossible reverse dependency. ### Incomplete and Confusing Abstractions - Developers had to understand server-specific encoding rules and implementation details. - Boolean values were sent as `"Y"` and `"N"`, requiring a custom `decodeBoolIfPresent(forKey:)` method. - The custom decoder’s name resembled Swift’s standard decoding API, making incorrect implementations easy to write and review. - Decoding failures could silently fall back to defaults, making the underlying problem difficult to diagnose. - The same default value often had to be declared three times: - A property-group default - A decoding fallback - A global `defaultConfiguration` entry - These duplicated defaults served subtly different purposes, although the distinctions were generally unnecessary. ### Lack of Thread Safety - Configuration groups were lazily decoded and replaced when new server values arrived. - Multiple services could read configuration values concurrently on different threads. - This caused use-after-free crashes— reportedly hundreds per day—leading to bug tickets and hotfix releases. - As the number of services and concurrent operations increased, this became a systemic issue rather than an occasional edge case. ### No Built-in Debug Overrides - QA frequently needed to temporarily change configuration values. - Because the system had no override mechanism, each feature required custom: - Persistent storage - Debug-menu UI - Value-display text - Implementing this repeatedly required edits across several files and modules. ### Fragmented Test Doubles - Since `LineConfigurationManager` was a singleton, modules created narrow protocols and custom mocks for the settings they used. - This resulted in dozens of duplicated protocols and test doubles. - These had to be updated alongside configuration keys and could fall out of sync. - Differences between mocks and production behavior could allow bugs to escape tests or create false failures. ## Looking to Established Designs The team first distilled the required properties of a replacement: - Type-safe access to a large number of key-value pairs - Independent key definitions by each module - Safe behavior under concurrency They identified Foundation’s `AttributedString` as a useful precedent because it manages many typed attributes while allowing UIKit, AppKit, SwiftUI, and other frameworks to define their own attributes independently. The article presents this as the starting point for redesigning Service Configuration around a more modular and type-safe architecture.

Read original

Ab Testing Database Design Swift Type Safety+4

spotifyFeb 19, 20263 min readCurated summary

Why We Use Separate Tech Stacks for Personalization and Experimentation | Spotify Engineering

Personalization and experimentation overlap, especially with contextual bandits, but they serve different purposes. Personalization chooses the best experience for each user, while experimentation evaluates whether a system or product change improves outcomes overall. Spotify therefore keeps personalization in its ML stack and uses its experimentation platform, Confidence, to measure the impact of those systems. ## What Personalization Does - Personalization adapts products to individual users instead of optimizing for an “average” user. - Spotify uses it for: - Personalized playlists and discovery recommendations - Search results based on listening and search history - Home-screen shortcuts tailored to user behavior - Recommendation systems use models ranging from regression and random forests to neural networks, LLMs, and reinforcement learning. - These systems combine user characteristics, historical behavior, and real-time context to select recommendations and improve them from user responses. ## Where Experimentation and Personalization Overlap - Traditional A/B tests randomly assign users to variants and identify the best option on average. - Multi-armed bandits adapt traffic allocation, sending more users to better-performing treatments. - Contextual bandits use user features to select different treatments for different users. - This creates a conceptual transition: - A/B tests seek the best average treatment. - Multi-armed bandits efficiently find the best average treatment. - Contextual bandits seek the best treatment for each user or context. - Once contextual information determines treatment assignment, the goal is no longer simply measuring one variant’s average effect. - The relevant comparison becomes the value of the personalization system versus a static treatment or an earlier personalization system. ## Why Contextual Bandits Are Not Experimentation Platforms - A contextual bandit can personalize a checkout button based on factors such as cart contents, age, and location. - The bandit is itself a product feature or recommendation system, not the experiment used to evaluate it. - Teams still need experiments to compare different versions of the bandit and determine whether personalization improves user outcomes. - Measuring average or conditional treatment effects may be less important than evaluating the overall benefit of the personalization system. ## Why Spotify Separates the Stacks - Personalization systems require specialized ML infrastructure for: - Training diverse model types - Managing rich feature sets - Serving models with low latency - Collecting real-time data - Computing recommendations at request time - Putting these capabilities into experimentation tools would either greatly expand their scope or limit the sophistication of personalization. - Combining unrelated responsibilities can create technical debt and operational complexity. - Spotify’s ML platform standardizes the development and deployment of personalization systems. - Its experimentation platform, Confidence, evaluates those systems alongside thousands of other product experiments. ## How the Systems Work Together - Personalization algorithms, including contextual bandits, remain in the ML stack. - The experimentation stack treats a personalization system as a treatment to be evaluated. - This separation lets teams improve recommendation logic independently while using controlled experiments to measure its product impact. - It also avoids confusing dependencies that arise when a bandit is both the experiment mechanism and the system being evaluated. Spotify’s approach is to separate building personalized experiences from measuring their effectiveness. Teams should use ML infrastructure for model development and real-time decision-making, then use an experimentation platform to compare personalization systems and quantify their impact.

Read original

Ab Testing Machine Learning Recommendation Systems Reinforcement Learning+4

tossJan 29, 2026Original article

6 Principles to Increase Marketing (opens in new tab)

Toss, a leader in the Korean fintech space, demonstrates that high marketing performance can be achieved without resorting to aggressive or deceptive copy. By analyzing hundreds of A/B tests within their app, they have identified specific UX writing patterns that prioritize user trust while significantly boosting engagement. The core conclusion is that clarity, psychological ease, and guaranteed rewards consistently outperform complex value propositions and exaggerated claims. ### The Power of One Core Message * Focusing on a single, immediate action is more effective than listing multiple service benefits. * In one test, replacing a complex benefit-driven headline with a simple "Take a 10-question test" resulted in a 10x increase in click-through rates (CTR). * Complexity creates friction; users are more likely to engage when they understand exactly what the next step entails without distractions. ### Prioritizing Guaranteed Rewards * Users show a stronger preference for "guaranteed small wins" over "potential big wins." * A campaign promising a "Minimum 100 won" reward saw 20x more exposure than one promising "Up to 1 million won," as large numbers can trigger skepticism or feel unattainable. * Phrases like "You will definitely get 1" outperform "Get as many as you want" because they provide a concrete promise rather than a vague possibility. ### Reducing Cognitive Load Through "Light" Language * The choice of verbs significantly impacts the perceived effort of a task. * Using "Prepare for travel insurance" instead of "Sign up for travel insurance" reduces the psychological burden, as "sign up" implies a long, bureaucratic process. * "Light" verbs make the service feel faster and easier to complete, encouraging immediate action. ### Strategic Information Framing * Clearly defining the nature of information—whether it is a "collection," a "list," or "new"—helps users categorize the value quickly. * Highlighting that a feature is "new" rather than explaining the specific benefits of the feature increased CTR by 6x. * Using terms like "View collection" for loan products provides a sense of organized efficiency that appeals to users looking for consolidated information. ### Specificity in Action and Conditions * Ambiguity leads to hesitation; providing exact numbers (e.g., "4 missions" or "8 blanks") increases conversion rates. * Specifying the number of tasks makes a goal feel attainable and removes the fear of an open-ended time commitment. * Quantifying the effort required (e.g., "takes 3 minutes") allows users to make an instant, friction-less decision to participate. ### Utilizing Intuitive, Everyday Experiences * Copy that mirrors real-life physical actions is more intuitive for users. * Changing a button from "View answer" to "Pick an answer" (accompanied by a stamp emoji) for an OX quiz significantly increased engagement by making the digital action feel more tactile and familiar. * Leveraging common vocabulary ensures that users do not have to "translate" marketing speak into practical reality. To maximize conversion, designers and writers should move away from broad marketing claims and toward radical specificity. By removing ambiguity and promising certain, low-effort outcomes, you can build a more effective and honest user experience.

Ab Testing User Experience Ux Writing Conversion Rate Optimization+3

pinterestDec 10, 20253 min readCurated summary

LLM-Powered Relevance Assessment for Pinterest Search

Pinterest Search uses fine-tuned multilingual LLMs to assess search-result relevance at a much larger scale than human labeling allows. The approach combines five-level relevance classification, stratified query sampling, and paired A/B-test evaluation to detect smaller overall effects and differences across query types. XLM-RoBERTa-large provides a practical balance of accuracy and cost, achieving strong agreement with human judgments while enabling substantially faster labeling. ## Relevance Measurement Challenges - Search relevance measures how well Pins satisfy a user’s query, rather than merely reflecting past engagement. - Human annotations are expensive and limited in volume. - Previous sampling designs could detect only relatively large topline changes, with minimum detectable effects (MDEs) around 1.3%–1.5%. - Limited labels also made it difficult to measure heterogeneous effects across query interests or popularity segments. ## Fine-Tuned LLM Relevance Model - Pinterest defines relevance using five labels: - L5: Highly Relevant - L4: Relevant - L3: Marginally Relevant - L2: Irrelevant - L1: Highly Irrelevant - A cross-encoder model predicts the relevance of each Pin for a query. - Open-source multilingual models are fine-tuned on human-annotated examples using multiclass cross-entropy loss. - Pin representations include: - Titles and descriptions - BLIP-generated image captions - Linked-page titles and descriptions - Board titles where Pins were saved - Highly engaged query tokens associated with the Pin - Models tested included multilingual BERT, T5, mDeBERTa, XLM-RoBERTa, and Llama 3. - The final relevance label is selected from the model’s five output scores using argmax. ## Stratified Query Sampling - Lower LLM labeling costs allow Pinterest to use much larger and more detailed samples. - Queries are stratified using: - A DistilBERT-based query-to-interest model - Query popularity, based on how many users issue each query - Stratification improves representativeness and reduces variance by grouping similar queries. - Pinterest moved from simple random sampling to stratified sampling with optimal allocation across strata. - Most of the MDE improvement came from variance reduction through stratification. - The redesigned process reduced MDEs from approximately 1.3%–1.5% to 0.25% or less. ## LLM-Based A/B-Test Measurement - Pinterest samples paired queries from control and treatment groups. - Pairing controls for differences between queries, which are a major source of relevance variance. - For each query, the top 25 results are retained and labeled by the LLM. - Query-level relevance is measured using sDCG@25, a variant of nDCG that assumes an unlimited supply of highly relevant L5 results. - Results are aggregated into topline experiment metrics. - Heterogeneous effects are analyzed by query popularity and interest categories such as beauty, fashion, and art. - The Benjamini–Hochberg procedure controls the false discovery rate when testing multiple segments. ## Model Choice and Validation - XLM-RoBERTa-large was selected for its balance of quality and efficiency. - On a single A10G GPU, it can label 150,000 rows in about 30 minutes. - Llama 3–8B produced slightly better accuracy but required roughly six times the inference time and cost. - LLM labels matched human labels exactly for 73.7% of Pins. - A total of 91.7% of predictions differed from human ratings by no more than one relevance point. Pinterest’s approach makes relevance evaluation cheaper, faster, and more statistically sensitive. Fine-tuned LLMs paired with stratified sampling are recommended for search experimentation when human labeling cannot provide enough coverage to detect small or heterogeneous ranking effects.

Read original

Ab Testing Large Language Models Natural Language Processing Fine Tuning+4

kakaoNov 19, 2025Original article

Were We Solving the Real Problem (opens in new tab)

The POPM (Product Owner/Product Manager) training course at Kakao focuses on restructuring existing professional knowledge into a cohesive framework for solving real-world business problems. Rather than simply delivering new information, the program emphasizes aligning strategy with execution, transforming "strategy" from a vague concept into a practical set of decision-making criteria. The ultimate goal is to move teams away from a "release-only" mindset toward a cycle of continuous hypothesis verification and learning. ### Strategic Thinking and Metric Modeling * **Strategic Decision Criteria**: Strategy is redefined as the standard for team judgment, utilizing frameworks like MECE, MVP, and priority-setting models to align daily tasks with long-term goals. * **Metrics as Problem-Solving Language**: Key indicators such as Funnel, Retention, Cohort, and LTV are treated not just as data points, but as a language used to define and reveal underlying product issues. * **Context-Based Design**: UX design is approached through "context-based logic" rather than intuition, encouraging teams to ask which specific design fits the current user journey. ### Systematic Experimentation and A/B Testing * **The MASS Framework**: Experiments are designed and evaluated based on being Measurable, Attributable, Sensitive, and having a Short-term cycle. * **Failure Analysis Routines**: The curriculum emphasizes the importance of establishing a routine for interpreting failed experiments, ensuring that every test contributes to the team's institutional knowledge. * **Incremental Testing**: Encourages a culture of "starting small," giving teams the confidence to run experiments without requiring massive resource allocation. ### Building Repeatable Execution Loops * **Metric-Based Retrospectives**: Teams transition from simply finishing a release to a structured loop of "Problem Definition → Hypothesis → Metric → Verification → Retrospective." * **Formalizing Problem Definitions**: Using templates to 명문화 (formally document) the problem, expected behavior, and success metrics ensures that the entire team—not just the PO—understands the "why" behind every task. * **Operational Rhythms**: Teams are adopting fixed weekly or bi-weekly cycles for sharing insights and adjusting priorities, turning data-driven execution into a natural habit. The most critical takeaway for product teams is to constantly ask: "Is the work we are doing right now actually a solution to a defined problem, or are we just busy releasing features?" Success lies in moving beyond the sense of accomplishment from a launch and establishing a repeatable rhythm that validates whether those efforts truly move the needle.

Ab Testing User Experience Product Management Data Analysis+3

discordFeb 27, 20253 min readCurated summary

Measuring Product Impact Without A/B Testing: How Discord Used the Synthetic Control Method for Voice Messages

Discord used the Synthetic Control Method to measure the impact of Voice Messages when network effects made traditional A/B testing unreliable. Because users’ behavior is interconnected, randomizing individuals could contaminate treatment and control groups, while country-level comparisons could introduce geographic bias. Synthetic controls offered a stronger alternative by constructing a weighted “synthetic” comparison region from multiple untreated countries. ## Why Traditional A/B Testing Was Difficult - Discord launched Voice Messages in 2023 for text channels, DMs, and Group DMs on mobile. - The feature inherently involves networks: one user sends a message and another receives it. - Network effects violate the assumption that treatment and control users behave independently, known as SUTVA. - Randomizing entire networks would be ideal, but Discord’s testing platform did not support cluster randomization. - User-level A/B testing risked cross-group interactions. - Country-level testing could reduce network contamination, but comparing countries directly would conflate the treatment with differences in language, culture, history, and user behavior. ## How Synthetic Controls Work - Synthetic controls compare one treated unit, such as Brazil, with a weighted combination of untreated units. - Instead of comparing Brazil only with Argentina, Discord might construct a synthetic Brazil from: - 50% Argentina - 30% Uruguay - 20% Chile - The weighted combination is designed to better reproduce the treated country’s pre-treatment outcomes. - This approach addresses omitted-variable bias more effectively than selecting a single “similar” control country. - It also produces a result that may be more representative than learning only how users in one specific country respond. ## Benefits and Evaluation - Synthetic controls can account for both observable and unobservable differences between regions. - They require: - Outcome data for the treated unit before and after treatment - Data from multiple untreated control units over the same periods - An analytical library, such as `Synth` in R or `SyntheticControlMethods` in Python - Discord evaluates the fit using Mean Squared Prediction Error (MSPE). - A close pre-treatment fit indicates that the synthetic control is a credible counterfactual. - A substantial increase in MSPE after rollout suggests that the feature changed outcomes in the treated region. - Additional placebo checks can test whether the method tracks outcomes accurately during periods without an intervention. Synthetic controls are a practical choice when network effects prevent conventional experimentation. For geographically distributed products like Discord, constructing a weighted counterfactual from multiple untreated regions can provide a more credible and generalizable estimate than either user-level A/B tests or simple geo-tests.

Read original

Ab Testing Python Causal Inference Statistical Analysis+3