Measure Less to Learn More: Using Fewer, Higher-quality Metrics to Capture What Matters (opens in new tab)
Discord argues that experiments should measure fewer, higher-quality metrics rather than automatically collecting every potentially useful signal. Large metric sets increase compute and cognitive costs while creating a tradeoff between false positives and missed real effects. Multiple-testing corrections such as Benjamini–Hochberg reduce false discoveries but also lower recall, so the most effective solution is selecting metrics that represent distinct, important concepts. ## The Cost of Measuring Too Much - Discord’s “Default Metric List” gradually expanded as teams added metrics and rarely removed them. - More metrics create: - Higher compute costs - More difficult experiment readouts - Increased risk of false positives - With 100 metrics and an uncorrected significance threshold of 0.05, roughly five metrics may appear significant purely by chance. - Correcting for multiple comparisons reduces false alarms but makes genuine changes harder to detect. ## The Multiple Comparisons Problem - Discord uses the Benjamini–Hochberg (BH) procedure to control the false discovery rate at 5%. - BH ranks p-values and compares each one with a rank-specific threshold: `i × α / n` where `i` is the metric’s rank, `α` is 0.05, and `n` is the total number of metrics. - A metric with an unadjusted p-value of 0.038 might be significant without correction but fail after BH adjustment. - BH treats all metrics equally because it has no information about which ones are more likely to reflect a real effect. - The resulting tradeoff is: - Fewer false alarms - Lower recall for real changes - The article notes that Bayesian methods could potentially incorporate prior knowledge, but Discord’s default system is frequentist. ## Simulation Results - Discord simulated 50,000 experiments containing: - Twenty null metrics generated from `N(0, 1)` - One metric with a real effect centered at `z = 2.8` - The simulations tested how metric count affects: - Experiment-level false alarm rates - Recall of the metric with the known effect - Without correction, false alarm rates rose sharply as more metrics were added—approximately from 23% with five metrics to 93% with 50. - BH kept false alarm rates near 5%, but recall declined as the metric pool grew, falling from roughly 60% to 30% across the same range. - These results demonstrate that adding metrics makes statistical correction stricter and makes genuine effects harder to identify. ## Fewer, Higher-Quality Metrics - Reducing the metrics automatically included in experiments improves the balance between false alarms and recall. - Metrics should be selected for quality and conceptual distinctness rather than added “just to be safe.” - The article’s central conclusion is that no sophisticated statistical method eliminates the underlying tradeoff created by excessive measurement. Teams should maintain a focused default metric set, regularly remove low-value or redundant metrics, and reserve specialized metrics for experiments where they are genuinely relevant.
