Better Experiments with LLM Evals — A funnel, not a fork | Spotify Engineering

LLM evaluations and A/B experiments serve different purposes and should work as a funnel rather than alternatives. Evals verify whether a change meets qualitative standards and filter out weak ideas, while experiments validate whether it improves real user and business outcomes. Their greatest value comes from a feedback loop that continuously calibrates eval scores against online results.

Evals as Verification, Experiments as Validation

• LLM judges can assess relevance, coherence, tone, intent alignment, and other qualities at scale.
• Evals identify promising candidates before they consume experiment capacity.
• Experiments determine whether users actually respond better to the change.
• An eval can uncover unexpected failure patterns, guide product fixes, and later verify that violations have decreased.
• It cannot establish whether those fixes improve long-term outcomes such as engagement, trust, or retention.

The Importance of Guardrails

• Evals typically measure selected quality dimensions, not every possible business risk.
• Spotify reports that roughly 42% of launched experiments are rolled back because of regressions in secondary metrics such as crashes, session length, or retention.
• Online experiments expose production effects and monitor guardrail metrics that offline evals may miss.
• Although only about 12% of Spotify’s A/B tests produce a shipped positive result, approximately 64% generate useful learning.

Calibrating Evals Against Outcomes

• Evals are proxies, so their scores must be validated against the outcomes they are intended to predict.
• LLM judges introduce an additional calibration layer on top of conventional metrics such as precision, recall, and ranking scores.
• Judges can reward superficial patterns or miss improvements that matter in real-world use. For example, Qodo’s coding evals did not detect improvements in Anthropic’s Opus 4.5 on longer tasks.
• Both judges and quantitative metrics can drift, making offline-to-online comparison essential.
• Continuous calibration clarifies when evals are reliable verification tools and whether they may eventually become strong enough to support validation.

Closing the Evaluation Loop

• Run evals early and frequently to select the strongest treatments.
• Use experiments to confirm user impact, assess business outcomes, and detect unintended regressions.
• Apply the same LLM evals to A/B test data to compare qualitative scores with actual user results.
• A mismatch between eval scores and experiment outcomes is valuable diagnostic information:
  • Improved scores and improved outcomes validate the judge.
  • Improved scores without better outcomes show that the judge is measuring something that may not drive value.
• Use lightweight experiments for iteration and learning, and more rigorous tests for launch decisions.

The practical recommendation is to treat LLM evals as an upstream filter and a downstream calibration tool—not a replacement for experimentation. This funnel improves test efficiency while making both the judges and product decisions more trustworthy over time.