Why We Use Separate Tech Stacks for Personalization and Experimentation | Spotify Engineering

Personalization and experimentation overlap, especially with contextual bandits, but they serve different purposes. Personalization chooses the best experience for each user, while experimentation evaluates whether a system or product change improves outcomes overall. Spotify therefore keeps personalization in its ML stack and uses its experimentation platform, Confidence, to measure the impact of those systems.

What Personalization Does

• Personalization adapts products to individual users instead of optimizing for an “average” user.
• Spotify uses it for:
  • Personalized playlists and discovery recommendations
  • Search results based on listening and search history
  • Home-screen shortcuts tailored to user behavior
• Recommendation systems use models ranging from regression and random forests to neural networks, LLMs, and reinforcement learning.
• These systems combine user characteristics, historical behavior, and real-time context to select recommendations and improve them from user responses.

Where Experimentation and Personalization Overlap

• Traditional A/B tests randomly assign users to variants and identify the best option on average.
• Multi-armed bandits adapt traffic allocation, sending more users to better-performing treatments.
• Contextual bandits use user features to select different treatments for different users.
• This creates a conceptual transition:
  • A/B tests seek the best average treatment.
  • Multi-armed bandits efficiently find the best average treatment.
  • Contextual bandits seek the best treatment for each user or context.
• Once contextual information determines treatment assignment, the goal is no longer simply measuring one variant’s average effect.
• The relevant comparison becomes the value of the personalization system versus a static treatment or an earlier personalization system.

Why Contextual Bandits Are Not Experimentation Platforms

• A contextual bandit can personalize a checkout button based on factors such as cart contents, age, and location.
• The bandit is itself a product feature or recommendation system, not the experiment used to evaluate it.
• Teams still need experiments to compare different versions of the bandit and determine whether personalization improves user outcomes.
• Measuring average or conditional treatment effects may be less important than evaluating the overall benefit of the personalization system.

Why Spotify Separates the Stacks

• Personalization systems require specialized ML infrastructure for:
  • Training diverse model types
  • Managing rich feature sets
  • Serving models with low latency
  • Collecting real-time data
  • Computing recommendations at request time
• Putting these capabilities into experimentation tools would either greatly expand their scope or limit the sophistication of personalization.
• Combining unrelated responsibilities can create technical debt and operational complexity.
• Spotify’s ML platform standardizes the development and deployment of personalization systems.
• Its experimentation platform, Confidence, evaluates those systems alongside thousands of other product experiments.

How the Systems Work Together

• Personalization algorithms, including contextual bandits, remain in the ML stack.
• The experimentation stack treats a personalization system as a treatment to be evaluated.
• This separation lets teams improve recommendation logic independently while using controlled experiments to measure its product impact.
• It also avoids confusing dependencies that arise when a bandit is both the experiment mechanism and the system being evaluated.

Spotify’s approach is to separate building personalized experiences from measuring their effectiveness. Teams should use ML infrastructure for model development and real-time decision-making, then use an experimentation platform to compare personalization systems and quantify their impact.