spotifyJun 3, 2026

Encoding Your Domain Expert: The Context Layer Behind Spotify's Data Assistant | Spotify Engineering (opens in new tab)

Spotify’s data assistant, Vedder, relies less on model size than on carefully curated domain context. With more than 70,000 datasets, schemas alone cannot capture business definitions, data quality issues, or preferred query patterns. Spotify’s solution is a cluster-based context layer owned by domain experts, making AI-generated SQL more reliable, transparent, and maintainable. ## Why Schemas Alone Are Not Enough - Spotify has petabytes of data across more than 70,000 datasets, making it impossible to provide an LLM with the entire warehouse. - Even large context windows cannot represent all available schemas effectively. - Schema types and column names omit critical meaning, such as: - Which values represent test or legacy data - What “active user” means in a particular domain - Which tables or columns are authoritative - Without this context, an AI assistant may confidently choose the wrong dataset. ## Spotify’s Data Agent - Users ask questions in natural language, and the agent: - Selects the relevant context - Generates SQL - Executes it against the warehouse - Returns the answer, query, and sources - It uses a ReAct loop to reason, call tools, inspect results, and revise its approach. - Users can see how an answer was produced rather than receiving an opaque result. - The assistant is available through: - Slack - An MCP server for IDEs and AI tools - A dedicated web interface - Since August 2025, it has supported more than 2,100 users, 13,000 conversations, and 60,000 messages across 177 domain clusters. ## The Cluster Model Spotify organizes data domains into “clusters,” each owned by a named team of experts. A cluster contains: - **Datasets** - Relevant warehouse tables with schemas and profiling - Column cardinality, common values, and partition information - Details that help the model construct accurate filters and queries - **Pairs** - Expert-approved natural-language questions paired with SQL - Examples of both query patterns and domain semantics - **Docs** - Business terminology and definitions - Known data pitfalls - Guidance about which columns to use or avoid Clusters can represent organizations, initiatives, or specialized areas of interest. Domain experts decide what belongs in each cluster and which examples best represent correct practice. ## Why Human Curation Matters - Spotify considered automatically generating training pairs from historical query logs. - That approach produced unreliable results because query history contains: - Exploratory analysis - Debugging queries - One-off investigations - Incorrect table choices - Technically valid but misleading patterns - Cluster curators accepted only 12.5% of the proposed question-SQL pairs. - Experts therefore determine what is canonical and trustworthy, while the model uses that curated knowledge to answer more users. - The goal is not to replace data specialists, but to scale their judgment and expertise. ## Keeping Context Current - Data models and business logic change continuously. - Cluster health scores monitor signals such as: - Underlying data quality - Whether curated SQL still works after schema changes - Coverage of users’ real questions - Reproducibility of generated SQL - Renamed columns or deprecated tables can immediately reduce the validity of existing examples. - Cluster owners use health dashboards and recommended actions to prioritize maintenance. ## Learning from Every Conversation - Vedder records conversations, queries, answers, generated SQL, and user feedback. - Cluster owners use this information to identify missing documentation, weak examples, and emerging needs. - Each approved example or clarified definition improves future answers. - The system treats context as an ongoing product that requires ownership and maintenance, not a one-time upload of metadata. Spotify’s approach suggests that trustworthy enterprise AI depends on a maintained context layer: curated datasets, expert-approved examples, clear documentation, and continuous feedback. The model supplies reasoning and automation, but domain experts remain responsible for defining what the data means.

llmaimodel-context-protocolslack+4

spotifyMay 18, 2026

When Can LLMs Replace Humans in A/B Tests? | Spotify Engineering (opens in new tab)

LLMs can approximate human outcomes in A/B tests only when strong assumptions hold; unlike randomized user experiments, those assumptions are not guaranteed by design. In the Upworthy headline dataset, raw GPT-4o-mini predictions recovered just 39% of the human treatment effect, but appropriate calibration and repeated sampling substantially improved accuracy. However, the assumptions are hardest to justify for genuinely new products or interventions—the cases where replacing human tests would be most valuable. ## Raw LLM Predictions Underestimate Treatment Effects - Researchers used the Upworthy Research Archive, containing click-through rates from thousands of headline experiments. - GPT-4o-mini predicted click-through rates for treatment and control headlines. - Treating those predictions as human outcomes recovered only 39% of the observed human treatment effect. - The error was systematic rather than random: LLM predictions attenuated effects toward zero. - This could lead companies to underestimate product improvements and make poor shipping decisions. ## Conditions for Valid LLM Surrogates Two assumptions are required: - **Surrogacy:** LLM predictions must capture everything about a treatment that affects human behavior. Once predictions and relevant pre-treatment characteristics are accounted for, treatment assignment should provide no additional information about the human outcome. - **Comparability:** The relationship between LLM predictions and human behavior—the calibration function—must remain stable between historical experiments and the new experiment. - If either condition fails, more LLM samples will not solve the problem; the procedure estimates an effect on the model rather than the effect on users. ## Calibration Methods Matter - Linear calibration with ordinary least squares failed a falsification test, producing estimates 3.8 standard errors away from the human benchmark. - Random forests and gradient-boosted trees performed better because they could model nonlinear relationships between LLM predictions and human click behavior. - Repeatedly sampling the LLM and averaging its outputs reduces temperature-driven noise, lowering bias toward zero and reducing variance. ## Limits for New Interventions - Historical data can partially test surrogacy and comparability, but neither assumption can be verified for a treatment never previously tested. - Trust declines as a new treatment becomes more different from past experiments. - The Upworthy setting is unusually favorable: binary outcomes, text-only treatments, similar headline variants, and an LLM trained on extensive textual data. - These assumptions are much less plausible for changes to layouts, recommendation algorithms, pricing, or entirely new product concepts. Human A/B tests remain essential for genuine product innovation. LLM-based testing is most defensible for incremental changes that resemble well-understood historical treatments, with human experiments used to validate the approach and calibrate its predictions.

llmmachine-learningab-testingcausal-inference+4

spotifyFeb 19, 2026

Background Coding Agents: Predictable Results Through Strong Feedback Loops (Honk, Part 3) | Spotify Engineering (opens in new tab)

Spotify argues that unsupervised coding agents become reliable only when surrounded by strong, automated feedback loops. Its “Honk” system uses component-specific verifiers, mandatory pre-PR checks, and an LLM judge to catch build failures, test failures, scope creep, and functionally incorrect changes. The conclusion is that constrained, sandboxed agents with rich verification are more predictable than flexible agents operating independently. ## Failure Modes at Scale - Agents may fail to produce a pull request, which is inconvenient but usually manageable. - They may produce PRs that fail CI, leaving engineers to repair incomplete work. - Most seriously, they may produce PRs that pass CI but are functionally wrong and potentially reach production. - These failures are more likely when components lack tests, agents modify code beyond the prompt, or agents cannot correctly run builds and tests. - Reviewing invalid or nonsensical PRs can become a significant engineering time sink. ## Verification Loops - Honk uses independent verifiers that provide incremental feedback while the agent works. - Verifiers activate automatically based on the repository contents; for example, a Maven verifier runs when a root-level `pom.xml` is present. - The agent sees an abstract MCP tool rather than the implementation details of Maven, test runners, or build systems. - Verifiers handle formatting, compilation, testing, and output parsing, returning concise error messages instead of consuming the agent’s context with raw logs. - All applicable verifiers run before a PR is opened. In Claude Code, this is enforced with a stop hook. - If verification fails, the PR is blocked and the user receives an error. ## An LLM as a Judge - Deterministic checks cannot detect every problem, especially when an agent makes unnecessary refactors or disables flaky tests. - Honk therefore sends the original prompt and proposed diff to a separate LLM judge. - The judge runs after the regular verifiers and can veto changes that exceed the requested scope. - Across thousands of sessions, the judge rejects roughly one quarter of proposed changes. - Agents successfully correct about half of the vetoed changes. - Spotify has not yet built formal evaluations for the judge, but observed that scope violations are its most common reason for rejection. ## Constrained Agents and Sandboxing - The agent has limited responsibilities: inspect the relevant code, edit files, and invoke verification tools. - Surrounding infrastructure handles prompt creation, pushing code, and user communication through systems such as Slack. - Restricting the agent’s capabilities improves predictability and provides security benefits. - Agents run in heavily sandboxed containers with limited permissions, few installed binaries, and almost no access to surrounding systems. - Spotify reports that agents solve increasingly complex tasks reliably when these feedback loops are present, but often produce unusable code without them. ## Future Expansion - Spotify plans to support more hardware and operating systems. - Current verifiers run only on Linux x86, limiting support for systems that require macOS, such as iOS applications, or ARM64 environments. - The company also intends to integrate Honk more deeply with existing CI/CD pipelines. The practical recommendation is to treat autonomous coding as an infrastructure and verification problem, not merely a prompting problem: keep agents narrowly scoped, isolate them securely, and require layered automated checks before accepting their changes.

llmai-agentmodel-context-protocolclaude-code+4

spotifyFeb 19, 2026

Inside the Archive: The Tech Behind Your 2025 Wrapped Highlights | Spotify Engineering (opens in new tab)

Spotify’s 2025 Wrapped Archive identified up to five remarkable listening days for each eligible user and turned them into personalized, LLM-generated stories. A distributed pipeline, carefully designed prompts, model distillation, and massive-scale pre-generation made it possible to create roughly 1.4 billion reports before launch. The system prioritized factual grounding, creative consistency, safety, and reliable parallel storage. ## Identifying Remarkable Listening Days - Spotify used a priority-ordered set of heuristics to evaluate each user’s full year of listening. - Straightforward categories included: - Biggest Music Listening Day - Biggest Podcast Listening Day - Biggest Discovery Day, based on first-time artists - Biggest Top Artist Day - Biggest Top Genre Day - More nuanced categories detected: - Nostalgic listening and throwback-heavy sessions - Unusual listening patterns that differed from a user’s typical taste - Contextual dates such as birthdays and New Year’s Day - Candidate days were ranked by narrative potential and statistical strength, reducing hundreds of millions of events to as many as five standout days per user. - A distributed data pipeline aggregated the results and stored listening data in object storage. - Messaging queues then moved each user’s data asynchronously into report generation. ## Prompt Engineering for Reliable Stories - Spotify spent more than three months iterating on prompts and evaluating edge cases. - The system prompt established: - Traceability to real listening behavior - A witty, sincere, and quietly playful tone - Safety constraints excluding references to drugs, alcohol, sex, violence, and offensive language - User prompts supplied: - Detailed daily listening logs - Precomputed statistics, since LLMs are unreliable at arithmetic - Overall Wrapped data - The remarkable-day category - Previously generated reports to reduce repetition - The user’s country for appropriate spelling and vocabulary - Outputs were improved through prototype comparisons, LLM-based judging, human review, and feedback from creative, technical, and safety teams. ## Distilling the Model for Scale - Larger frontier models produced strong results during prototyping but were too expensive for more than a billion generations. - Spotify generated high-quality reference outputs and curated them into a reviewed “gold” dataset. - A smaller, faster production model was fine-tuned on that dataset. - Direct Preference Optimization (DPO), based on curated human A/B evaluations, further aligned the smaller model with the preferred output style. - The resulting model achieved preference performance comparable to the larger baseline. ## Generating 1.4 Billion Reports - Approximately 350 million users were eligible, with up to five reports each. - Spotify pre-generated about 1.4 billion reports before Wrapped launch. - The system sustained thousands of model requests per second over several days. - After remarkable days were computed, snapshots were published to a pub/sub queue. - Reports were generated sequentially per user so earlier reports could inform later ones and prevent repetition. - Real-time dashboards tracked throughput, reliability, errors, and projected completion time. - The generation engine ran continuously for four days, followed by checks for missing reports, inconsistencies, and necessary re-generation. ## Designing Storage for Concurrent Writes - Completed reports were stored in a distributed, column-oriented key-value database optimized for high-throughput writes. - Each user occupied a single row, with separate columns representing completed remarkable days. - Instead of maintaining a serialized list—which could cause race conditions during read-modify-write operations—each date received its own column qualifier in `YYYYMMDD` format. - Independent reports could therefore be written concurrently to separate cells without locks or coordination. - Report content was written first, followed by lightweight metadata marking the report complete. - This ordering prevented the system from exposing a completion marker before the underlying report was safely stored. ## Practical Conclusion Building Wrapped Archive required treating creative AI generation as a large-scale production system: ground outputs in structured data, use smaller specialized models when volume demands it, evaluate continuously, and design storage schemas that make concurrency safe by default.

llmdatabase-designprompt-engineeringhigh-throughput+4