LLM-Powered Relevance Assessment for Pinterest Search (opens in new tab)
Pinterest Search uses fine-tuned multilingual LLMs to assess search-result relevance at a much larger scale than human labeling allows. The approach combines five-level relevance classification, stratified query sampling, and paired A/B-test evaluation to detect smaller overall effects and differences across query types. XLM-RoBERTa-large provides a practical balance of accuracy and cost, achieving strong agreement with human judgments while enabling substantially faster labeling. ## Relevance Measurement Challenges - Search relevance measures how well Pins satisfy a user’s query, rather than merely reflecting past engagement. - Human annotations are expensive and limited in volume. - Previous sampling designs could detect only relatively large topline changes, with minimum detectable effects (MDEs) around 1.3%–1.5%. - Limited labels also made it difficult to measure heterogeneous effects across query interests or popularity segments. ## Fine-Tuned LLM Relevance Model - Pinterest defines relevance using five labels: - L5: Highly Relevant - L4: Relevant - L3: Marginally Relevant - L2: Irrelevant - L1: Highly Irrelevant - A cross-encoder model predicts the relevance of each Pin for a query. - Open-source multilingual models are fine-tuned on human-annotated examples using multiclass cross-entropy loss. - Pin representations include: - Titles and descriptions - BLIP-generated image captions - Linked-page titles and descriptions - Board titles where Pins were saved - Highly engaged query tokens associated with the Pin - Models tested included multilingual BERT, T5, mDeBERTa, XLM-RoBERTa, and Llama 3. - The final relevance label is selected from the model’s five output scores using argmax. ## Stratified Query Sampling - Lower LLM labeling costs allow Pinterest to use much larger and more detailed samples. - Queries are stratified using: - A DistilBERT-based query-to-interest model - Query popularity, based on how many users issue each query - Stratification improves representativeness and reduces variance by grouping similar queries. - Pinterest moved from simple random sampling to stratified sampling with optimal allocation across strata. - Most of the MDE improvement came from variance reduction through stratification. - The redesigned process reduced MDEs from approximately 1.3%–1.5% to 0.25% or less. ## LLM-Based A/B-Test Measurement - Pinterest samples paired queries from control and treatment groups. - Pairing controls for differences between queries, which are a major source of relevance variance. - For each query, the top 25 results are retained and labeled by the LLM. - Query-level relevance is measured using sDCG@25, a variant of nDCG that assumes an unlimited supply of highly relevant L5 results. - Results are aggregated into topline experiment metrics. - Heterogeneous effects are analyzed by query popularity and interest categories such as beauty, fashion, and art. - The Benjamini–Hochberg procedure controls the false discovery rate when testing multiple segments. ## Model Choice and Validation - XLM-RoBERTa-large was selected for its balance of quality and efficiency. - On a single A10G GPU, it can label 150,000 rows in about 30 minutes. - Llama 3–8B produced slightly better accuracy but required roughly six times the inference time and cost. - LLM labels matched human labels exactly for 73.7% of Pins. - A total of 91.7% of predictions differed from human ratings by no more than one relevance point. Pinterest’s approach makes relevance evaluation cheaper, faster, and more statistically sensitive. Fine-tuned LLMs paired with stratified sampling are recommended for search experimentation when human labeling cannot provide enough coverage to detect small or heterogeneous ranking effects.