meta·Jan 14, 2026Original article

Adapting the Facebook Reels RecSys AI Model Based on User Feedback (opens in new tab)

Meta has enhanced the Facebook Reels recommendation engine by shifting focus from traditional engagement signals, like watch time and likes, to direct user feedback. By implementing the User True Interest Survey (UTIS) model, the system now prioritizes content that aligns with genuine user preferences rather than just short-term interactions. This shift has resulted in significant improvements in recommendation relevance, high-quality content delivery, and long-term user retention. **Limitations of Engagement-Based Metrics** * Traditional signals like "likes" and "watch time" are often noisy and may not reflect a user’s actual long-term interests. * Models optimized solely for engagement tend to favor short-term value over the long-term utility of the product. * Internal research found that previous heuristic-based interest models only achieved 48.3% precision in identifying what users truly care about. * Effective interest matching requires understanding nuanced factors such as production style, mood, audio, and motivation, which implicit signals often miss. **The User True Interest Survey (UTIS) Model** * Meta collects direct feedback via randomized, single-question surveys asking users to rate video interest on a 1–5 scale. * The raw survey data is binarized to denoise responses and weighted to correct for sampling and nonresponse bias. * The UTIS model functions as a lightweight "alignment model layer" built on top of the main multi-task ranking system. * The architecture uses existing model predictions as input features, supplemented by engineered features that capture content attributes and user behavior. **Integration into the Ranking Funnel** * **Late Stage Ranking (LSR):** The UTIS score is used as an additional input feature in the final value formula, allowing the system to boost high-interest videos and demote low-interest ones. * **Early Stage Ranking (Retrieval):** The model aggregates survey data to reconstruct user interest profiles, helping the system source more relevant candidates during the initial retrieval phase. * **Knowledge Distillation:** Large sequence-based retrieval models are aligned using UTIS predictions as labels through distillation objectives. **Performance and Impact** * The deployment of UTIS has led to a measurable increase in the delivery of niche, high-quality content. * Generic, popularity-based recommendations that often lack depth have been reduced. * Meta observed robust improvements across core metrics, including higher follow rates, more shares, and increased user retention. * The system now offers better interpretability, allowing engineers to understand which specific factors contribute to a user’s sense of "interest match." To continue improving the Reels ecosystem, Meta is focusing on doubling down on personalization by tackling challenges related to sparse data and sampling bias while exploring more advanced AI architectures to further diversify recommendations.

Ranking SystemsAIMachine LearningRecommendation Systems+5

pinterest·Dec 10, 2025·3 min readCurated summary

LLM-Powered Relevance Assessment for Pinterest Search

Pinterest Search uses fine-tuned multilingual LLMs to assess search-result relevance at a much larger scale than human labeling allows. The approach combines five-level relevance classification, stratified query sampling, and paired A/B-test evaluation to detect smaller overall effects and differences across query types. XLM-RoBERTa-large provides a practical balance of accuracy and cost, achieving strong agreement with human judgments while enabling substantially faster labeling. ## Relevance Measurement Challenges - Search relevance measures how well Pins satisfy a user’s query, rather than merely reflecting past engagement. - Human annotations are expensive and limited in volume. - Previous sampling designs could detect only relatively large topline changes, with minimum detectable effects (MDEs) around 1.3%–1.5%. - Limited labels also made it difficult to measure heterogeneous effects across query interests or popularity segments. ## Fine-Tuned LLM Relevance Model - Pinterest defines relevance using five labels: - L5: Highly Relevant - L4: Relevant - L3: Marginally Relevant - L2: Irrelevant - L1: Highly Irrelevant - A cross-encoder model predicts the relevance of each Pin for a query. - Open-source multilingual models are fine-tuned on human-annotated examples using multiclass cross-entropy loss. - Pin representations include: - Titles and descriptions - BLIP-generated image captions - Linked-page titles and descriptions - Board titles where Pins were saved - Highly engaged query tokens associated with the Pin - Models tested included multilingual BERT, T5, mDeBERTa, XLM-RoBERTa, and Llama 3. - The final relevance label is selected from the model’s five output scores using argmax. ## Stratified Query Sampling - Lower LLM labeling costs allow Pinterest to use much larger and more detailed samples. - Queries are stratified using: - A DistilBERT-based query-to-interest model - Query popularity, based on how many users issue each query - Stratification improves representativeness and reduces variance by grouping similar queries. - Pinterest moved from simple random sampling to stratified sampling with optimal allocation across strata. - Most of the MDE improvement came from variance reduction through stratification. - The redesigned process reduced MDEs from approximately 1.3%–1.5% to 0.25% or less. ## LLM-Based A/B-Test Measurement - Pinterest samples paired queries from control and treatment groups. - Pairing controls for differences between queries, which are a major source of relevance variance. - For each query, the top 25 results are retained and labeled by the LLM. - Query-level relevance is measured using sDCG@25, a variant of nDCG that assumes an unlimited supply of highly relevant L5 results. - Results are aggregated into topline experiment metrics. - Heterogeneous effects are analyzed by query popularity and interest categories such as beauty, fashion, and art. - The Benjamini–Hochberg procedure controls the false discovery rate when testing multiple segments. ## Model Choice and Validation - XLM-RoBERTa-large was selected for its balance of quality and efficiency. - On a single A10G GPU, it can label 150,000 rows in about 30 minutes. - Llama 3–8B produced slightly better accuracy but required roughly six times the inference time and cost. - LLM labels matched human labels exactly for 73.7% of Pins. - A total of 91.7% of predictions differed from human ratings by no more than one relevance point. Pinterest’s approach makes relevance evaluation cheaper, faster, and more statistically sensitive. Fine-tuned LLMs paired with stratified sampling are recommended for search experimentation when human labeling cannot provide enough coverage to detect small or heterogeneous ranking effects.

Read original(opens in new tab)

Ranking SystemsLarge Language ModelsNatural Language ProcessingFine Tuning+4