Reinforcement Learning - Netflix

netflixJul 30, 20264 min readCurated summary

GenRec: Towards LLM-Native Recommendation at Netflix

GenRec is Netflix’s LLM-backed recommendation ranker, designed to reduce dependence on thousands of hand-engineered features. It verbalizes user history, item metadata, and context, then post-trains a Netflix-adapted foundation model for catalog-aware ranking and long-term member value. In large-scale A/B testing, it reportedly improved both short- and long-term metrics while using far fewer labeled examples and input signals than an established production ranker. ## Motivation for an LLM-Native Recommender - Netflix’s existing recommendation stack supports many content types and product surfaces but is costly to extend. - New use cases can require substantial feature engineering, architectural changes, infrastructure work, and experimentation. - LLMs offer: - Shared semantic representations for users, items, and interactions - Natural-language steering through prompts - Rich understanding of content and user histories - General-purpose LLMs are not production-ready on their own because they may: - Over-recommend globally popular titles - Hallucinate items outside the catalog - Ignore business constraints - Provide limited personalization ## Ranking Problem and Long-Term Utility - GenRec ranks the full Netflix catalog, or a provided candidate set. - It uses the user, interaction history, current context, and time to produce a personalized ranking. - The optimization target is expected long-term member utility—a proxy for satisfaction and retention—rather than only immediate clicks or plays. ## Two-Phase Training ### Netflix-Adapted Foundation Model - Netflix first adapts an open-source LLM using proprietary Netflix data. - The model learns: - Netflix content and metadata - Member behavior and preference patterns - General language understanding and generation - This relatively stable foundation can support multiple Netflix applications. ### GenRec Post-Training - A second training phase specializes the foundation model for recommendation. - It focuses on ranking quality, steering, reward alignment, and serving-cost constraints. - This phase is refreshed more frequently to reflect new content and changing member preferences. ## Interaction Data as Conversations - Hundreds of billions of events—including views, play duration, feedback, add-to-list actions, and abandons—are converted into single- or multi-turn recommender conversations. - Each user message includes verbalized: - Context and profile - Interaction history - Item metadata - The recommendation task - Assistant messages represent actual member behavior, such as titles played, viewing duration, or feedback. - During inference, GenRec uses the verbalized request and a catalog-aware scoring head; it does not generate conversational responses. - The conversational format mainly supports language-model training and comprehension of rich textual inputs. ## Context Engineering Instead of Feature Engineering - GenRec represents histories and context as natural language rather than relying primarily on dense, manually designed features. - The token budget becomes the new feature budget, so histories are selectively compressed: - Retain detailed, high-signal events such as long plays and thumbs-up - Remove low-signal events such as brief plays and hovers - Summarize repetitive behavior such as binge-watching - Add detail for important or cold-start items, including new releases - Recent and high-value interactions receive priority, while older information is compressed or dropped. - Prompt structure is also optimized for shared prefixes and prefix caching, reducing serving costs. ## Ranking, Language, and Reward Objectives - GenRec combines several training objectives: - **Catalog-aware ranking:** Uses high-value engagements as positive labels, applies denoising and thresholds, and trains the model with cross-entropy over the catalog or candidate set. - **Language modeling:** Preserves understanding of natural-language histories and metadata and supports potential future features such as recommendation explanations. - **Reward-weighted alignment:** Incorporates business requirements and longer-term satisfaction into training. - Reward signals can help balance content categories such as movies, series, games, live content, and podcasts instead of optimizing only immediate engagement. ## Serving and Results - GenRec runs in prefill-only mode on Netflix’s LLM serving infrastructure, avoiding the cost of decoding generated text. - A catalog-aware scoring head converts the model’s representations into item rankings. - Compared with a mature production ranker, GenRec achieved statistically significant improvements in short- and long-term online metrics. - It did so with a small fraction of the labeled data and input signals used by the existing system. GenRec suggests that recommendation systems can shift from extensive manual feature construction toward careful context engineering, LLM post-training, and reward alignment. The approach is most promising when paired with catalog constraints, efficient serving, and objectives that reflect long-term member value rather than raw engagement alone.

Read original

Reinforcement Learning Large Language Models Natural Language Processing Recommendation Systems+2

netflixJun 29, 20263 min readCurated summary

GenPage: Towards End-to-End Generative Homepage Construction at Netflix

GenPage is Netflix’s end-to-end generative approach to building personalized homepages. Instead of separately ranking rows and items, one transformer autoregressively generates the entire page—including rows, entities, and layout—from user and request context. In production, it outperformed Netflix’s mature multi-stage recommender on a core engagement metric while reducing serving latency by 20%. ## Reframing Homepage Recommendation - Netflix’s homepage is a personalized two-dimensional structure, not a single ranked list. - Traditional systems use separate candidate-generation and ranking stages for rows and entities. - GenPage treats homepage construction as a prompt-response task: - The prompt contains user history, profile information, and request context. - The response is the complete homepage generated autoregressively. - The approach aims to: - Replace complex multi-stage pipelines with one end-to-end model. - Optimize the whole page using reinforcement learning. - Capture interactions such as diversity and the trade-off between high-value rows and continued browsing. - Scale more predictably with additional data, compute, and model capacity. - Support new content types, layouts, UI components, and personalized artwork with fewer architectural changes. ## Production Challenges and Results - Real-time generation makes serving latency a major constraint. - The system must address: - Cold-start entities in a constantly changing catalog. - Shifting user interests and cultural trends. - Product and business rules that constrain generated pages. - An online A/B test against Netflix’s optimized production recommender produced: - Statistically significant improvement on Netflix’s primary launch engagement metric. - A 20% reduction in end-to-end serving latency. - Offline experiments found that: - Improving the prompt helped more than increasing model capacity in the tested regime. - Reinforcement-learning post-training improved homepage diversity, even though diversity was not an explicit objective. ## Tokenizing Context and Pages - Each training example contains: - **Context:** user history, profile attributes, and request information. - **Page:** displayed rows and entities in layout order. - **Feedback:** interactions such as plays, thumbs-up, and abandonment. - Context and page are tokenized as model inputs and outputs. - Feedback is used to derive reward and supervision signals rather than being directly generated. ## Domain-Specific Tokenization - GenPage uses a custom recommender-system tokenizer instead of a general-purpose text tokenizer. - This reduces sequence length and improves inference cost and latency. - For example, an action such as watching *Orange Is the New Black* can be represented with four tokens: - Entity ID - Action type - Time bucket - Duration bucket - Direct token mappings to product concepts, such as rows and entities, also make it easier to enforce generation rules and business constraints. ## Context Representation - User-history tokens encode: - Action type - Entity ID - Timestamp - Duration - The history includes explicit signals, such as playback, adding titles to My List, and thumbs-up, as well as implicit signals such as trailer views and detail-page visits. - Profile tokens represent attributes including language and profile type. - Request-context tokens include time of day, day of week, and device. - Long data sources, such as complete impression histories, are summarized to control sequence length and cost. - These summaries improve practicality but introduce handcrafted prompt engineering; learning to compress such information end to end remains a future direction. - Special segment markers help the model distinguish between different context sources.

Read original

Reinforcement Learning Generative AI Transformer Personalization+2

netflixJun 19, 20263 min readCurated summary

Thinking Fast & Slow for a Personalized Notification System

Netflix’s personalized notification system separates long-term messaging strategy from real-time content selection. A “slow” policy sets each member’s personalized weekly pacing plan, while a “fast” policy chooses the best message when an opportunity arises. This hierarchy addresses the limits of short-term optimization by balancing immediate engagement with fatigue, opt-outs, and long-term member experience. ## Limitations of the Previous System - The earlier system used a causal model to estimate the short-term incremental effect of sending a single notification. - It optimized immediate actions, but could not capture cumulative effects such as: - Notification fatigue - Declining responsiveness over time - Sustained viewing behavior - Gradual opt-out risk - Send frequency and message ranking were coupled: - A relevance threshold implicitly controlled overall send volume. - Changing the threshold affected both frequency and the quality or distribution of selected messages. - Frequency could not be explicitly personalized according to each member’s engagement patterns. ## The Hierarchical Slow-Fast Architecture - The **Slow policy** makes strategic decisions over a longer horizon, such as a week. - It selects a personalized “Pacing Plan Action” that defines intended push and email frequencies. - The action space contains roughly 100 combinations of cross-channel pacing strategies. - The **Fast policy** operates in real time, selecting the most relevant message within the limits established by the slow policy. ## Utility-Based Strategic Planning The Slow policy chooses the action that maximizes a personalized utility function: `U(member, action) = Σ wₖ · Rewardₖ(member, action) — Cost(action)` - Positive signals estimate whether a member will value and engage with notifications. - Negative signals estimate fatigue and the likelihood of opting out of a channel. - Explicit negative feedback is sparse, so predicted messaging costs alone are too small to prevent excessive sending. - Netflix adds a universal cost to every message, ensuring that the utility remains well-behaved and discourages “always send” strategies. - This cost is tuned through online experiments and offline evaluation. ## Pacing Messages Over Time - A basic pacing strategy converts the target frequency into a per-opportunity probability. - At each eligible opportunity, the system uses weighted randomization to decide whether to send. - This produces a naturally varied schedule while maintaining the desired expected frequency. - The architecture can also support structured patterns, including: - Day-of-week preferences - User-activity-based pacing - Bursts aligned with product launches ## Communication Between Policies - The Slow policy calculates a member’s plan and stores it in a low-latency feature store. - The Fast policy retrieves that plan whenever a notification opportunity occurs. - This asynchronous event-and-state design lets the planner focus on long-term member health while the executor focuses on immediate relevance. The main recommendation is to decouple notification frequency and pacing from message ranking. A hierarchical system can explicitly manage long-term communication strategy while preserving the responsiveness and personalization of real-time selection.

Read original

Reinforcement Learning Feature Store Personalization Notification Systems+3

netflixOct 25, 2025Original article

Post-Training Generative Recommenders with Advantage-Weighted Supervised Finetuning | by Netflix Technology Blog | Netflix TechBlog (opens in new tab)

Netflix is evolving its recommendation systems by moving beyond simple behavior imitation toward generative recommenders that better align with true user preferences. While generative models like HSTU and OneRec effectively capture sequential user patterns, they often struggle to distinguish between habitual clicks and genuine satisfaction. To bridge this gap, Netflix developed Advantage-Weighted Supervised Fine-tuning (A-SFT), a post-training method that leverages noisy reward signals to refine model performance without the need for complex counterfactual data. ### The Shift to Generative Recommenders * Modern generative recommenders (GRs), such as HSTU and OneRec, utilize transformer architectures to treat recommendation as a sequential transduction task. * The models are typically trained using next-item prediction, where the system learns to imitate the chronological sequence of a user’s activities. * A significant drawback of this "behavior cloning" approach is that it captures external trends and noise rather than long-term user satisfaction, potentially recommending content the user finished but did not actually enjoy. ### Barriers to Reinforcement Learning in RecSys * Traditional post-training methods used in Large Language Models, such as Proximal Policy Optimization (PPO) or Direct Preference Optimization (DPO), require counterfactual feedback that is difficult to obtain in recommendation contexts. * Because user sequences span weeks or years, it is impractical to generate and test hypothetical, counterfactual experiences for real-time user validation. * Reward signals in recommendation systems are inherently noisy; for instance, high watch time might indicate interest, but it can also be a result of external circumstances, making it an unreliable metric for optimization. ### Advantage-Weighted Supervised Fine-tuning (A-SFT) * A-SFT is a hybrid approach that sits between offline reinforcement learning and standard supervised fine-tuning. * The algorithm incorporates an advantage function to weight training examples, allowing the model to prioritize actions that lead to higher rewards while filtering out noise from the reward model. * This method is specifically designed to handle high-variance reward signals, using them as directional guides rather than absolute truth, which prevents the model from over-exploiting inaccurate data. * Benchmarks against other representative methods show that A-SFT achieves superior alignment between the generative recommendation policy and the underlying reward model. For organizations managing large-scale recommendation engines, A-SFT offers a practical path to implementing post-training improvements. By focusing on advantage-weighted signals, developers can improve recommendation quality using existing implicit feedback—like watch time and clicks—without the infrastructure hurdles of online reinforcement learning.

Reinforcement Learning AI Machine Learning Transformer+4