Scaling LLM Post-Training at Netflix (opens in new tab)

Netflix argues that LLM post-training at production scale is as much an infrastructure challenge as a modeling challenge. Its internal framework abstracts distributed data processing, model sharding, GPU orchestration, checkpointing, and complex training workflows so developers can focus on experimentation. The result is a flexible system supporting SFT, DPO, reinforcement learning, and knowledge distillation across hundreds of GPUs.

Why Post-Training Becomes an Engineering Problem

• Pre-training provides general language ability, but post-training adapts models to Netflix’s catalog, member histories, recommendation tasks, personalization, and search.
• Production-scale training introduces challenges involving:
  • Large proprietary datasets
  • Multi-node GPU coordination
  • Distributed model state
  • Workflows that combine training and inference
  • Failure recovery and experiment tracking
• A simple Hugging Face fine-tuning script is insufficient for reliable, large-scale jobs.

Preparing Data Correctly

• Chat templates serialize conversations but do not determine which tokens should contribute to the loss.
• Netflix applies explicit loss masking so training focuses on assistant responses rather than prompts or other non-target text.
• Variable-length examples can waste GPU memory through padding and create synchronization overhead across FSDP workers.
• Sequence packing combines multiple samples into fixed-length sequences.
• A document mask prevents attention across separately packed samples while improving GPU utilization.

Loading and Optimizing Large Models

• Models that do not fit on one GPU require sharding strategies such as FSDP or tensor parallelism.
• Partial weights should be loaded directly onto the device mesh rather than materializing the entire checkpoint on a single device.
• Developers can choose full fine-tuning or LoRA and use:
  • Activation checkpointing
  • Compilation
  • Appropriate precision settings
• Reinforcement learning requires compatible precision between rollout generation and policy training.
• Large vocabularies create memory pressure because logits have dimensions [batch, seq_len, vocab].
• The framework reduces peak memory by removing ignored tokens before projection and computing logits and loss in sequence chunks.

Distributed Training and Workflow Management

• The framework supports standard forward/backward training for SFT as well as workflows that interleave:
  • Rollout generation
  • Reward-model and reference-model inference
  • Policy updates
• Ray actors orchestrate distributed jobs while keeping hardware concerns separate from modeling code.
• Experiment tracking covers both quality metrics, such as loss, and efficiency metrics, such as Model FLOPS Utilization (MFU).
• Standardized checkpointing allows jobs to resume after failures.

Netflix’s Post-Training Framework

• The stack is built on:
  • Mako for AWS GPU provisioning
  • PyTorch, Ray, and vLLM
  • Netflix’s framework library for reusable utilities and training recipes
• Jobs are generally defined through configuration files that select a recipe and provide task-specific components.
• Unlike narrower fine-tuning systems, the framework supports:
  • Custom output heads
  • Expanded vocabularies and semantic IDs
  • Special tokens
  • Transformer models trained on non-natural-language sequences
• This flexibility is important for Netflix-specific recommendation and personalization use cases.

Four Core Abstractions

Data

• Dataset abstractions cover SFT, reward modeling, and RL.
• Streaming supports datasets larger than local disk capacity.
• Asynchronous sequence packing overlaps CPU preprocessing with GPU execution to reduce idle time.

Model

• The framework supports architectures such as Qwen3 and Gemma3, including Mixture-of-Experts variants.
• LoRA is integrated into model definitions.
• High-level sharding APIs distribute models across device meshes without requiring developers to write low-level distributed code.

Compute

• A unified job interface scales from one node to hundreds of GPUs.
• MFU measurement remains accurate for custom architectures and LoRA configurations.
• Checkpoints include parameters, optimizer state, dataloader state, and data-mixer state, enabling exact resumption.

Workflow

• The system supports SFT, DPO, RL, and knowledge distillation.
• Online RL uses a hybrid architecture combining a single controller with Single Program, Multiple Data (SPMD) workers.
• This extends conventional SPMD training to multi-stage workflows that cannot be represented as a simple training loop.

Netflix’s approach is to standardize the difficult operational parts of post-training while preserving enough flexibility for unconventional models and objectives. A framework built around reusable data, model, compute, and workflow abstractions can help teams iterate faster and scale experiments without repeatedly rebuilding distributed infrastructure.