MediaFM: The Multimodal AI Foundation for Media Understanding at Netflix

Netflix’s Media Foundational Model (MediaFM) is a tri-modal AI system that combines video, audio, and timed text to understand long-form entertainment. It represents sequences of shots while using title-level metadata and temporal context to produce richer content embeddings. Netflix concludes that these contextual embeddings improve many downstream tasks, including advertising relevance, clip selection, tone classification, and popularity prediction.

Motivation for MediaFM

• Netflix needs machine-readable understanding of its expanding catalog, including films, series, live events, and podcasts.
• Long-form media requires recognizing narrative dependencies, emotional arcs, scene transitions, and subtle tones across entire episodes or films.
• Combining visual, audio, and textual signals provides a more complete understanding than relying on video alone.
• The resulting embeddings support applications such as:
  • Cold-start recommendations for new titles
  • Promotional art and trailer optimization
  • Advertising relevance
  • Clip tagging and internal content analysis

Multimodal Input Representation

• The model uses a shot as its fundamental unit, with titles segmented using shot-boundary detection.
• Each shot receives three modality-specific embeddings:
  • Video: Frames sampled from the shot are encoded with SeqCLIP, Netflix’s video-retrieval model.
  • Audio: Sound is encoded using Meta FAIR’s wav2vec2.
  • Timed text: Captions, subtitles, or audio descriptions are encoded with OpenAI’s text-embedding-3-large.
• The three embeddings are concatenated and unit-normalized into a 2,304-dimensional fused vector.
• Training examples consist of temporally ordered shot sequences from a movie or episode, with up to 512 shots.
• Title metadata, such as synopses and tags, is also embedded and supplied as global context.

Transformer Architecture

• MediaFM uses a BERT-like Transformer encoder.
• Fused shot embeddings are first projected into the model’s hidden dimension.
• Two special tokens are prepended:
  • [CLS], a learnable sequence-level embedding
  • [GLOBAL], containing projected title-level metadata
• Positional embeddings and self-attention allow each shot representation to incorporate surrounding narrative context.
• A final projection maps contextualized representations back into the original 2,304-dimensional embedding space.

Masked Shot Modeling

• The model masks 20% of shot embeddings in each training sequence.
• Masked inputs are replaced with a learnable [MASK] embedding.
• The Transformer must reconstruct the original fused embedding for each masked shot.
• Training minimizes cosine distance between predicted and ground-truth embeddings.
• Hidden parameters are optimized with Muon, while other parameters use AdamW; Netflix reports noticeable gains after adopting Muon.

Evaluation Through Linear Probes

• Netflix evaluates MediaFM by freezing its representations and training task-specific linear layers on top.
• Most evaluation tasks involve short clips extracted from larger titles.
• Embedding a clip within the context of its surrounding episode or film performs better than embedding the clip in isolation, demonstrating the value of long-range contextualization.

Downstream Applications

• Ad relevancy: Multilabel classification identifies clips suitable for relevant advertising; MediaFM helps retrieve candidate clips before ad-serving optimization.
• Clip popularity ranking: The model predicts relative clip performance and click-through rate within a title, evaluated using Kendall’s tau.
• Clip tone: Clips are classified into 100 categories, such as creepy, scary, or humorous.
• Clip genre: Clips are assigned to core genres including Action, Comedy, Documentary, Drama, Horror, Romance, and Thriller.
• Clip retrieval: The system distinguishes “clip-worthy” content from unsuitable clips based on human annotations, using Average Precision.

MediaFM’s main practical lesson is that effective media understanding depends on fusing all available modalities and preserving long-form temporal context. Netflix’s approach provides a reusable embedding foundation for recommendation, promotion, advertising, and content-analysis systems rather than building a separate representation for every task.