From User Sequences to Scaling Laws: A Multi-Stage Architecture for Meta’s Ads Ranking

Meta’s new sequence-learning platform improves ads recommendations by separating deep offline user modeling from fast online ranking. Combined with dense tokenization and target-aware attention, it enables richer behavioral representations, predictable compute-to-performance scaling, and major gains: 6% more Instagram conversions, 3% more Facebook conversions, and 3.5% more Facebook ad clicks. The system is also a core part of Meta’s Generative Ads Recommendation Model (GEM).

Challenges of Earlier Sequence Models

• Ads systems must rank thousands of candidates within milliseconds and process millions of candidates per second.
• Hybrid architectures typically use:
  • One model for user event sequences.
  • Another for sparse feature interactions.
• This design can cause:
  • Lossy knowledge transfer between components.
  • Continued dependence on manually engineered features.
  • Scaling limits caused by interference between sequence modeling and ranking.
• Increasing sequence lengths and transformer capacity can therefore raise serving costs without delivering proportional improvements.

Multi-Stage Sequence Modeling

Meta separates sequence learning into two complementary stages:

• Offline user modeling

  • Processes long user histories asynchronously.
  • Uses deep transformer models with thousands of events and multiple layers.
  • Produces cached, user-level embeddings that represent long-term behavioral patterns.
  • Keeps user features separate from ad and context features so embeddings remain independent of individual candidates.

• Online ranking

  • Combines cached user embeddings with fresh user signals, ad features, and context.
  • Performs final ranking under strict latency requirements.
  • Uses a lightweight architecture optimized for real-time serving.

This separation allows the offline model to grow in depth, width, and sequence length without proportionally increasing online serving costs.

Dense Tokenization and Target-Aware Attention

• Dense tokenization

  • Converts sparse features and sequential behavioral data into a shared dense vocabulary.
  • Allows the model to learn feature interactions directly instead of relying on manually engineered cross-features.

• Target-aware multi-head attention

  • Combines user behavior sequences with the specific ad candidate being scored.
  • Lets each attention layer determine which past behaviors matter for that candidate.
  • Stacked attention blocks capture increasingly complex interactions and compress long histories into compact representations.
  • The approach is designed to be memory-efficient while preserving candidate-specific information.

Predictable Scaling Laws

• On real-world ads traffic, the architecture shows an LLM-like log-linear relationship between compute and recommendation performance.
• Improvements were measured using normalized entropy across:
  • Model depth.
  • Model width.
  • Sequence length.
  • Content and semantic enrichment.
• The scaling behavior suggests the architecture is well suited to continued investment in sequence learning, despite recommendation systems combining sparse IDs with temporal data rather than dense text.

Scaling Strategies

• Balanced model shape

  • Depth, width, and sequence length should grow together.
  • Scaling only one dimension can create bottlenecks and diminishing returns.
  • Meta calls this the “scaling synergy principle.”

• Multi-stage tunability

  • Online models offer strong improvements per unit of compute but are constrained by request latency.
  • Offline models improve more gradually but can scale aggressively because inference is asynchronous.

• Sequence composition

  • Longer sequences generally improve performance.
  • Diversity of actions is more valuable than simply adding more homogeneous events.

Practical Conclusion

Meta’s approach makes sequence learning more scalable and operationally practical by moving expensive user-history processing offline while retaining fast, target-specific ranking online. Dense tokenization and target-aware attention reduce manual feature engineering, while the observed scaling laws provide a framework for deciding where additional model capacity and compute will produce the greatest gains.