From User Sequences to Scaling Laws: A Multi-Stage Architecture for Meta’s Ads Ranking
Meta’s new sequence-learning platform improves ads recommendations by separating deep offline user modeling from fast online ranking. Combined with dense tokenization and target-aware attention, it enables richer behavioral representations, predictable compute-to-performance scaling, and major gains: 6% more Instagram conversions, 3% more Facebook conversions, and 3.5% more Facebook ad clicks. The system is also a core part of Meta’s Generative Ads Recommendation Model (GEM). ## Challenges of Earlier Sequence Models - Ads systems must rank thousands of candidates within milliseconds and process millions of candidates per second. - Hybrid architectures typically use: - One model for user event sequences. - Another for sparse feature interactions. - This design can cause: - Lossy knowledge transfer between components. - Continued dependence on manually engineered features. - Scaling limits caused by interference between sequence modeling and ranking. - Increasing sequence lengths and transformer capacity can therefore raise serving costs without delivering proportional improvements. ## Multi-Stage Sequence Modeling Meta separates sequence learning into two complementary stages: - **Offline user modeling** - Processes long user histories asynchronously. - Uses deep transformer models with thousands of events and multiple layers. - Produces cached, user-level embeddings that represent long-term behavioral patterns. - Keeps user features separate from ad and context features so embeddings remain independent of individual candidates. - **Online ranking** - Combines cached user embeddings with fresh user signals, ad features, and context. - Performs final ranking under strict latency requirements. - Uses a lightweight architecture optimized for real-time serving. This separation allows the offline model to grow in depth, width, and sequence length without proportionally increasing online serving costs. ## Dense Tokenization and Target-Aware Attention - **Dense tokenization** - Converts sparse features and sequential behavioral data into a shared dense vocabulary. - Allows the model to learn feature interactions directly instead of relying on manually engineered cross-features. - **Target-aware multi-head attention** - Combines user behavior sequences with the specific ad candidate being scored. - Lets each attention layer determine which past behaviors matter for that candidate. - Stacked attention blocks capture increasingly complex interactions and compress long histories into compact representations. - The approach is designed to be memory-efficient while preserving candidate-specific information. ## Predictable Scaling Laws - On real-world ads traffic, the architecture shows an LLM-like log-linear relationship between compute and recommendation performance. - Improvements were measured using normalized entropy across: - Model depth. - Model width. - Sequence length. - Content and semantic enrichment. - The scaling behavior suggests the architecture is well suited to continued investment in sequence learning, despite recommendation systems combining sparse IDs with temporal data rather than dense text. ## Scaling Strategies - **Balanced model shape** - Depth, width, and sequence length should grow together. - Scaling only one dimension can create bottlenecks and diminishing returns. - Meta calls this the “scaling synergy principle.” - **Multi-stage tunability** - Online models offer strong improvements per unit of compute but are constrained by request latency. - Offline models improve more gradually but can scale aggressively because inference is asynchronous. - **Sequence composition** - Longer sequences generally improve performance. - Diversity of actions is more valuable than simply adding more homogeneous events. ## Practical Conclusion Meta’s approach makes sequence learning more scalable and operationally practical by moving expensive user-history processing offline while retaining fast, target-specific ranking online. Dense tokenization and target-aware attention reduce manual feature engineering, while the observed scaling laws provide a framework for deciding where additional model capacity and compute will produce the greatest gains.
Read original(opens in new tab)