Attention Mechanisms - Meta

metaAug 5, 20263 min readCurated summary

From User Sequences to Scaling Laws: A Multi-Stage Architecture for Meta’s Ads Ranking

Meta’s new sequence-learning platform improves ads recommendations by separating deep offline user modeling from fast online ranking. Combined with dense tokenization and target-aware attention, it enables richer behavioral representations, predictable compute-to-performance scaling, and major gains: 6% more Instagram conversions, 3% more Facebook conversions, and 3.5% more Facebook ad clicks. The system is also a core part of Meta’s Generative Ads Recommendation Model (GEM). ## Challenges of Earlier Sequence Models - Ads systems must rank thousands of candidates within milliseconds and process millions of candidates per second. - Hybrid architectures typically use: - One model for user event sequences. - Another for sparse feature interactions. - This design can cause: - Lossy knowledge transfer between components. - Continued dependence on manually engineered features. - Scaling limits caused by interference between sequence modeling and ranking. - Increasing sequence lengths and transformer capacity can therefore raise serving costs without delivering proportional improvements. ## Multi-Stage Sequence Modeling Meta separates sequence learning into two complementary stages: - **Offline user modeling** - Processes long user histories asynchronously. - Uses deep transformer models with thousands of events and multiple layers. - Produces cached, user-level embeddings that represent long-term behavioral patterns. - Keeps user features separate from ad and context features so embeddings remain independent of individual candidates. - **Online ranking** - Combines cached user embeddings with fresh user signals, ad features, and context. - Performs final ranking under strict latency requirements. - Uses a lightweight architecture optimized for real-time serving. This separation allows the offline model to grow in depth, width, and sequence length without proportionally increasing online serving costs. ## Dense Tokenization and Target-Aware Attention - **Dense tokenization** - Converts sparse features and sequential behavioral data into a shared dense vocabulary. - Allows the model to learn feature interactions directly instead of relying on manually engineered cross-features. - **Target-aware multi-head attention** - Combines user behavior sequences with the specific ad candidate being scored. - Lets each attention layer determine which past behaviors matter for that candidate. - Stacked attention blocks capture increasingly complex interactions and compress long histories into compact representations. - The approach is designed to be memory-efficient while preserving candidate-specific information. ## Predictable Scaling Laws - On real-world ads traffic, the architecture shows an LLM-like log-linear relationship between compute and recommendation performance. - Improvements were measured using normalized entropy across: - Model depth. - Model width. - Sequence length. - Content and semantic enrichment. - The scaling behavior suggests the architecture is well suited to continued investment in sequence learning, despite recommendation systems combining sparse IDs with temporal data rather than dense text. ## Scaling Strategies - **Balanced model shape** - Depth, width, and sequence length should grow together. - Scaling only one dimension can create bottlenecks and diminishing returns. - Meta calls this the “scaling synergy principle.” - **Multi-stage tunability** - Online models offer strong improvements per unit of compute but are constrained by request latency. - Offline models improve more gradually but can scale aggressively because inference is asynchronous. - **Sequence composition** - Longer sequences generally improve performance. - Diversity of actions is more valuable than simply adding more homogeneous events. ## Practical Conclusion Meta’s approach makes sequence learning more scalable and operationally practical by moving expensive user-history processing offline while retaining fast, target-specific ranking online. Dense tokenization and target-aware attention reduce manual feature engineering, while the observed scaling laws provide a framework for deciding where additional model capacity and compute will produce the greatest gains.

Read original

Attention Mechanisms Transformer Recommendation Systems Scaling Laws+3

metaAug 3, 20264 min readCurated summary

GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model

Meta’s Generative Ads Recommendation Model (GEM), which powers ad recommendations across Instagram and Facebook, now trains at LLM scale across several thousand GPUs. By co-designing kernels, numerical precision, parallelism, networking, and memory management, Meta doubled end-to-end training efficiency to 20–25% Model FLOPs Utilization (MFU) while increasing training compute fourfold in 12 months. The work shows that recommendation models require infrastructure specifically adapted to their hybrid architecture and data patterns rather than a direct reuse of LLM techniques. ## GEM’s Architecture and Training Challenges - GEM combines: - Trillions of sparse embedding parameters. - Billions of dense parameters. - Sequence features, such as user activity history. - Non-sequence features, such as user location and ad representations. - Different feature groups use customized attention mechanisms while still supporting cross-feature learning. - Recommendation workloads differ substantially from typical LLMs: - User histories have highly variable lengths, making padding inefficient and potentially wasting up to 50% of computation. - Attention patterns are asymmetric, including long sequences with short windows and long queries with short key/value sets. - Small embedding dimensions and normalization layers create memory-bound operations. - CTR and CVR optimization are numerically sensitive, so aggressive low-precision training can harm model quality. ## Scaling Across Thousands of GPUs - GEM’s distributed training latency is determined by the slowest rank and the larger of its local computation or communication time. - Efficient scaling requires: - Computation to dominate communication. - Communication to overlap with computation without resource contention. - Minimal activation recomputation. - Balanced workloads across GPU ranks. - GEM makes these requirements difficult because: - Trillion-scale sparse parameters generate substantial communication. - Different layer types provide uneven opportunities for communication overlap. - Long sequences and large activations pressure GPU memory. - Jagged inputs create changing load imbalance and stragglers. ## Separating Compute and Scaling Efficiency - Meta measures end-to-end efficiency with: - **E2E MFU = Local MFU × Scaling Ratio** - **Local MFU** measures how effectively one GPU uses its compute hardware, including Tensor Cores and memory hierarchies. - **Scaling Ratio** measures how much single-GPU performance is retained across thousands of GPUs. - This framework separates: - Kernel design and numerical precision issues affecting individual GPUs. - Parallelism, networking, memory, and load-balancing issues affecting distributed training. ## Compute-Efficiency Optimizations - Meta developed recommendation-specific GPU kernels, including: - Jagged Flash Attention (JFA) for variable-length sequences. - Generalized Dot-Product Attention (GDPA). - BlockAttention. - These kernels are designed around GEM’s irregular shapes and asymmetric attention patterns rather than conventional LLM assumptions. - Mixed ultra-low-precision training, including MXFP8 for attention and MLP layers, improves throughput while accounting for recommendation models’ numerical sensitivity. - The kernels and precision recipes are customized to exploit the architecture of the latest-generation GPUs. ## Scaling-Efficiency Optimizations - Meta uses topology-aware five-dimensional parallelism to distribute GEM efficiently. - Dense parameters use: - Two-dimensional Fully Sharded Data Parallelism (FSDP). - Expert Parallelism. - Sparse parameters use fully sharded two-dimensional model parallelism. - These strategies are co-designed with Meta’s multi-tier network hierarchy to reduce communication overhead. - Streaming Multiprocessor (SM)-free collectives help communication run with less interference from GPU computation. - The overall design targets communication overlap, memory constraints, load balance, and the differing behavior of dense and sparse parameters. ## Results - GEM’s end-to-end training efficiency increased to 20–25% MFU. - Efficiency doubled over a 12-month period. - Total training FLOPs increased fourfold. - The results demonstrate that recommendation foundation models can reach LLM-scale training, but only through coordinated hardware and software optimization across kernels, precision, parallelism, networking, and memory. For large recommendation models, LLM infrastructure provides a starting point but is not sufficient. The practical recommendation is to optimize compute and distributed scaling as separate but connected problems, using workload-specific kernels, carefully validated low precision, topology-aware parallelism, and communication strategies tailored to sparse and dense model components.

Read original

Attention Mechanisms Large Language Models Recommendation Systems Distributed Training+3