Large Language Models - Meta

metaAug 3, 20264 min readCurated summary

GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model

Meta’s Generative Ads Recommendation Model (GEM), which powers ad recommendations across Instagram and Facebook, now trains at LLM scale across several thousand GPUs. By co-designing kernels, numerical precision, parallelism, networking, and memory management, Meta doubled end-to-end training efficiency to 20–25% Model FLOPs Utilization (MFU) while increasing training compute fourfold in 12 months. The work shows that recommendation models require infrastructure specifically adapted to their hybrid architecture and data patterns rather than a direct reuse of LLM techniques. ## GEM’s Architecture and Training Challenges - GEM combines: - Trillions of sparse embedding parameters. - Billions of dense parameters. - Sequence features, such as user activity history. - Non-sequence features, such as user location and ad representations. - Different feature groups use customized attention mechanisms while still supporting cross-feature learning. - Recommendation workloads differ substantially from typical LLMs: - User histories have highly variable lengths, making padding inefficient and potentially wasting up to 50% of computation. - Attention patterns are asymmetric, including long sequences with short windows and long queries with short key/value sets. - Small embedding dimensions and normalization layers create memory-bound operations. - CTR and CVR optimization are numerically sensitive, so aggressive low-precision training can harm model quality. ## Scaling Across Thousands of GPUs - GEM’s distributed training latency is determined by the slowest rank and the larger of its local computation or communication time. - Efficient scaling requires: - Computation to dominate communication. - Communication to overlap with computation without resource contention. - Minimal activation recomputation. - Balanced workloads across GPU ranks. - GEM makes these requirements difficult because: - Trillion-scale sparse parameters generate substantial communication. - Different layer types provide uneven opportunities for communication overlap. - Long sequences and large activations pressure GPU memory. - Jagged inputs create changing load imbalance and stragglers. ## Separating Compute and Scaling Efficiency - Meta measures end-to-end efficiency with: - **E2E MFU = Local MFU × Scaling Ratio** - **Local MFU** measures how effectively one GPU uses its compute hardware, including Tensor Cores and memory hierarchies. - **Scaling Ratio** measures how much single-GPU performance is retained across thousands of GPUs. - This framework separates: - Kernel design and numerical precision issues affecting individual GPUs. - Parallelism, networking, memory, and load-balancing issues affecting distributed training. ## Compute-Efficiency Optimizations - Meta developed recommendation-specific GPU kernels, including: - Jagged Flash Attention (JFA) for variable-length sequences. - Generalized Dot-Product Attention (GDPA). - BlockAttention. - These kernels are designed around GEM’s irregular shapes and asymmetric attention patterns rather than conventional LLM assumptions. - Mixed ultra-low-precision training, including MXFP8 for attention and MLP layers, improves throughput while accounting for recommendation models’ numerical sensitivity. - The kernels and precision recipes are customized to exploit the architecture of the latest-generation GPUs. ## Scaling-Efficiency Optimizations - Meta uses topology-aware five-dimensional parallelism to distribute GEM efficiently. - Dense parameters use: - Two-dimensional Fully Sharded Data Parallelism (FSDP). - Expert Parallelism. - Sparse parameters use fully sharded two-dimensional model parallelism. - These strategies are co-designed with Meta’s multi-tier network hierarchy to reduce communication overhead. - Streaming Multiprocessor (SM)-free collectives help communication run with less interference from GPU computation. - The overall design targets communication overlap, memory constraints, load balance, and the differing behavior of dense and sparse parameters. ## Results - GEM’s end-to-end training efficiency increased to 20–25% MFU. - Efficiency doubled over a 12-month period. - Total training FLOPs increased fourfold. - The results demonstrate that recommendation foundation models can reach LLM-scale training, but only through coordinated hardware and software optimization across kernels, precision, parallelism, networking, and memory. For large recommendation models, LLM infrastructure provides a starting point but is not sufficient. The practical recommendation is to optimize compute and distributed scaling as separate but connected problems, using workload-specific kernels, carefully validated low precision, topology-aware parallelism, and communication strategies tailored to sparse and dense model components.

Read original

Large Language Models Recommendation Systems Distributed Training Gpu Optimization+3

metaJul 15, 20263 min readCurated summary

Exploring Hierarchical Interest Representation For Meta Ads Deep Funnel Optimization

Hierarchical Interest Representation is an upstream embedding layer for Meta Ads that connects users’ inferred interests with advertisers, products, and services. It combines engagement data, multimodal content, graph learning, and hierarchical abstractions to address sparse deep-funnel signals and rare or unseen entities. The resulting universal embeddings and “Bag-of-Meaning” interest tokens could support retrieval, personalization, supervision, and ranking across Meta’s advertising systems. ## Purpose and Role in Deep-Funnel Optimization - The system aims to identify people with genuine latent interest in an advertiser’s offerings. - It is intended to complement systems such as Meta’s Generative Ads Model (GEM), Andromeda, and the Adaptive Ranking Model. - It uses user behavior—including scrolling, engagement, and explicit “Interested/Not interested” feedback—to infer preferences. - The broader goal is to improve discovery-oriented ad experiences and downstream conversion performance. ## Technical Challenges ### Sparse Engagement and Large-Scale Graphs - Users, advertisers, products, services, and campaigns are modeled as graph nodes. - Interactions and activities form edges in a graph spanning millions of advertisers, millions of ads, and billions of users. - Deep-funnel feedback is relatively scarce, leaving many entities with limited direct evidence. ### Long-Range Relationships - Useful signals may come from indirectly connected users and entities rather than direct interactions. - Capturing these relationships at Meta’s scale requires memory-efficient sparse attention and high-performance graph-learning algorithms. ### Dynamic and Unseen Entities - The ads ecosystem changes rapidly, while individual entities may have little historical engagement. - Representations must generalize to rare and previously unseen businesses and products. ## Core Design Properties ### Dimension Reduction - The raw graph is projected into a configurable “super-graph.” - Learned latent interest primitives act as super-nodes. - Sparse user-ad relationships become denser connections at the interest level. - The primitive graph is more stable and stationary than the constantly changing ads vocabulary. ### Knowledge Enrichment - Advertiser and product representations incorporate text, images, video, metadata, and catalog attributes. - Vision and language models process this multimodal content. - Content helps the system understand what a product or business is, not merely how users interacted with it. - This enables better generalization to new or sparsely observed entities. ### Unified Relational Representation - Users, advertisers, products, and latent interest primitives are embedded in a shared metric space. - The system can estimate: - Relationships between interest primitives - Similarity between users, ads, and products - A user’s proximity to particular interests - Which interests an advertiser or product serves - Embedding operations support both primitive-to-primitive and cluster-to-cluster relationship modeling. ### Multiple Hierarchical Granularities - Coarse representations capture dense, stable, high-level interests. - Finer representations capture sparse and specific deep-funnel intent. - Cascading hierarchical layers allow the embeddings to serve different needs across retrieval, personalization, ranking, and supervision. ## Architecture and Training - The architecture combines: - An in-house transformer-based graph learner - Bias-aware attention - Self-supervised cross-view distillation - Sparse attention for long-range graph relationships - It combines real-world semantic knowledge with users’ temporal engagement histories. - The model learns multi-hierarchical interest representations across a large graph. - Training is performed end-to-end on real Meta Ads data involving billions of interactions. ## Outputs and Potential Applications - Universal embeddings for users and ads entities. - “Bag-of-Meaning” interest tokens representing latent interests at different granularities. - Potential uses include: - Ad retrieval - Personalization - Ranking - Specialized ranking architectures - Training supervision - Cross-entity similarity and discovery Hierarchical Interest Representation is best understood as shared infrastructure for Meta’s ads recommendation stack. By combining sparse behavioral evidence with multimodal world knowledge and hierarchical graph abstractions, it could make deep-funnel optimization more robust, especially for specialized, rare, or newly introduced products and advertisers.

Read original

Large Language Models Multimodal AI Transformer Embeddings+4

metaJun 25, 20263 min readCurated summary

Privacy-Aware Infrastructure in the AI-Native Era: An Asset Classification Case Study

Privacy-aware infrastructure depends on accurate asset classification before it can enforce retention, access, purpose, sharing, or anonymization policies. Because data is noisy, distributed, and constantly changing—especially in AI-native systems—LLMs are useful for ambiguity but should not make routine production decisions. The recommended approach combines rich contextual evidence, human-reviewed labels, narrowly used LLMs, and versioned deterministic rules that are faster, replayable, and auditable. ## Why Asset Classification Matters - Assets include more than tables and columns: they may be nested payload fields, logs, event parameters, API fields, ML features, embeddings, or derived datasets. - Classification must track the meaning of data as it moves through pipelines and changes representation. - A field such as `age` could represent sensitive personal information or an infrastructure cache TTL, making context essential. - Four recurring challenges shape the problem: - **Noisy signals:** Raw metadata can overwhelm models and hide relevant evidence. - **Distributed context:** Code, lineage, ownership, documentation, annotations, and usage patterns reside in separate systems. - **Changing requirements:** Product and policy changes can outpace static rules and periodic reviews. - **Enforcement consequences:** False positives cause unnecessary restrictions, while false negatives create protection gaps. - Classification must reason about ambiguity while producing decisions that can later be explained and reproduced. ## The Hybrid Classification Pattern - **Context beats prompts:** Improving the evidence supplied to a model generally matters more than endlessly tuning instructions. - Evidence briefs should organize: - Supporting and contradicting signals - Provenance - Relevant code and lineage - Masked or circular fields that could distort reasoning - **Evaluation must remain independent:** Human-reviewed reference labels, frozen test sets, separate models or prompts, and regression gates prevent the classifier from defining its own ground truth. - **Stable behavior should be distilled into rules:** LLMs handle novelty and uncertainty, while validated patterns become deterministic, versioned, and auditable logic. - Over time, the LLM’s production role should shrink as routine cases move to low-latency deterministic enforcement. ## A Stable Classification Contract - The classifier should operate as a platform service with a small, explicit interface. - Inputs include: - An asset identifier - A structured bundle of contextual evidence - Outputs include: - A taxonomy category - A confidence score calibrated against reviewed labels - A decision trace explaining influential evidence - The matching deterministic rule, when applicable - Versions for the context, rules, and prompt - Classifiers should answer one scoped, domain-specific question rather than use a universal taxonomy. - Narrow classifiers are easier to evaluate, debug, govern, and compose across downstream privacy decisions. ## Privacy-Aware Infrastructure Responsibilities Asset classification supports the broader PAI lifecycle: - Understanding what data exists and how it is governed - Discovering data flows relevant to a policy - Enforcing retention, access, purpose, and sharing constraints - Producing verifiable evidence of compliance ## Practical Recommendation Use LLMs selectively for ambiguous or novel assets, but build the surrounding system around structured context, independent human-reviewed evaluation, and deterministic rule promotion. This preserves the flexibility of AI while making routine privacy enforcement predictable, auditable, and operationally efficient.

Read original

Large Language Models Data Governance Human In The Loop Data Lineage+3

metaFeb 11, 20262 min readCurated summary

The Death of Traditional Testing: Agentic Development Broke a 50-Year-Old Field, JiTTesting Can Revive It

Just-in-Time Tests (JiTTests) are an LLM-driven testing approach designed for fast, agentic software development. Instead of maintaining static test suites, the system generates tests for each code change, simulates likely faults, and reports only meaningful regressions. The goal is to reduce test maintenance and false positives while catching serious bugs before production. ## Limitations of Traditional Testing - Tests are manually written as code changes enter the system. - They must account for both current behavior and unknown future changes. - This often leads to: - Tests that fail to detect relevant bugs. - False positives when intended changes break outdated assumptions. - Ongoing maintenance and review costs. - Agentic development increases the volume and speed of changes, making these problems harder and more expensive to manage. ## How Catching JiTTests Work - A new code change or pull request is submitted. - An LLM infers the likely intent of the change. - The system creates mutants—versions of the code containing deliberately introduced faults. - It generates and runs tests designed to expose those faults. - Rule-based and LLM-based assessors evaluate failures and filter out likely false positives. - Engineers receive a focused report when the system identifies an unexpected behavior change. Because the tests are tailored to a specific change, they can reason about intended behavior and distinguish legitimate updates from regressions. ## Benefits for Agentic Development - Tests are generated on demand and do not remain in the codebase. - There is no ongoing test maintenance or test-code review. - Each test is specific to the change being evaluated. - Tests automatically adapt as the code evolves. - Human attention is required mainly when an actual bug is detected. - Testing shifts from measuring generic code quality to determining whether a specific change introduces a real fault. Catching JiTTests are presented as a way to make testing scale with AI-assisted development by moving routine test creation and maintenance from engineers to automated systems.

Read original

Large Language Models Automated Testing Regression Testing Just In Time Testing+3