meta·Aug 12, 2026·3 min readCurated summary

How We’re Building Scam Alert on WhatsApp With End-to-End Encryption and Verifiability Guarantees

WhatsApp’s optional Scam Alert uses an on-device machine-learning model to identify likely scam messages without sending message content to WhatsApp, Meta, or third parties. The system is designed to preserve end-to-end encryption, give users control over warnings and reporting, and make its model and privacy safeguards independently reviewable. It is being introduced gradually in Beta while security researchers test its implementation. ## Design Principles - **On-device only:** The model and messages it analyzes remain on the user’s device. - **No automatic reporting:** WhatsApp receives message content or scam-detection information only if the user explicitly reports a chat. - **User control:** Users can enable or disable Scam Alert and decide how to respond to warnings. - Recent advances in mobile machine learning make it practical to run a small, reviewable text-classification model locally. ## How Scam Alert Works - After activation, the device downloads the model and analyzes incoming messages from non-contacts. - Classification is based on conversational structure, language signals, and patterns found in previously reported scam conversations. - When a message appears suspicious, the user sees a private warning visible only to them. - The user can: - Block the sender - Report the chat - Continue the conversation - Mark the chat as trusted - Trusted chats no longer receive Scam Alert warnings. Users may optionally share the last five received messages from a trusted chat to help improve accuracy. ## Foundational Safeguards - **Privacy-preserving analytics:** Only anonymous, aggregate warning and user-action counts are collected. - **Confidential computing:** Metrics are processed inside confidential virtual machines using trusted execution environments. - **No targeted model delivery:** WhatsApp cannot send a specific model to an individual user. - **Public transparency:** Every model version, including experimental versions, is recorded in a public transparency ledger before deployment. - **Verifiable behavior:** Model weights are published so researchers can confirm that the model is designed specifically to detect scams. ## Privacy-Preserving Analytics WhatsApp wants to measure whether Scam Alert catches scams accurately without collecting message content. The system therefore limits telemetry to two categories: - **Warning counts:** Approximate aggregate counts of how often the model displays warnings, helping measure detection rates and identify regressions. - **User action counts:** Aggregate counts of whether users trust, block, or report after receiving warnings, helping estimate false-positive rates. These metrics are protected using differential privacy, which adds carefully calibrated noise so that the presence or absence of one person’s data has negligible impact on the aggregate results. ## Confidential Federated Analytics - Devices aggregate local events before transmitting them; raw signals never leave the device. - Metrics contain no device identifiers, use coarse time intervals, and are sent at randomized times. - Data is encrypted between the device and the trusted execution environment. - Devices verify the environment’s software through hardware-backed attestations and a third-party record of approved binaries. - The confidential environment prevents WhatsApp, Meta, relays, and other intermediaries from accessing individual measurements. WhatsApp’s approach aims to provide scam detection without weakening message privacy: processing remains local, reporting remains user initiated, and system performance is measured only through minimized, privacy-protected aggregates. The feature is currently best viewed as an early Beta system whose effectiveness and security will depend on continued public review and bug-bounty testing.

Read original(opens in new tab)

Differential PrivacyEnd To End EncryptionConfidential ComputingFederated Analytics+3

meta·Aug 5, 2026·3 min readCurated summary

From User Sequences to Scaling Laws: A Multi-Stage Architecture for Meta’s Ads Ranking

Meta’s new sequence-learning platform improves ads recommendations by separating deep offline user modeling from fast online ranking. Combined with dense tokenization and target-aware attention, it enables richer behavioral representations, predictable compute-to-performance scaling, and major gains: 6% more Instagram conversions, 3% more Facebook conversions, and 3.5% more Facebook ad clicks. The system is also a core part of Meta’s Generative Ads Recommendation Model (GEM). ## Challenges of Earlier Sequence Models - Ads systems must rank thousands of candidates within milliseconds and process millions of candidates per second. - Hybrid architectures typically use: - One model for user event sequences. - Another for sparse feature interactions. - This design can cause: - Lossy knowledge transfer between components. - Continued dependence on manually engineered features. - Scaling limits caused by interference between sequence modeling and ranking. - Increasing sequence lengths and transformer capacity can therefore raise serving costs without delivering proportional improvements. ## Multi-Stage Sequence Modeling Meta separates sequence learning into two complementary stages: - **Offline user modeling** - Processes long user histories asynchronously. - Uses deep transformer models with thousands of events and multiple layers. - Produces cached, user-level embeddings that represent long-term behavioral patterns. - Keeps user features separate from ad and context features so embeddings remain independent of individual candidates. - **Online ranking** - Combines cached user embeddings with fresh user signals, ad features, and context. - Performs final ranking under strict latency requirements. - Uses a lightweight architecture optimized for real-time serving. This separation allows the offline model to grow in depth, width, and sequence length without proportionally increasing online serving costs. ## Dense Tokenization and Target-Aware Attention - **Dense tokenization** - Converts sparse features and sequential behavioral data into a shared dense vocabulary. - Allows the model to learn feature interactions directly instead of relying on manually engineered cross-features. - **Target-aware multi-head attention** - Combines user behavior sequences with the specific ad candidate being scored. - Lets each attention layer determine which past behaviors matter for that candidate. - Stacked attention blocks capture increasingly complex interactions and compress long histories into compact representations. - The approach is designed to be memory-efficient while preserving candidate-specific information. ## Predictable Scaling Laws - On real-world ads traffic, the architecture shows an LLM-like log-linear relationship between compute and recommendation performance. - Improvements were measured using normalized entropy across: - Model depth. - Model width. - Sequence length. - Content and semantic enrichment. - The scaling behavior suggests the architecture is well suited to continued investment in sequence learning, despite recommendation systems combining sparse IDs with temporal data rather than dense text. ## Scaling Strategies - **Balanced model shape** - Depth, width, and sequence length should grow together. - Scaling only one dimension can create bottlenecks and diminishing returns. - Meta calls this the “scaling synergy principle.” - **Multi-stage tunability** - Online models offer strong improvements per unit of compute but are constrained by request latency. - Offline models improve more gradually but can scale aggressively because inference is asynchronous. - **Sequence composition** - Longer sequences generally improve performance. - Diversity of actions is more valuable than simply adding more homogeneous events. ## Practical Conclusion Meta’s approach makes sequence learning more scalable and operationally practical by moving expensive user-history processing offline while retaining fast, target-specific ranking online. Dense tokenization and target-aware attention reduce manual feature engineering, while the observed scaling laws provide a framework for deciding where additional model capacity and compute will produce the greatest gains.

Read original(opens in new tab)

TransformerRecommendation SystemsScaling LawsAttention Mechanisms+3

meta·Aug 3, 2026·4 min readCurated summary

GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model

Meta’s Generative Ads Recommendation Model (GEM), which powers ad recommendations across Instagram and Facebook, now trains at LLM scale across several thousand GPUs. By co-designing kernels, numerical precision, parallelism, networking, and memory management, Meta doubled end-to-end training efficiency to 20–25% Model FLOPs Utilization (MFU) while increasing training compute fourfold in 12 months. The work shows that recommendation models require infrastructure specifically adapted to their hybrid architecture and data patterns rather than a direct reuse of LLM techniques. ## GEM’s Architecture and Training Challenges - GEM combines: - Trillions of sparse embedding parameters. - Billions of dense parameters. - Sequence features, such as user activity history. - Non-sequence features, such as user location and ad representations. - Different feature groups use customized attention mechanisms while still supporting cross-feature learning. - Recommendation workloads differ substantially from typical LLMs: - User histories have highly variable lengths, making padding inefficient and potentially wasting up to 50% of computation. - Attention patterns are asymmetric, including long sequences with short windows and long queries with short key/value sets. - Small embedding dimensions and normalization layers create memory-bound operations. - CTR and CVR optimization are numerically sensitive, so aggressive low-precision training can harm model quality. ## Scaling Across Thousands of GPUs - GEM’s distributed training latency is determined by the slowest rank and the larger of its local computation or communication time. - Efficient scaling requires: - Computation to dominate communication. - Communication to overlap with computation without resource contention. - Minimal activation recomputation. - Balanced workloads across GPU ranks. - GEM makes these requirements difficult because: - Trillion-scale sparse parameters generate substantial communication. - Different layer types provide uneven opportunities for communication overlap. - Long sequences and large activations pressure GPU memory. - Jagged inputs create changing load imbalance and stragglers. ## Separating Compute and Scaling Efficiency - Meta measures end-to-end efficiency with: - **E2E MFU = Local MFU × Scaling Ratio** - **Local MFU** measures how effectively one GPU uses its compute hardware, including Tensor Cores and memory hierarchies. - **Scaling Ratio** measures how much single-GPU performance is retained across thousands of GPUs. - This framework separates: - Kernel design and numerical precision issues affecting individual GPUs. - Parallelism, networking, memory, and load-balancing issues affecting distributed training. ## Compute-Efficiency Optimizations - Meta developed recommendation-specific GPU kernels, including: - Jagged Flash Attention (JFA) for variable-length sequences. - Generalized Dot-Product Attention (GDPA). - BlockAttention. - These kernels are designed around GEM’s irregular shapes and asymmetric attention patterns rather than conventional LLM assumptions. - Mixed ultra-low-precision training, including MXFP8 for attention and MLP layers, improves throughput while accounting for recommendation models’ numerical sensitivity. - The kernels and precision recipes are customized to exploit the architecture of the latest-generation GPUs. ## Scaling-Efficiency Optimizations - Meta uses topology-aware five-dimensional parallelism to distribute GEM efficiently. - Dense parameters use: - Two-dimensional Fully Sharded Data Parallelism (FSDP). - Expert Parallelism. - Sparse parameters use fully sharded two-dimensional model parallelism. - These strategies are co-designed with Meta’s multi-tier network hierarchy to reduce communication overhead. - Streaming Multiprocessor (SM)-free collectives help communication run with less interference from GPU computation. - The overall design targets communication overlap, memory constraints, load balance, and the differing behavior of dense and sparse parameters. ## Results - GEM’s end-to-end training efficiency increased to 20–25% MFU. - Efficiency doubled over a 12-month period. - Total training FLOPs increased fourfold. - The results demonstrate that recommendation foundation models can reach LLM-scale training, but only through coordinated hardware and software optimization across kernels, precision, parallelism, networking, and memory. For large recommendation models, LLM infrastructure provides a starting point but is not sufficient. The practical recommendation is to optimize compute and distributed scaling as separate but connected problems, using workload-specific kernels, carefully validated low precision, topology-aware parallelism, and communication strategies tailored to sparse and dense model components.

Read original(opens in new tab)

Large Language ModelsRecommendation SystemsDistributed TrainingGpu Optimization+3

meta·Jul 15, 2026·3 min readCurated summary

Exploring Hierarchical Interest Representation For Meta Ads Deep Funnel Optimization

Hierarchical Interest Representation is an upstream embedding layer for Meta Ads that connects users’ inferred interests with advertisers, products, and services. It combines engagement data, multimodal content, graph learning, and hierarchical abstractions to address sparse deep-funnel signals and rare or unseen entities. The resulting universal embeddings and “Bag-of-Meaning” interest tokens could support retrieval, personalization, supervision, and ranking across Meta’s advertising systems. ## Purpose and Role in Deep-Funnel Optimization - The system aims to identify people with genuine latent interest in an advertiser’s offerings. - It is intended to complement systems such as Meta’s Generative Ads Model (GEM), Andromeda, and the Adaptive Ranking Model. - It uses user behavior—including scrolling, engagement, and explicit “Interested/Not interested” feedback—to infer preferences. - The broader goal is to improve discovery-oriented ad experiences and downstream conversion performance. ## Technical Challenges ### Sparse Engagement and Large-Scale Graphs - Users, advertisers, products, services, and campaigns are modeled as graph nodes. - Interactions and activities form edges in a graph spanning millions of advertisers, millions of ads, and billions of users. - Deep-funnel feedback is relatively scarce, leaving many entities with limited direct evidence. ### Long-Range Relationships - Useful signals may come from indirectly connected users and entities rather than direct interactions. - Capturing these relationships at Meta’s scale requires memory-efficient sparse attention and high-performance graph-learning algorithms. ### Dynamic and Unseen Entities - The ads ecosystem changes rapidly, while individual entities may have little historical engagement. - Representations must generalize to rare and previously unseen businesses and products. ## Core Design Properties ### Dimension Reduction - The raw graph is projected into a configurable “super-graph.” - Learned latent interest primitives act as super-nodes. - Sparse user-ad relationships become denser connections at the interest level. - The primitive graph is more stable and stationary than the constantly changing ads vocabulary. ### Knowledge Enrichment - Advertiser and product representations incorporate text, images, video, metadata, and catalog attributes. - Vision and language models process this multimodal content. - Content helps the system understand what a product or business is, not merely how users interacted with it. - This enables better generalization to new or sparsely observed entities. ### Unified Relational Representation - Users, advertisers, products, and latent interest primitives are embedded in a shared metric space. - The system can estimate: - Relationships between interest primitives - Similarity between users, ads, and products - A user’s proximity to particular interests - Which interests an advertiser or product serves - Embedding operations support both primitive-to-primitive and cluster-to-cluster relationship modeling. ### Multiple Hierarchical Granularities - Coarse representations capture dense, stable, high-level interests. - Finer representations capture sparse and specific deep-funnel intent. - Cascading hierarchical layers allow the embeddings to serve different needs across retrieval, personalization, ranking, and supervision. ## Architecture and Training - The architecture combines: - An in-house transformer-based graph learner - Bias-aware attention - Self-supervised cross-view distillation - Sparse attention for long-range graph relationships - It combines real-world semantic knowledge with users’ temporal engagement histories. - The model learns multi-hierarchical interest representations across a large graph. - Training is performed end-to-end on real Meta Ads data involving billions of interactions. ## Outputs and Potential Applications - Universal embeddings for users and ads entities. - “Bag-of-Meaning” interest tokens representing latent interests at different granularities. - Potential uses include: - Ad retrieval - Personalization - Ranking - Specialized ranking architectures - Training supervision - Cross-entity similarity and discovery Hierarchical Interest Representation is best understood as shared infrastructure for Meta’s ads recommendation stack. By combining sparse behavioral evidence with multimodal world knowledge and hierarchical graph abstractions, it could make deep-funnel optimization more robust, especially for specialized, rare, or newly introduced products and advertisers.

Read original(opens in new tab)

Large Language ModelsMultimodal AITransformerEmbeddings+4

meta·Jul 13, 2026·3 min readCurated summary

Modernizing the Meta Ads Service With an Open-Source Kernel Scheduler

At Meta’s scale, small latency regressions can materially affect ad relevance, ranking, and revenue. When Linux kernel 6.9’s EEVDF scheduler reduced ad-serving performance, Meta used sched_ext to deploy a workload-specific BPF scheduler without modifying the kernel. The solution reduced p99 ads-retrieval latency by 28%, saved 3.28 MW of power, increased ads ranked by 1.1%, and enabled further improvements through rapid user-space updates. ## Why Ads Latency Matters - Meta’s ads platform processes more than 5 million requests per second, or over 400 billion requests daily. - Lower p99 latency allows more relevant ads to be retrieved and ranked within each request. - General-purpose schedulers such as CFS and EEVDF balance CPU usage without understanding which threads are most important to ad delivery. - Ads-specific scheduling can prioritize work on the critical request path while deferring less-sensitive tasks. ## The Kernel Upgrade Problem - During a move from Linux 6.4 to 6.9, Meta found that EEVDF introduced a latency regression. - The regression reduced the number of ads ranked in responses. - Some servers had to remain on Linux 6.4, creating operational fragmentation and technical debt. - sched_ext provided a way to address the regression without waiting for another kernel release. ## How sched_ext Customizes Scheduling - sched_ext is an upstream, BPF-based framework that entered Linux 6.12. - It lets developers implement scheduling policies in BPF programs responding to events such as: - Thread wake-ups - Run-queue insertion - Dispatching the next thread - CPU idle-state transitions - Meta’s policy divides CPUs into two dynamically sized pools: - Latency-critical request-path threads - Less latency-sensitive background work - Keeping related work on the same CPUs improves L3 cache locality and reduces DRAM access. - The scheduler is loaded by a user-space binary, so new policies can be deployed by restarting the scheduler process rather than rebuilding the kernel. ## Performance and Operational Results The initial deployment on the largest ads-serving server type achieved: - 28% lower p99 latency on the ads retrieval path - 1.1% more weighted ads ranked - 3.28 MW of fleet-wide power savings Two subsequent policy updates produced additional gains: - A further 60% reduction in service p99 latency - 18% fewer timeout errors on the critical path - Delivery in days instead of the months typically required for kernel changes ## From Fix to Optimization Platform - sched_ext gives Meta an independent scheduling-development path alongside upstream Linux evolution. - BPF updates support rapid experimentation with: - Cache-aware thread placement - ROI-based executor routing - NUMA-aware scheduling - Because sched_ext is upstream, other organizations can implement workload-specific policies without maintaining a Linux kernel fork. - Meta plans to use application-level hints, such as request importance, to adjust scheduling slices and queue priority dynamically. sched_ext demonstrates that application-aware scheduling can produce measurable business, latency, and energy benefits. For workloads with priorities that general-purpose schedulers cannot see, an extensible BPF-based scheduler offers a practical way to optimize continuously without coupling improvements to kernel release cycles.

Read original(opens in new tab)

LinuxEbpfLinux KernelNuma+4

meta·Jul 1, 2026·3 min readCurated summary

Meta’s AI Storage Blueprint at Scale

Meta argues that AI progress increasingly depends on storage that can deliver massive datasets with predictable, low latency. Traditional BLOB-storage designs optimized for durable, cost-efficient HDD storage create metadata and proxying bottlenecks that stall GPUs and slow research. Meta is therefore rebuilding its storage foundation around unified metadata, direct client-to-storage access, and regional deployments colocated with GPUs. ## Storage Architecture and AI’s Growing Demands - Meta operates hundreds of exabyte-scale storage clusters supporting products such as Facebook, Instagram, Meta AI, Ads, and internal databases. - Its storage APIs are built on Tectonic, a horizontally scalable block layer providing: - High durability and availability through erasure coding - HDD and flash tiering - Placement of hot, warm, and cold data - Multi-tenant regional storage - BLOB-storage layers built on Tectonic provide globally scalable object storage and configurable durability/availability policies. - Meta’s training systems historically used an NFS-like filesystem interface over Tectonic, but are increasingly moving to BLOB storage for unified access to massive data lakes and higher performance. ## Why Storage Latency Limits GPU Utilization - AI workloads require bursty and sustained high throughput with predictable worst-case latency. - Training runs use hundreds of thousands of GPUs processing data in batches and periodically synchronizing state. - A single slow GPU can delay synchronization and extend the completion time for every GPU. - Data loaders prefetch future batches while GPUs process current ones, but high-latency storage reads can still create GPU stalls. - These stalls directly increase training costs and extend time to market. ## Problems with the Legacy BLOB Architecture - The older service-oriented design accumulated multiple stateful layers, each with its own metadata store. - A single `getObject("/bucket/path")` request could require lookups across the namelayer, volumeslayer, and containerlayer. - Cross-region metadata requests could add hundreds of milliseconds, and one slow lookup could delay the entire operation. - The architecture’s original assumptions no longer matched AI requirements: - **Latency:** AI needs bounded pMax latency, not merely acceptable average performance. - **Reliability:** AI requires high availability, but does not always need global replication by default. - **Cost:** Flash is necessary for AI-level IOPS, making storage cost-per-byte less important. - **Power:** Power used by storage competes directly with power available for GPUs. ## Rebuilding the Storage Foundation Meta redesigned the system around three major changes: - **Unified metadata schema** - Metadata from separate layers was consolidated into a flat schema backed by ZippyDB. - Path resolution can now use O(1) lookups to map objects to `(blockId, offset, size)` locations. - **Direct data access** - The dataplane proxy was removed. - A “fat client” SDK streams data directly from Tectonic storage servers. - This reduces latency, increases throughput, and lowers storage power consumption. - **Regional deployment** - The BLOB stack can operate regionally or globally. - Regional instances are colocated with GPUs in AI regions, reducing cross-region access. With the new flow, the SDK requests a read plan from the API server, which performs the metadata lookup and returns storage locations. The SDK’s embedded Tectonic BlockClient then reads directly from the underlying blocks, adding essentially no extra dataplane overhead. The redesigned architecture is intended to improve GPU utilization, reduce latency, and preserve power for computation. The provided excerpt ends as Meta begins discussing how it handles workload spikes and hot spots during data and checkpoint loading.

Read original(opens in new tab)

Ai TrainingObject StorageDistributed StorageErasure Coding+3

meta·Jun 30, 2026·2 min readCurated summary

10 Years of Meta’s Commitment to Python

Meta marks its 10th consecutive year sponsoring the Python Software Foundation (PSF), emphasizing that Python is central to its infrastructure, products, and AI work. The company views sponsorship as both a responsibility to the open-source community and a strategic investment in the long-term health, security, and innovation of the technology it relies on. ## Python’s Role at Meta - Python is Meta’s most widely used programming language. - It supports infrastructure for products including Instagram and Threads, as well as AI research and data-driven initiatives. - Meta engineers contribute directly to Python’s development, including core maintenance and Python Enhancement Proposals. - Meta’s open-source contributions include: - PyTorch, originally developed at Meta before becoming an independent foundation. - Pyrefly, a fast Python type checker and language server. - Meta expects Python to remain important as it expands AI capabilities and scales its infrastructure. ## Why Meta Supports the PSF - Open-source adoption creates a shared responsibility to maintain a healthy, secure, and sustainable ecosystem. - PSF funding supports the Developer-in-Residence program, enabling full-time developers to work on Python improvements that might otherwise be neglected or left to volunteers. - Sponsorship helps strengthen PyPI, including critical security improvements that protect package distribution and consumption. - Funding also supports education and community development through: - PyCon US workshops, summits, and discounted or free passes. - Fundraising and support for groups such as PyLadies. - Meta considers these efforts an investment in the tools, infrastructure, and people behind its own technology stack. ## Ways to Support the Python Software Foundation - Individuals can make one-time donations or become PSF members. - Membership may include voting rights and can be supported through financial contributions or volunteer time. - Organizations can become annual sponsors at different contribution levels. - Sponsorship offers public recognition, community engagement opportunities, event participation, and—in higher tiers—greater visibility and invitations to special initiatives. Meta concludes by thanking Python’s maintainers, contributors, educators, and advocates, while encouraging other individuals and organizations to help sustain the language through PSF donations, membership, or sponsorship.

Read original(opens in new tab)

AIMachine LearningOpen SourcePython+3

meta·Jun 25, 2026·3 min readCurated summary

Privacy-Aware Infrastructure in the AI-Native Era: An Asset Classification Case Study

Privacy-aware infrastructure depends on accurate asset classification before it can enforce retention, access, purpose, sharing, or anonymization policies. Because data is noisy, distributed, and constantly changing—especially in AI-native systems—LLMs are useful for ambiguity but should not make routine production decisions. The recommended approach combines rich contextual evidence, human-reviewed labels, narrowly used LLMs, and versioned deterministic rules that are faster, replayable, and auditable. ## Why Asset Classification Matters - Assets include more than tables and columns: they may be nested payload fields, logs, event parameters, API fields, ML features, embeddings, or derived datasets. - Classification must track the meaning of data as it moves through pipelines and changes representation. - A field such as `age` could represent sensitive personal information or an infrastructure cache TTL, making context essential. - Four recurring challenges shape the problem: - **Noisy signals:** Raw metadata can overwhelm models and hide relevant evidence. - **Distributed context:** Code, lineage, ownership, documentation, annotations, and usage patterns reside in separate systems. - **Changing requirements:** Product and policy changes can outpace static rules and periodic reviews. - **Enforcement consequences:** False positives cause unnecessary restrictions, while false negatives create protection gaps. - Classification must reason about ambiguity while producing decisions that can later be explained and reproduced. ## The Hybrid Classification Pattern - **Context beats prompts:** Improving the evidence supplied to a model generally matters more than endlessly tuning instructions. - Evidence briefs should organize: - Supporting and contradicting signals - Provenance - Relevant code and lineage - Masked or circular fields that could distort reasoning - **Evaluation must remain independent:** Human-reviewed reference labels, frozen test sets, separate models or prompts, and regression gates prevent the classifier from defining its own ground truth. - **Stable behavior should be distilled into rules:** LLMs handle novelty and uncertainty, while validated patterns become deterministic, versioned, and auditable logic. - Over time, the LLM’s production role should shrink as routine cases move to low-latency deterministic enforcement. ## A Stable Classification Contract - The classifier should operate as a platform service with a small, explicit interface. - Inputs include: - An asset identifier - A structured bundle of contextual evidence - Outputs include: - A taxonomy category - A confidence score calibrated against reviewed labels - A decision trace explaining influential evidence - The matching deterministic rule, when applicable - Versions for the context, rules, and prompt - Classifiers should answer one scoped, domain-specific question rather than use a universal taxonomy. - Narrow classifiers are easier to evaluate, debug, govern, and compose across downstream privacy decisions. ## Privacy-Aware Infrastructure Responsibilities Asset classification supports the broader PAI lifecycle: - Understanding what data exists and how it is governed - Discovering data flows relevant to a policy - Enforcing retention, access, purpose, and sharing constraints - Producing verifiable evidence of compliance ## Practical Recommendation Use LLMs selectively for ambiguous or novel assets, but build the surrounding system around structured context, independent human-reviewed evaluation, and deterministic rule promotion. This preserves the flexibility of AI while making routine privacy enforcement predictable, auditable, and operationally efficient.

Read original(opens in new tab)

Large Language ModelsData GovernanceHuman In The LoopData Lineage+3

meta·Jun 23, 2026·3 min readCurated summary

How Meta Engineered Ultra-Narrow Batteries for AI Glasses

Smart glasses require batteries that fit inside extremely narrow temple arms while powering cameras, speakers, AI processing, and displays. Meta addressed this limitation by developing ultra-narrow steel-can cells, including batteries as thin as 7 mm, with redesigned electrode structures and tighter manufacturing tolerances. The approach increased capacity and peak-power performance while enabling different configurations across successive generations of Meta’s wearables. ## Why Traditional Pouch Cells Fall Short - Common in phones and laptops, pouch cells are difficult to shrink and reshape. - Folding, manufacturing tolerances, and wasted internal volume are especially costly in glasses. - Small pouch cells may also struggle to deliver peak power when several features operate simultaneously, such as recording video while an AI model processes a request. - Smart glasses instead need rigid, precisely shaped batteries that use nearly every available micron. ## Designing Ultra-Narrow Steel-Can Cells - Steel-can batteries are established in products such as watches and power tools, but Meta needed unprecedented widths down to 7 mm. - Engineers replaced the conventional wound “jelly roll” electrode with die-cut, stacked layers. - This architecture reduces impedance, helping prevent power drops or brownouts during simultaneous high-demand tasks. - Steel cans maintain their shape to approximately 100 microns, preserving usable space and improving energy density in narrow cells. ## Increasing Capacity Through System Design - The second-generation Ray-Ban Meta battery increased from 160 mAh to 210 mAh, about a 30% capacity increase. - The glasses nevertheless claimed roughly twice the runtime because of broader hardware and software improvements. - Gains came from better power management, tighter firmware control, and a form factor that accommodated a larger cell. - This demonstrates that battery life depends on the entire system, not chemistry alone. ## Managing Multiple Batteries and Higher Power Demands - Oakley Meta Vanguards use one battery in each temple arm. - Although the cells are symmetrical, the electrical loads are not evenly distributed. - Engineers had to address cross-charging risks and coordinate battery sequencing during startup and shutdown. - Meta Ray-Ban Display glasses created a sustained power demand because the display draws energy continuously rather than in short bursts. - They use a 248 mAh steel-can cell, the largest in Meta’s lineup. ## Scaling the Technology - Meta’s narrow steel-can design could support other wearable form factors beyond smart glasses. - The company is working to scale production across multiple vendors and build a more resilient supply chain. - Developing these cells required coordination among electrical, mechanical, firmware, manufacturing, and global collaboration teams. Meta’s steel-can battery technology shows how wearable battery improvements come from rethinking both cell construction and overall device engineering. For future compact wearables, precisely shaped, low-impedance cells combined with system-level power optimization offer a practical path to longer runtime and more demanding features.

Read original(opens in new tab)

WearablesSmart GlassesBattery TechnologySteel Can Cells+3

meta·Jun 22, 2026·3 min readCurated summary

Adopting AV1 for Real-Time Communication (RTC) at Scale

Meta’s adoption of AV1 for real-time communication has been a multi-year effort focused on codec efficiency, device compatibility, latency, power consumption, and resilience to poor networks. AV1 can reduce bitrate by at least 20% versus H.264/AVC while preserving quality, especially below 100 kbps and for screen content. However, making it practical for mobile video calls required low-complexity encoding, careful decoder selection, and systems designed to avoid freezes caused by bitrate spikes and packet loss. ## Why Meta Adopted AV1 for RTC - AV1 delivers comparable visual quality at substantially lower bitrates than H.264/AVC. - Offline testing showed at least a 20% bitrate reduction on low-end and mid-range devices under Meta’s product settings. - Lower bitrate is especially valuable in real-time calls, where network rates may fluctuate between 10 and 400 kbps. - At a 100 kbps limit, AV1 video remained much clearer than H.264/AVC, which appeared blurry. - AV1 is also well suited to screen sharing: - **Palette mode** efficiently represents frames with a limited number of colors. - **Intra-block copy** predicts repeated patterns within the same frame. - Both tools improve the readability of text and other computer-generated content. ## Challenges of Real-Time Video - RTC requires end-to-end latency below roughly 300 milliseconds; techniques common in video-on-demand, such as multi-pass encoding and extensive buffering, can introduce unacceptable delay. - Sudden bitrate increases can cause freezes. - Network bandwidth changes may require adjustments to resolution or frame rate. - Resolution changes generally require a key frame, producing a temporary bitrate spike. - Packet loss can trigger retransmissions or additional key frames, also increasing the risk of freezes. - Mobile devices must encode and decode video simultaneously, making power efficiency essential. ## Encoder Selection and Complexity - AV1’s advanced coding tools improve compression but can significantly increase encoding complexity. - An open-source AV1 encoder consumed 14% more power than H.264/AVC on a Pixel 8 during testing. - AV1 also used more memory, contributing to application crash regressions. - Meta therefore adopted an internal low-complexity AV1 encoder with power consumption comparable to H.264 baseline. - The encoder supports multiple presets: - Higher-complexity presets prioritize quality. - An ultra-low-complexity preset offers complexity comparable to H.264/AVC. - Meta selects the encoder preset according to device capabilities, allowing AV1 deployment beyond high-end phones. ## Decoder Selection - Although decoding is generally less demanding than encoding, AV1 decoding was still challenging for low-end mobile devices. - Initial tests found real-time decoding failures, video freezes, and audio/video synchronization problems on some devices. - Meta evaluated multiple open-source decoders and selected **dav1d** based on its power efficiency and reliability. Meta’s experience shows that AV1 adoption in RTC requires more than simply enabling a new codec. A practical deployment depends on device-specific complexity controls, efficient decoding, strict latency management, and mechanisms that prevent bandwidth changes or packet loss from interrupting calls.

Read original(opens in new tab)

Real Time CommunicationAv1Video CodecDav1d+4

meta·Jun 3, 2026·3 min readCurated summary

Lights Out, Systems On: Validating Instant Power Loss Readiness

Meta’s Instantaneous PowerLoss Storm is a disaster-readiness testing program designed to prepare data centers for sudden, zero-notice power loss. It extends existing fault-tolerance mechanisms across facilities, servers, storage, compute, and the Twine orchestrator, while addressing region-wide failures and autonomous recovery. Through incremental testing and carefully defined tradeoffs, Meta aims to make losing an entire region as manageable as losing a smaller fault domain. ## Defense-in-Depth for Instant Failures - Power-loss tolerance was built into the full data-center stack, including mechanical and electrical systems, server racks, storage, compute, and Twine. - Batteries and Power Loss Siren (PLS) preserve in-memory data when racks lose power. - Twine services use region-wide asynchronous unavailability events (UEs) to coordinate shutdown and recovery. - Existing mechanisms had been tested against smaller fault domains, but region-wide failures introduced new challenges involving scale, replica placement, and autonomous startup. ## Solving Region Bootstrap Problems - Restarting a region may require millions of services to start simultaneously and discover their dependencies. - Circular dependencies among Twine control-plane services—such as Scheduler, Allocator, Broker, and Zelos—could prevent the orchestrator from starting itself. - Belljar CI/CD tests continuously identify critical startup dependencies before deployment. - A Twine recovery kit, supported by Belljar and Twrko, provides a manual “jumpstart” mechanism for breaking unexpected dependency cycles. - Meta also encountered a “boomerang” problem in which UEs shut down the control-plane services responsible for generating and distributing those signals. - The simpler solution was to let control-plane services ignore power-related shutdown UEs, preventing orphaned services that could not be reaped or recovered. ## Balancing Reliability and Engineering Velocity - Absolute tolerance to instant power loss could require costly or overly complex infrastructure and might create false positives during normal operations. - Meta defined unacceptable impacts as: - Storage or database data loss - Permanent damage to data-center facilities - Sustained disruption beyond one region - The company accepted bounded risks such as transient service errors, limited rack failures, and temporary staleness in routing or region-availability information. - Issues were considered tolerable when they could be remediated after the incident within a reasonable mean time to respond (MTTR). ## Incremental Validation Through PowerLoss Storms - Because testing a full region carried significant risk, Meta validated readiness progressively: - Dependency tests in new and pre-production regions - Exercises in shadow regions that mirror production - Tests in small production regions - Full tests in large regions supporting storage, AI, and data-warehouse workloads - During a Storm, Meta injects a power-supply fault to immediately de-energize an entire region. - After a short, realistic MTTR, remedial drain actions isolate the region from global controllers and schedulers. - The tests avoid preemptive preparation so they accurately represent an unexpected power failure. - Repeated exercises train both systems and engineers to handle regional loss with the resilience normally expected from smaller fault domains. Meta’s approach is to expand disaster readiness gradually: define unacceptable consequences, build layered recovery mechanisms, test at increasing scale, and use each exercise to improve both architecture and operational practice.

Read original(opens in new tab)

CI/CDDistributed SystemsFault ToleranceDisaster Recovery+3

meta·May 26, 2026·4 min readCurated summary

SilverTorch: Index as Model — A New Retrieval Paradigm for Recommendation Systems

SilverTorch is a unified, GPU-based recommendation retrieval system designed to replace fragmented microservices with one integrated neural network. Its “Index as Model” architecture represents retrieval components—including item indices, filtering, reranking, and user modeling—as PyTorch modules. The system reportedly delivers up to 23.7× higher throughput and 20.9× better compute-cost efficiency than comparable CPU-based or traditional multi-service systems, while improving recommendation quality. ## Limits of Microservice-Based Retrieval - Traditional retrieval pipelines use separate services for: - Computing user embeddings - Finding similar content - Applying eligibility rules - Scoring and reranking candidates - An orchestrator coordinates these services before passing thousands of candidates to downstream ranking, all within roughly 100 milliseconds. - This architecture creates several structural problems: - **Data movement:** Network calls, serialization, and service coordination consume latency that could otherwise support more computation. - **Version inconsistency:** User models, item indices, and filtering rules may be updated independently, causing mismatches between user and item representations. - **Siloed engineering:** ML teams typically work in PyTorch while infrastructure teams work in C++, making improvements difficult to translate, test, and deploy. - GPU optimizations such as Faiss-GPU can accelerate individual services but do not eliminate the architectural overhead or enable deep coordination between components. ## Index as Model - SilverTorch replaces the service mesh with a single neural network. - Its central design principle, **Index as Model**, turns traditional retrieval artifacts into model components: - Item indices become tensors. - Eligibility filters become operators. - User towers, scoring layers, and rerankers become modules. - A single request passes through the integrated model, which: - Finds content relevant to the user’s interests - Applies language, geography, and policy constraints - Predicts multiple engagement outcomes - Produces a combined score for the final candidate set - This integration enables more complex models and larger candidate evaluations without exceeding the sub-100-millisecond latency target. ## Unified Retrieval Components - SilverTorch incorporates multiple functional regions within one model: - Approximate nearest-neighbor search identifies relevant items efficiently. - Eligibility filtering removes content that cannot be shown to a user. - Multi-task reranking predicts actions such as likes, shares, and comments. - Composite scoring combines these predictions into a final ranking signal. - Some components are hand-engineered, while others can be trained end-to-end through backpropagation. - From the runtime’s perspective, every component is a standard PyTorch `nn.Module`, regardless of whether it performs search, filtering, or learned prediction. ## Pure PyTorch Implementation - SilverTorch reimplements ANN search, Bloom-filter indexing, eligibility checks, neural reranking, and composite scoring as pure PyTorch modules. - The unified design requires: - Tensor-based data representation - Tensor-in, tensor-out operations - A consistent `nn.Module` interface - This allows modules to share memory, execution graphs, and compilation steps. - Engineers can co-design stages—for example, selecting promising clusters, filtering within them, and scoring only surviving candidates—instead of treating each operation as an isolated service. - The approach reduces the separation between ML and infrastructure engineering, allowing both groups to work within the same programmable layer. ## Performance and Scale - In an 80-million-item end-to-end evaluation, SilverTorch achieved: - **23.7× higher requests per second** than a strong traditional multi-service baseline using the same model architecture. - **20.9× better estimated total-cost-of-ownership efficiency** than a CPU-based solution. - The system is intended to support retrieval across multiple applications and large-scale feeds and video products. - Its increased efficiency makes neural reranking and multi-task engagement scoring practical within strict production latency budgets. SilverTorch’s main recommendation is architectural: consolidate retrieval into a single, composable model rather than optimizing disconnected services. Representing every retrieval stage as a PyTorch module can reduce overhead, improve consistency, enable deeper cross-stage optimization, and make more sophisticated recommendations feasible at scale.

Read original(opens in new tab)

MicroservicesRecommendation SystemsNeural NetworksPytorch+3

meta·May 13, 2026·1 min readCurated summary

Reel Friends: Building Social Discovery that Scales to Billions

Friend Bubbles may look like a simple Reels feature, but building it required substantial engineering work. The feature surfaces Reels that friends have watched or reacted to, relying on an evolving machine-learning model and platform-specific behavior. Meta engineers explain that a key, unexpected discovery ultimately helped make the experience work. ### What Friend Bubbles Does - Highlights Reels that a user’s friends have watched or reacted to. - Connects social activity with Reels recommendations in a more visible way. ### Engineering Challenges - The team had to evolve the machine-learning model powering the feature. - iOS and Android users exhibited different behaviors, requiring the team to account for platform-specific usage patterns. - The feature’s apparent simplicity concealed complex recommendation and product-engineering challenges. ### Podcast Discussion - Meta Tech Podcast host Pascal Hartig speaks with Facebook Reels engineers Subasree and Joseph. - They discuss the model’s development, differences between mobile platforms, and the surprising insight that helped the feature succeed. - The episode is available through Meta’s podcast channels and services including Spotify, Apple Podcasts, and Pocket Casts. The episode illustrates why seemingly straightforward social features can demand deep experimentation, modeling, and cross-platform engineering.

Read original(opens in new tab)

Machine LearningAndroidiOSSoftware Engineering+1

meta·May 12, 2026·3 min readCurated summary

Migrating Data Ingestion Systems at Meta Scale

Meta rebuilt its hyperscale MySQL data ingestion system to improve reliability, efficiency, and data-langing latency. The migration moved workloads from customer-owned pipelines to a simpler, self-managed warehouse service and ultimately transitioned 100% of jobs. Success depended on staged validation, continuous data comparison, and fast rollback mechanisms. ## Why Meta Migrated - The system incrementally moved several petabytes of social graph data from MySQL into Meta’s data warehouse each day. - This data supports analytics, reporting, machine learning, and product development. - The legacy architecture became increasingly unstable as data-landing requirements grew stricter. - Customer-owned pipelines worked at smaller scales but became difficult to manage reliably at hyperscale. ## Migration Success Criteria Each job had to meet defined requirements before advancing: - **Data correctness:** Old and new systems had matching row counts and checksums. - **Landing latency:** The new system performed at least as well as the legacy system. - **Resource usage:** Compute and storage consumption did not regress. - **Critical-table requirements:** Additional criteria were agreed upon with dependent teams. ## Three-Phase Migration Lifecycle ### Shadow Phase - New-system shadow jobs ran against the same production sources as existing jobs. - Their output was written to separate shadow tables. - Row counts and checksums were continuously compared with production data. - Compute and storage requirements were measured before production rollout. - Once validated in pre-production, shadow jobs were tested in production. ### Reverse Shadow Phase - The new system began writing to the production table. - The legacy system continued running, but wrote to a shadow table. - This preserved continuous comparison between both systems. - If discrepancies appeared, Meta could quickly restore the old system without rebuilding its configuration. ### Migration Cleanup - Both systems continued to be monitored for mismatches. - After validation, the legacy shadow job was removed. - The new system became the sole production pipeline. ## Data Quality and Debugging Tooling - Meta built tooling to compare corresponding table partitions from the two systems. - Comparisons included row counts, checksums, and example rows responsible for mismatches. - Mismatch records and debugging details were logged to Scuba for real-time analysis. - Hourly queries helped engineers identify root causes and determine whether issues were already known. - The same tooling remains part of post-migration release validation. ## Rollout and Rollback Controls - Both systems used change data capture (CDC), with internal full-dump and delta tables feeding customer-facing target tables. - Because CDC builds new data from previously landed data, an existing defect could propagate after migration. - Meta therefore emphasized: - Detecting problems before they reached data consumers. - Stopping further propagation quickly during rollback. - The reverse-shadow design provided early quality signals and preserved a ready-to-use legacy pipeline for rapid recovery. Meta’s migration demonstrates that large-scale infrastructure changes are safest when treated as controlled, observable lifecycle transitions rather than one-time cutovers. Parallel execution, automated data validation, explicit resource checks, and reversible rollouts enabled the company to migrate the entire workload while protecting downstream consumers.

Read original(opens in new tab)

Machine LearningMysqlData IngestionChange Data Capture+3

meta·May 11, 2026·2 min readCurated summary

Labyrinth 1.1: Making End-to-End Encrypted Backups Even More Reliable

Meta is rolling out Labyrinth 1.1, an updated encrypted storage protocol for Messenger. Its main improvement is more reliable end-to-end encrypted backups: messages can be backed up as they are sent, even when the recipient’s device is offline. This helps preserve message history after device loss, replacement, or long periods without signing in, while keeping messages unreadable to Meta and other parties. ## Labyrinth 1.1’s Backup Improvements - The new sub-protocol sends messages to the recipient’s encrypted backup immediately rather than waiting for their device to reconnect. - This addresses limitations in Messenger’s current encrypted backup process. - Backups remain protected by end-to-end encryption, so only the users involved in the conversation can access the message contents. ## How Message Encryption Works - Each message is wrapped with a message encryption key. - The sender places that key directly into the recipient’s encrypted backup. - The design is compared to putting a sealed envelope into a locked box that only the recipient can open. - Meta cannot read the stored messages or their encryption keys. ## Rollout and Results - Labyrinth 1.1 is being broadly rolled out across Messenger. - Meta reports that more messages are being backed up successfully. - More users are also restoring their complete message history when switching devices. The updated “Labyrinth Encrypted Message Storage Protocol” white paper provides the detailed technical specification.

Read original(opens in new tab)

End To End EncryptionEncrypted BackupsMessengerCryptographic Protocol+1