Transformer | Techlist.io

metaAug 5, 20263 min readCurated summary

From User Sequences to Scaling Laws: A Multi-Stage Architecture for Meta’s Ads Ranking

Meta’s new sequence-learning platform improves ads recommendations by separating deep offline user modeling from fast online ranking. Combined with dense tokenization and target-aware attention, it enables richer behavioral representations, predictable compute-to-performance scaling, and major gains: 6% more Instagram conversions, 3% more Facebook conversions, and 3.5% more Facebook ad clicks. The system is also a core part of Meta’s Generative Ads Recommendation Model (GEM). ## Challenges of Earlier Sequence Models - Ads systems must rank thousands of candidates within milliseconds and process millions of candidates per second. - Hybrid architectures typically use: - One model for user event sequences. - Another for sparse feature interactions. - This design can cause: - Lossy knowledge transfer between components. - Continued dependence on manually engineered features. - Scaling limits caused by interference between sequence modeling and ranking. - Increasing sequence lengths and transformer capacity can therefore raise serving costs without delivering proportional improvements. ## Multi-Stage Sequence Modeling Meta separates sequence learning into two complementary stages: - **Offline user modeling** - Processes long user histories asynchronously. - Uses deep transformer models with thousands of events and multiple layers. - Produces cached, user-level embeddings that represent long-term behavioral patterns. - Keeps user features separate from ad and context features so embeddings remain independent of individual candidates. - **Online ranking** - Combines cached user embeddings with fresh user signals, ad features, and context. - Performs final ranking under strict latency requirements. - Uses a lightweight architecture optimized for real-time serving. This separation allows the offline model to grow in depth, width, and sequence length without proportionally increasing online serving costs. ## Dense Tokenization and Target-Aware Attention - **Dense tokenization** - Converts sparse features and sequential behavioral data into a shared dense vocabulary. - Allows the model to learn feature interactions directly instead of relying on manually engineered cross-features. - **Target-aware multi-head attention** - Combines user behavior sequences with the specific ad candidate being scored. - Lets each attention layer determine which past behaviors matter for that candidate. - Stacked attention blocks capture increasingly complex interactions and compress long histories into compact representations. - The approach is designed to be memory-efficient while preserving candidate-specific information. ## Predictable Scaling Laws - On real-world ads traffic, the architecture shows an LLM-like log-linear relationship between compute and recommendation performance. - Improvements were measured using normalized entropy across: - Model depth. - Model width. - Sequence length. - Content and semantic enrichment. - The scaling behavior suggests the architecture is well suited to continued investment in sequence learning, despite recommendation systems combining sparse IDs with temporal data rather than dense text. ## Scaling Strategies - **Balanced model shape** - Depth, width, and sequence length should grow together. - Scaling only one dimension can create bottlenecks and diminishing returns. - Meta calls this the “scaling synergy principle.” - **Multi-stage tunability** - Online models offer strong improvements per unit of compute but are constrained by request latency. - Offline models improve more gradually but can scale aggressively because inference is asynchronous. - **Sequence composition** - Longer sequences generally improve performance. - Diversity of actions is more valuable than simply adding more homogeneous events. ## Practical Conclusion Meta’s approach makes sequence learning more scalable and operationally practical by moving expensive user-history processing offline while retaining fast, target-specific ranking online. Dense tokenization and target-aware attention reduce manual feature engineering, while the observed scaling laws provide a framework for deciding where additional model capacity and compute will produce the greatest gains.

Read original

Transformer Recommendation Systems Scaling Laws Attention Mechanisms+3

lineJul 16, 20263 min readCurated summary

Developing a Model to Assess Harmfulness from Open Chat Names and Descriptions

The AI Services Lab developed a model to automatically detect harmful LINE OpenChat names and descriptions, reducing the need for manual review. The project improved an existing moderation system by cleaning inconsistent labels, selecting a lightweight safety-tuned decoder model, and adapting it to predict both penalty levels and reasons. Granite Guardian 3.1 2B was ultimately fine-tuned with LoRA and deployed using token-probability-based inference. ## OpenChat Monitoring - Users must provide an OpenChat name and may add a description. - Names and descriptions are reviewed whenever they are created or modified. - LINE processes a large volume of global OpenChats, making fully manual moderation impractical. - The project aimed to: - Expand automated moderation to countries requiring more detailed judgments. - Improve accuracy in regions already using automation. - Reduce the amount of content requiring human review. ## Data Cleansing - Training data consisted of previously manually reviewed OpenChat names and descriptions. - Only records reviewed under the current moderation guidelines were used. - Identical name-description pairs sometimes had conflicting penalty outcomes. - Labels were consolidated using these rules: - Select the most severe penalty if it appeared at least twice. - If it appeared only once, treat it as possible noise and select the second-most-severe penalty. - Choose the most frequent penalty reason. - If reasons were tied, choose the globally rarer reason, following a TF-IDF-like principle that rarer reasons may be more specific. - This process produced a single, consistent label for each identical input. ## Selecting the Pretrained Model The team evaluated models according to four requirements: - Decoder-based architecture. - Fine-tuned for safety moderation. - Approximately 2 billion parameters. - Apache license for commercial use. Granite Guardian 3.1 2B was selected because: - It is designed to classify harmfulness through the probabilities of “Yes” and “No” tokens. - Restricting predictions to predefined tokens avoids unpredictable free-form responses. - Token probabilities provide confidence scores that can be thresholded for operational needs. - Its relatively small size supports lower serving costs and faster responses. ## Fine-Tuning for Penalty Prediction - A simple harmful/not-harmful classification was insufficient because moderation decisions include different penalty levels and reasons. - The model was trained to produce structured responses containing: - An `Action` penalty code. - A `Reason` penalty reason. - Cross-entropy loss was calculated only over the assistant’s response tokens, not the entire prompt. - This focuses training on predicting moderation decisions rather than reproducing the input text. - LoRA was used instead of full-parameter fine-tuning: - The base model parameters remained frozen. - Only small trainable matrices representing parameter updates were optimized. - This reduced memory and training costs while preserving pretrained capabilities. ## Inference Design - During inference, the model calculates logits for all possible next tokens. - The system extracts only the logits corresponding to valid penalty-code tokens, converts them to probabilities, and selects the highest-scoring code. - It then predicts the penalty reason in a second step. - Existing operational codes consisted of arbitrary letters and numbers that tokenized into multiple pieces. - To simplify probability calculations, penalty codes and reasons were mapped to meaningful natural-language tokens, each represented by a single tokenizer token. - KV caching was used between the penalty-code and penalty-reason predictions to improve efficiency. The resulting approach combines cleaned moderation labels, lightweight decoder-model fine-tuning, structured output targets, and constrained token-level inference. It is intended to broaden automated OpenChat moderation while maintaining the accuracy and response speed required for real-time LINE operations.

Read original

Transformer Pytorch Lora Tokenization+4

metaJul 15, 20263 min readCurated summary

Exploring Hierarchical Interest Representation For Meta Ads Deep Funnel Optimization

Hierarchical Interest Representation is an upstream embedding layer for Meta Ads that connects users’ inferred interests with advertisers, products, and services. It combines engagement data, multimodal content, graph learning, and hierarchical abstractions to address sparse deep-funnel signals and rare or unseen entities. The resulting universal embeddings and “Bag-of-Meaning” interest tokens could support retrieval, personalization, supervision, and ranking across Meta’s advertising systems. ## Purpose and Role in Deep-Funnel Optimization - The system aims to identify people with genuine latent interest in an advertiser’s offerings. - It is intended to complement systems such as Meta’s Generative Ads Model (GEM), Andromeda, and the Adaptive Ranking Model. - It uses user behavior—including scrolling, engagement, and explicit “Interested/Not interested” feedback—to infer preferences. - The broader goal is to improve discovery-oriented ad experiences and downstream conversion performance. ## Technical Challenges ### Sparse Engagement and Large-Scale Graphs - Users, advertisers, products, services, and campaigns are modeled as graph nodes. - Interactions and activities form edges in a graph spanning millions of advertisers, millions of ads, and billions of users. - Deep-funnel feedback is relatively scarce, leaving many entities with limited direct evidence. ### Long-Range Relationships - Useful signals may come from indirectly connected users and entities rather than direct interactions. - Capturing these relationships at Meta’s scale requires memory-efficient sparse attention and high-performance graph-learning algorithms. ### Dynamic and Unseen Entities - The ads ecosystem changes rapidly, while individual entities may have little historical engagement. - Representations must generalize to rare and previously unseen businesses and products. ## Core Design Properties ### Dimension Reduction - The raw graph is projected into a configurable “super-graph.” - Learned latent interest primitives act as super-nodes. - Sparse user-ad relationships become denser connections at the interest level. - The primitive graph is more stable and stationary than the constantly changing ads vocabulary. ### Knowledge Enrichment - Advertiser and product representations incorporate text, images, video, metadata, and catalog attributes. - Vision and language models process this multimodal content. - Content helps the system understand what a product or business is, not merely how users interacted with it. - This enables better generalization to new or sparsely observed entities. ### Unified Relational Representation - Users, advertisers, products, and latent interest primitives are embedded in a shared metric space. - The system can estimate: - Relationships between interest primitives - Similarity between users, ads, and products - A user’s proximity to particular interests - Which interests an advertiser or product serves - Embedding operations support both primitive-to-primitive and cluster-to-cluster relationship modeling. ### Multiple Hierarchical Granularities - Coarse representations capture dense, stable, high-level interests. - Finer representations capture sparse and specific deep-funnel intent. - Cascading hierarchical layers allow the embeddings to serve different needs across retrieval, personalization, ranking, and supervision. ## Architecture and Training - The architecture combines: - An in-house transformer-based graph learner - Bias-aware attention - Self-supervised cross-view distillation - Sparse attention for long-range graph relationships - It combines real-world semantic knowledge with users’ temporal engagement histories. - The model learns multi-hierarchical interest representations across a large graph. - Training is performed end-to-end on real Meta Ads data involving billions of interactions. ## Outputs and Potential Applications - Universal embeddings for users and ads entities. - “Bag-of-Meaning” interest tokens representing latent interests at different granularities. - Potential uses include: - Ad retrieval - Personalization - Ranking - Specialized ranking architectures - Training supervision - Cross-entity similarity and discovery Hierarchical Interest Representation is best understood as shared infrastructure for Meta’s ads recommendation stack. By combining sparse behavioral evidence with multimodal world knowledge and hierarchical graph abstractions, it could make deep-funnel optimization more robust, especially for specialized, rare, or newly introduced products and advertisers.

Read original

Transformer Large Language Models Multimodal AI Embeddings+4

googleJun 30, 20263 min readCurated summary

Introducing TabFM: A zero-shot foundation model for tabular data

TabFM is a foundation model for tabular classification and regression that uses in-context learning instead of dataset-specific training. It processes an entire table in one forward pass, eliminating much of the usual hyperparameter tuning and feature engineering. The authors report that it performs competitively with or better than tuned tree-based models, with planned integration into BigQuery. ## The Challenge of Traditional Tabular ML - Tabular prediction supports applications such as churn prediction and fraud detection. - Algorithms including XGBoost, AdaBoost, and random forests remain widely used. - Building reliable models typically requires: - Hyperparameter optimization - Cross-validation - Domain-specific feature engineering - Repeated model training for each dataset ## Tabular Prediction Through In-Context Learning - TabFM treats tabular prediction as an in-context learning problem. - Users provide historical training rows and target rows together as a single input. - The model learns relationships between rows and columns during inference without updating model weights. - This enables zero-shot predictions on previously unseen datasets. ## Hybrid Architecture TabFM combines ideas from TabPFN and TabICL to address the two-dimensional, order-independent nature of tables. - **Alternating row and column attention** - Attends across both features and examples. - Learns feature interactions and dependencies directly from the table. - Reduces the need for manually engineered features. - **Row compression** - Converts the contextualized information for each row into a dense vector. - **In-context Transformer** - Operates on compressed row representations rather than the full table grid. - Reduces computation and supports larger datasets efficiently. ## Training with Synthetic Data - Large, diverse real-world tables are difficult to obtain because they often contain proprietary or sensitive information. - TabFM is trained entirely on hundreds of millions of synthetic datasets. - Structural causal models generate tables using varied random functions and relationships. - The goal is to reproduce the broad range of distributions and feature interactions found in real-world data. ## Benchmark Results and Model Variants - Evaluation used TabArena, an Elo-based benchmark covering: - 38 classification datasets - 13 regression datasets - Dataset sizes from 700 to 150,000 samples - **TabFM** - Works out of the box. - Produces predictions in one forward pass without tuning or cross-validation. - **TabFM-Ensemble** - Adds cross features and SVD-derived features. - Uses a 32-model ensemble with non-negative least squares to optimize weights. - Applies Platt scaling for classification calibration. - The authors state that TabFM consistently outperforms heavily tuned conventional supervised algorithms. ## Availability and BigQuery Integration - TabFM is available through Google’s Hugging Face and GitHub repositories. - Google plans to integrate it into BigQuery. - Users will be able to run regression and classification through an `AI.PREDICT` SQL command without specialized machine-learning expertise. TabFM’s main promise is to make high-quality tabular prediction as simple as supplying a dataset and requesting predictions. Its zero-shot approach could reduce the engineering burden of traditional workflows, though practical adoption will depend on benchmark reproducibility, inference costs, and performance across specific production datasets.

Read original

Transformer Synthetic Data Hugging Face Xgboost+4

netflixJun 29, 20263 min readCurated summary

GenPage: Towards End-to-End Generative Homepage Construction at Netflix

GenPage is Netflix’s end-to-end generative approach to building personalized homepages. Instead of separately ranking rows and items, one transformer autoregressively generates the entire page—including rows, entities, and layout—from user and request context. In production, it outperformed Netflix’s mature multi-stage recommender on a core engagement metric while reducing serving latency by 20%. ## Reframing Homepage Recommendation - Netflix’s homepage is a personalized two-dimensional structure, not a single ranked list. - Traditional systems use separate candidate-generation and ranking stages for rows and entities. - GenPage treats homepage construction as a prompt-response task: - The prompt contains user history, profile information, and request context. - The response is the complete homepage generated autoregressively. - The approach aims to: - Replace complex multi-stage pipelines with one end-to-end model. - Optimize the whole page using reinforcement learning. - Capture interactions such as diversity and the trade-off between high-value rows and continued browsing. - Scale more predictably with additional data, compute, and model capacity. - Support new content types, layouts, UI components, and personalized artwork with fewer architectural changes. ## Production Challenges and Results - Real-time generation makes serving latency a major constraint. - The system must address: - Cold-start entities in a constantly changing catalog. - Shifting user interests and cultural trends. - Product and business rules that constrain generated pages. - An online A/B test against Netflix’s optimized production recommender produced: - Statistically significant improvement on Netflix’s primary launch engagement metric. - A 20% reduction in end-to-end serving latency. - Offline experiments found that: - Improving the prompt helped more than increasing model capacity in the tested regime. - Reinforcement-learning post-training improved homepage diversity, even though diversity was not an explicit objective. ## Tokenizing Context and Pages - Each training example contains: - **Context:** user history, profile attributes, and request information. - **Page:** displayed rows and entities in layout order. - **Feedback:** interactions such as plays, thumbs-up, and abandonment. - Context and page are tokenized as model inputs and outputs. - Feedback is used to derive reward and supervision signals rather than being directly generated. ## Domain-Specific Tokenization - GenPage uses a custom recommender-system tokenizer instead of a general-purpose text tokenizer. - This reduces sequence length and improves inference cost and latency. - For example, an action such as watching *Orange Is the New Black* can be represented with four tokens: - Entity ID - Action type - Time bucket - Duration bucket - Direct token mappings to product concepts, such as rows and entities, also make it easier to enforce generation rules and business constraints. ## Context Representation - User-history tokens encode: - Action type - Entity ID - Timestamp - Duration - The history includes explicit signals, such as playback, adding titles to My List, and thumbs-up, as well as implicit signals such as trailer views and detail-page visits. - Profile tokens represent attributes including language and profile type. - Request-context tokens include time of day, day of week, and device. - Long data sources, such as complete impression histories, are summarized to control sequence length and cost. - These summaries improve practicality but introduce handcrafted prompt engineering; learning to compress such information end to end remains a future direction. - Special segment markers help the model distinguish between different context sources.

Read original

Transformer Generative AI Reinforcement Learning Personalization+2

googleJun 26, 20263 min readCurated summary

Accelerating Gemini Nano models on Pixel with frozen Multi-Token Prediction

Google introduces a way to add Multi-Token Prediction (MTP) to already-deployed, frozen Gemini Nano models, accelerating on-device generation without a separate drafting model. The approach attaches a lightweight Transformer head to the existing model, reuses its hidden states and KV cache, and preserves identical final outputs through verification. On Pixel 9 and 10 devices, it delivers faster generation, lower memory use, and reduced energy consumption for features such as Notification Summaries and Proofread. ## The Mobile Inference Bottleneck - Autoregressive models generate one token at a time, creating latency and underusing mobile hardware. - Phones face strict RAM and energy constraints that make conventional acceleration techniques difficult. - A standalone speculative-decoding drafter consumes additional memory and must independently process the prompt. ## A “Late Exit” MTP Strategy - Speculative decoding uses: - A small drafter to propose several tokens. - The large model to verify those tokens in parallel. - MTP replaces the separate drafter with a lightweight Transformer head attached near the end of the main model. - The head uses the backbone’s high-dimensional activations to predict future tokens, benefiting from semantic context already computed by the larger model. ## Retrofitting a Frozen Backbone - Google freezes the fully trained Gemini Nano v3 weights and trains only the attached MTP head. - This avoids retraining or fine-tuning the production foundation model. - Incorrect draft tokens are discarded during verification, so the final output remains bit-for-bit identical to the original model. - The method therefore improves efficiency without changing the model’s capabilities or safety alignment. ## Zero-Copy Memory Architecture - The MTP head cross-attends directly to the backbone’s existing KV cache instead of maintaining a duplicate cache. - This eliminates separate prompt-prefill work for the drafter. - It also removes redundant embedding tables, attention variants, and application-specific tuning parameters. - Compared with a standalone drafter, the design saves up to 130 MB per instance. ## Accuracy and Speed Improvements - Access to the backbone’s richer internal representations makes MTP predictions more accurate than those of similarly sized standalone drafters. - Instruction-following tasks such as summarization and rewriting show especially strong gains. - For predictable formats such as smart replies, token acceptance improved by up to 55%. - Pixel 9 experiments showed speedups of 50% or more depending on the task. ## Production Impact - The updated inference stack coordinates drafting and verification on Pixel 9 and 10 devices. - In workloads including AI Notification Summaries and Proofread, MTP predicts nearly two additional tokens per inference pass on average. - Fewer verification cycles reduce processor wake-ups, improving latency and battery efficiency. MTP provides a practical way to accelerate existing on-device models without maintaining a separate drafter for every application. Reusing the frozen backbone’s computations and memory state makes it particularly well suited to mobile deployments where RAM, energy, and backward compatibility are critical.

Read original

Transformer Kubernetes Edge Computing Speculative Decoding+3

lineJun 24, 20264 min readCurated summary

Designing a Semantic Context OS: Beyond Token Stuffing in Agent Systems

The article argues that larger LLM context windows do not automatically produce better software-engineering agents. In long-running workflows, indiscriminately filling the context window can cause attention dilution, context rot, reasoning failures, and potential data exposure. It proposes a “Semantic Context OS,” a local runtime layer that actively governs context as a finite, structured system resource rather than treating it as an unmanaged text stream. ## The Context Window Is Not RAM - The article uses the “Karpathy metaphor”: - The LLM acts like a CPU: a largely stateless inference engine driven by pretrained parameters. - The context window acts like RAM: volatile working memory containing current state, instructions, telemetry, and runtime data. - Unlike physical RAM, LLM context is probabilistic rather than deterministic: - Traditional RAM provides precise address-based retrieval with predictable performance. - LLM retrieval depends on attention weights across Q, K, and V matrices. - Increasing capacity from 32K tokens to 1M or 2M tokens therefore does not guarantee proportionally better retrieval. Larger sequences also increase computational cost and structural noise. ## Attention Dilution and Long-Context Failure - Large codebases and logs contain substantial irrelevant material, including: - Boilerplate definitions - Unused imports - Duplicate syntax - Repeated utilities and naming patterns - As sequence length grows, the attention calculation `QKᵀ` accumulates entropy and background noise. - Softmax then spreads attention energy across more tokens, weakening the sharp attention peaks needed to retrieve important facts. - This contributes to the “lost in the middle” effect: - Information near the beginning and end of a prompt is often retrieved more reliably. - Retrieval accuracy can fall sharply across the middle portion of the context. - The article considers relying on massive, unmanaged contexts an architectural anti-pattern for tasks such as large-scale code review, dependency tracing, and automated refactoring. ## Context Rot in Long-Running Agents The article defines “context rot” as the degradation of an agent’s working context during extended autonomous tasks. - **Context poisoning** - Raw logs, obsolete errors, and previous execution data accumulate over multiple turns. - The model may treat temporary historical failures as current architectural constraints. - **Context distraction** - Monorepos often contain similar names, overloaded methods, and duplicated helper code. - Broad retrieval can overwhelm the model with structurally similar but logically irrelevant code. - **Context clash** - Old instructions may remain after the plan has evolved. - Contradictory directives can cause indecision, infinite reasoning loops, timeouts, or hallucinations. - The article claims that, without active management, failure rates increase nonlinearly with context depth and may reach roughly 40% in deeply nested codebases. ## Semantic Context OS as an AI Kernel The proposed Semantic Context OS sits between agent application logic and external foundation-model APIs, operating as a localhost loopback proxy at `localhost:8080`. Its responsibilities include: - Treating context as a finite hardware-like resource. - Tracking token lifecycles and state access. - Filtering and isolating data before it reaches the model. - Separating physical token limits from semantic governance. - Protecting downstream inference engines from structural noise and helping prevent intellectual-property leakage. The architecture includes: - A POSIX-like virtual file system for managing state topology. - A proprietary “PathAlign” stage for AST-based code-tree pruning. - An asynchronous “sawtooth” memory model for runtime token optimization. ## MVC: Minimum Viable Context The core MVC pipeline—described as “minimum viable context”—aims to provide only the smallest dense set of information required for the agent’s current reasoning step. Its processing stages include: - **Collection and token mapping** - Gather source files, dependency graphs, and runtime logs. - Map them using the target model’s tokenizer, such as `cl100k_base` or `o200k_base`. - **Structural pruning** - Use static analysis and structural rules to remove compiler comments, unused imports, boilerplate, and unrelated utilities. - The broader design replaces passive string concatenation with active context selection, lifecycle management, and bounded transmission policies. The article concludes that reliable enterprise agents require active context orchestration rather than larger prompts alone. A dedicated governance layer should prune, isolate, and refresh context throughout execution so that models receive minimal, relevant, and internally consistent information.

Read original

Transformer Large Language Models AI Agents Abstract Syntax Tree+4

netflixJun 23, 20263 min readCurated summary

Toward More Controllable AI Video Editing: An Early Research Exploration at Netflix

Netflix explores AI video-editing tools designed to preserve artists’ creative control rather than regenerate entire clips indiscriminately. The research addresses two major problems: unintended changes to untouched footage and physically implausible results when objects are removed. Its proposed systems, Vera and VOID, generate targeted edits while preserving scene identity, performance, and continuity. ## Challenges in Generative Video Editing - Full-video regeneration can unintentionally change: - Actors’ identities and performances - Backgrounds and objects - Important scene details - Object removal often produces unnatural results because models erase the target without reconstructing realistic motion and physical interactions. - Professional editors need precise control over what changes and what remains untouched. ## Vera: Layered Video Diffusion - Vera generates: - An edit layer containing the requested visual change - An alpha matte defining where that change should appear - These layers are composited with the original footage, leaving pixels outside the edited region intact. - The approach supports tasks such as: - Adding objects - Changing backgrounds - This layered design helps preserve original identities, performances, and details. ## Training Dataset - Netflix created a custom dataset because existing public datasets lacked high-quality layered video data. - The dataset contains 486,000 frames at 832×480 resolution. - It includes: - **Synthetic composites:** Foreground objects with alpha mattes placed over generated backgrounds. - **Realistic single-object videos:** Real footage processed with segmentation, matting, background generation, and human review. - **Realistic multi-object videos with effects:** Objects isolated along with shadows, reflections, and other scene effects. ## Vera’s Model Architecture - Vera uses a Mixture-of-Transformers design with three specialized DiTs for: - The edit layer - The alpha matte - The composite video - Each branch has its own attention projections and feed-forward weights, allowing specialization while joint attention enables communication between layers. - The model is initialized from a pretrained text-to-video model. - Additional embeddings and input layers help distinguish source-video, mask, alpha, and composite information. ## Evaluation and Results - Netflix tested Vera on: - 72 object-addition video-prompt pairs - 69 background-change pairs - The benchmark included varied motion speeds, camera movements, object counts, and scene complexity. - Evaluation measured: - Preservation of untouched content - Compliance with text instructions - Temporal and per-frame video quality - Vera-1.3B and Vera-14B substantially outperformed existing methods on content preservation while achieving comparable instruction-following and visual quality. Netflix’s research favors localized, layered editing over unrestricted video regeneration. Vera demonstrates how separating edits from original footage can make generative tools safer and more controllable for professional workflows; the accompanying VOID research aims to apply similar principles to physically plausible object and interaction removal.

Read original

Transformer Generative AI Computer Vision Diffusion Models+4

daangnFeb 27, 20265 min readCurated summary

How will long-term user modeling

Long-term user modeling captures persistent interests, cross-vertical behavior, and signals beyond what short-term recommendation logs can reveal. 당근 built a Transformer-based user encoder that learns from tens of billions of actions across its local marketplace, jobs, real estate, and other services, then exposes the resulting embedding as a shared feature for ranking, retrieval, and advertising models. The approach improved scalability and reuse, but introduced freshness and representation-transfer limitations. ## Why Long-Term User Modeling Matters - Recent actions reveal immediate intent, but miss recurring interests such as seasonal shopping or repeated moving-related searches. - Long-term, cross-vertical activity can connect behaviors such as: - Searching for real estate - Looking for furniture and appliances - Reading neighborhood moving advice - Longer histories can reduce selection bias caused by training only on items previously exposed by recommendation models. - Simply adding more history is insufficient because ranking systems are latency-sensitive and long sequences increase computation and infrastructure complexity. ## Shared User Embeddings as a Common Feature - A separate user encoder processes long-term history offline. - Home-feed ranking, candidate generation, and advertising models consume the resulting embedding as a shared user feature. - Benefits: - Downstream models avoid directly processing massive histories. - The encoder can scale independently in model size, data, and compute. - One embedding can be reused across multiple recommendation surfaces. - Limitations: - A downstream model receives only a fixed vector, so it cannot fully exploit the encoder’s richer representations. - Batch inference means recent actions are not reflected immediately. - Possible future improvements include more frequent or real-time updates, fine-tuning, and distillation. ## Contrastive User Modeling - The encoder uses a two-tower architecture: - A causal Transformer converts the user’s action sequence into a user embedding. - An MLP converts item features into item embeddings. - InfoNCE loss trains the user embedding to predict the next interacted item. - In-batch negatives provide alternative items for contrastive learning. - Training uses clicks and conversion actions across all major verticals and surfaces. - The dataset contains tens of billions of actions—around 150 times more than the existing home-feed candidate model’s training data. ## Item ID Embeddings vs. Content Embeddings ### Problems with Item ID Embeddings - New items have no learned ID embedding, creating a cold-item problem. - Hundreds of millions of item IDs require enormous embedding tables. - In the ID-based model, embedding tables accounted for over 99% of parameters, leaving little GPU capacity for the Transformer. - Hashing and embedding-sharding techniques were considered but did not provide a sufficient solution. ### Content Embeddings - The system switched to LLM-generated embeddings based on post metadata. - This enables: - Representations for newly created items - Much larger Transformer models, with Transformer parameters becoming roughly 1,000 times larger than in the ID-based setup - Large-scale lookup required two memory-efficient techniques: - `memmap` loads only needed embedding segments from disk and benefits from shared OS page caches during distributed training. - `bbhash` maps item IDs to embedding locations using roughly three bits per key, reducing mapping memory by about 97% compared with Python dictionaries. - Together, these methods made training with hundreds of gigabytes of item embeddings practical. ## Region-Constrained Batch Sampling - Standard in-batch negatives assume that other items in the batch were visible but not selected. - This assumption fails in a local service: users generally cannot view items outside their geographic area. - More than 86% of transactions occur within five kilometers, yet random batches mixed users and items nationwide. - Consequently, about 98% of random in-batch negatives were “impossible negatives”—items users could never have seen. - These negatives teach geographic unavailability rather than user preference, weakening the contrastive signal. ### RCBS Solution - Region-Constrained Batch Sampling (RCBS) groups users from the same region into a batch. - This reduced impossible negatives from 98% to 30%. - The remaining impossible negatives mainly came from differences in viewing radius or users’ historical activity in other regions. - Feasible negatives are harder because they represent items users could have viewed but rejected, forcing the model to distinguish genuine preferences among similar local items. ### Why Sampling Was Better Than Masking - Masking impossible negatives would remove most of the batch, drastically reducing effective batch size. - Hard-negative mining would require checking feasibility separately for each user and could be expensive and complex. - RCBS naturally produces more feasible and difficult negatives without changing the loss function or adding specialized mining. ## Applying the Embeddings - For home-feed and advertising ranking, the embedding is projected and concatenated with existing features. - The long-term encoder supplies persistent preference signals, while existing ranking models continue handling short-term and real-time signals. - For retrieval models, the best-performing approach used the user embedding alone to generate candidates rather than merely adding it as another feature. - The separate candidate source appeared to improve recommendation diversity. ## Embedding Refresh and Serving - Offline tests showed little difference between frozen embeddings and 12- or 24-hour refreshes. - Online A/B tests favored periodic updates, with shorter intervals performing better. - A 24-hour refresh cycle was selected as the best cost-performance trade-off. - GPU inference runs through a Beam pipeline on GCP Dataflow. - Only users who acted during the refresh window are reprocessed, avoiding unnecessary inference for inactive users. - Near-real-time inference remains a major future engineering challenge. The overall recommendation is to treat long-term user modeling as a separate, reusable representation system rather than forcing every downstream model to process extensive histories directly. For geographically constrained services, the training data pipeline—especially negative sampling—must reflect actual item visibility, making region-aware batching as important as the model architecture itself.

Read original

Transformer Contrastive Learning Recommendation System Two Tower Model+4

netflixFeb 24, 20263 min readCurated summary

MediaFM: The Multimodal AI Foundation for Media Understanding at Netflix

Netflix’s Media Foundational Model (MediaFM) is a tri-modal AI system that combines video, audio, and timed text to understand long-form entertainment. It represents sequences of shots while using title-level metadata and temporal context to produce richer content embeddings. Netflix concludes that these contextual embeddings improve many downstream tasks, including advertising relevance, clip selection, tone classification, and popularity prediction. ## Motivation for MediaFM - Netflix needs machine-readable understanding of its expanding catalog, including films, series, live events, and podcasts. - Long-form media requires recognizing narrative dependencies, emotional arcs, scene transitions, and subtle tones across entire episodes or films. - Combining visual, audio, and textual signals provides a more complete understanding than relying on video alone. - The resulting embeddings support applications such as: - Cold-start recommendations for new titles - Promotional art and trailer optimization - Advertising relevance - Clip tagging and internal content analysis ## Multimodal Input Representation - The model uses a shot as its fundamental unit, with titles segmented using shot-boundary detection. - Each shot receives three modality-specific embeddings: - **Video:** Frames sampled from the shot are encoded with SeqCLIP, Netflix’s video-retrieval model. - **Audio:** Sound is encoded using Meta FAIR’s wav2vec2. - **Timed text:** Captions, subtitles, or audio descriptions are encoded with OpenAI’s `text-embedding-3-large`. - The three embeddings are concatenated and unit-normalized into a 2,304-dimensional fused vector. - Training examples consist of temporally ordered shot sequences from a movie or episode, with up to 512 shots. - Title metadata, such as synopses and tags, is also embedded and supplied as global context. ## Transformer Architecture - MediaFM uses a BERT-like Transformer encoder. - Fused shot embeddings are first projected into the model’s hidden dimension. - Two special tokens are prepended: - `[CLS]`, a learnable sequence-level embedding - `[GLOBAL]`, containing projected title-level metadata - Positional embeddings and self-attention allow each shot representation to incorporate surrounding narrative context. - A final projection maps contextualized representations back into the original 2,304-dimensional embedding space. ## Masked Shot Modeling - The model masks 20% of shot embeddings in each training sequence. - Masked inputs are replaced with a learnable `[MASK]` embedding. - The Transformer must reconstruct the original fused embedding for each masked shot. - Training minimizes cosine distance between predicted and ground-truth embeddings. - Hidden parameters are optimized with Muon, while other parameters use AdamW; Netflix reports noticeable gains after adopting Muon. ## Evaluation Through Linear Probes - Netflix evaluates MediaFM by freezing its representations and training task-specific linear layers on top. - Most evaluation tasks involve short clips extracted from larger titles. - Embedding a clip within the context of its surrounding episode or film performs better than embedding the clip in isolation, demonstrating the value of long-range contextualization. ## Downstream Applications - **Ad relevancy:** Multilabel classification identifies clips suitable for relevant advertising; MediaFM helps retrieve candidate clips before ad-serving optimization. - **Clip popularity ranking:** The model predicts relative clip performance and click-through rate within a title, evaluated using Kendall’s tau. - **Clip tone:** Clips are classified into 100 categories, such as creepy, scary, or humorous. - **Clip genre:** Clips are assigned to core genres including Action, Comedy, Documentary, Drama, Horror, Romance, and Thriller. - **Clip retrieval:** The system distinguishes “clip-worthy” content from unsuitable clips based on human annotations, using Average Precision. MediaFM’s main practical lesson is that effective media understanding depends on fusing all available modalities and preserving long-form temporal context. Netflix’s approach provides a reusable embedding foundation for recommendation, promotion, advertising, and content-analysis systems rather than building a separate representation for every task.

Read original

Transformer Database Design Multimodal Video Embedding+4

pinterestJan 28, 20263 min readCurated summary

Ads Candidate Generation using Behavioral Sequence Modeling

Pinterest’s Ads team uses behavioral sequence modeling to improve ad candidate generation by predicting what users are likely to convert on next. Transformer-based two-tower models first predict relevant advertisers and then specific products, using offsite activity such as views, purchases, and add-to-cart events. The advertiser model is already in production, while item-level modeling addresses Pinterest’s rapidly growing catalog and enables more precise, scalable personalization. ## Predicting Advertiser Interaction - A bidirectional Transformer encodes each user’s behavioral event sequence. - An MLP-based advertiser tower represents candidate advertisers. - Training uses: - In-batch negative samples - Sampled softmax loss - Positive events consisting of checkout, add-to-cart, or signup conversions within a future K-day window - Log-Q bias correction to avoid excessively penalizing popular advertisers - The model is evaluated with Recall@K by comparing user and advertiser embedding similarity against an indexed set of roughly 2 million advertisers. - An offline batch job generates each user’s top 100 advertisers and publishes them to the online feature store. - During ad serving, eligible ads from those advertisers are passed to the L1 ranker, blended with other candidate sources, and scored by heavier downstream models and the marketplace auction. - Online experiments produced higher conversion volume and lower cost per action. - The advertiser-level model has served production traffic for Standard ads since Spring 2024. ## Moving from Advertisers to Products - Pinterest next sought to predict the specific products a user would interact with, rather than only the likely advertiser. - Item-level prediction better matches the item-based ad delivery funnel and avoids forcing downstream models to score an impractically large set of products from selected advertisers. - The approach aims to capture both immediate intent and longer-term interests. ## Item-Level Model Architecture - The model retains the two-tower design: - A user tower encodes behavioral sequences. - An item tower represents individual shopping product Pins. - Item representations combine: - Internal Pin embeddings learned from Pinterest’s engagement graph - Product metadata from the merchant catalog - Because the catalog exceeds 1 billion items, training uses both in-batch negatives and a randomly sampled negative set of 20 million Pins. - The model uses the same conversion labels as the advertiser model. - Label weights and log-Q parameters are tuned to balance retrieval quality with diversity across both products and advertisers. - Daily inference updates user embeddings only for users with new activity, appending them to a previous feature-store snapshot to reduce computation. - The trained item tower indexes hundreds of millions of ad items. ## Evaluation and Diversity - Item retrieval is evaluated using cosine similarity and hit rates at different K values. - Final model selection considers both: - Item-level Recall@K - Advertiser-level Recall@K - Qualitative review is also important because offsite activity is sparse and noisy. - The model is compared with max-pooling and mean-pooling baselines that use aggregated embeddings without Transformer-based sequence modeling. - The evaluation emphasizes that strong retrieval must also produce semantically relevant and sufficiently diverse recommendations. Pinterest’s progression from advertiser prediction to item prediction shows how behavioral sequence models can make ad retrieval more personalized while remaining scalable. A practical system should combine sequence-aware user representations, large-scale approximate retrieval, and explicit controls for popularity, diversity, and computational efficiency.

Read original

Transformer Embeddings Feature Store Contrastive Learning+4

woowahanDec 11, 2025Original article

Enhancing the “Frequently Bought (opens in new tab)

Baedal Minjok (Baemin) has significantly improved its cart recommendation system by transitioning from a basic Item2Vec model to a sophisticated two-stage architecture that combines graph-based embeddings with Transformer sequence modeling. This evolution addresses the "substitutability bias" and lack of sequential context found in previous methods, allowing the system to understand the specific intent behind a user's shopping journey. By moving beyond simple item similarity, the new model effectively identifies cross-selling opportunities that align with the logical flow of a customer's purchase behavior. ### Limitations of the Item2Vec Approach * **Substitutability Bias:** The original Item2Vec model, based on the Skip-gram architecture, tended to map items from the same category into similar vector spaces. This resulted in recommending alternative brands of the same product (e.g., suggesting another brand of milk) rather than complementary goods (e.g., cereal or bread). * **Loss of Sequential Context:** Because Item2Vec treats a basket of goods as a "bag of words," it ignores the order in which items are added. This prevents the model from distinguishing between different user intents, such as a user starting with meat to grill versus a user starting with ingredients for a stew. * **Failure in Cross-Selling:** The primary goal of cart recommendations is to encourage cross-selling, but the reliance on embedding similarity alone limited the diversity of suggestions, often trapping users within a single product category. ### Stage 1: Graph-Based Product and Category Embeddings * **Node2Vec Implementation:** To combat data sparsity and the "long-tail" problem where many items have low purchase frequency, the team utilized Node2Vec. This method uses random walks to generate sequences that help the model learn structural relationships even when direct transaction data is thin. * **Heterogeneous Graph Construction:** The graph consists of both "Item Nodes" and "Category Nodes." Connecting items to their respective categories allows the system to generate initial vectors for new or low-volume products that lack sufficient historical purchase data. * **Association Rule Weighting:** Rather than using simple co-occurrence counts for edge weights, the team applied Association Rules. This ensures that weights reflect the actual strength of the complementary relationship, preventing popular "mega-hit" items from dominating all recommendation results. ### Stage 2: Transformer-Based Sequence Recommendation * **Capturing Purchase Context:** The second stage employs a Transformer model to analyze the sequence of items currently in the user's cart. This architecture is specifically designed to understand how the meaning of an item changes based on what preceded it. * **Next Item Prediction:** Using the pre-trained embeddings from Stage 1 as inputs, the Transformer predicts the most likely "next item" a user will add. This allows the system to provide dynamic recommendations that evolve as the user continues to shop. * **Integration of Category Data:** By feeding both item-level and category-level embeddings into the Transformer, the model maintains a high level of accuracy even when a user interacts with niche products, as the category context provides a fallback for the recommendation logic. ### Practical Conclusion For production-scale recommendation systems, relying solely on item similarity often leads to redundant suggestions that do not drive incremental sales. By decoupling the learning of structural relationships (via graphs) from the learning of temporal intent (via Transformers), engineers can build a system that is robust against data sparsity while remaining highly sensitive to the immediate context of a user's session. This two-stage approach is recommended for e-commerce environments where cross-category discovery is a key business metric.

Transformer AI Machine Learning Deep Learning+5

googleDec 3, 2025Original article

Titans + MIRAS: Helping AI have long-term memory (opens in new tab)

Google Research has introduced Titans, a new architecture, and MIRAS, a theoretical framework, designed to overcome the computational limitations of Transformers while maintaining high-fidelity long-term memory. These innovations utilize "test-time memorization," allowing models to update their core parameters in real-time as they process data without requiring offline retraining. By combining the speed of linear recurrent neural networks (RNNs) with the accuracy of attention mechanisms, the system enables AI to handle massive contexts such as genomic analysis or full-document understanding. ## Titans and Neural Long-Term Memory * Unlike traditional RNNs that compress context into fixed-size vectors or matrices, Titans uses a multi-layer perceptron (MLP) as a dedicated long-term memory module. * This deep neural memory provides significantly higher expressive power, allowing the model to synthesize and understand entire narratives rather than just storing passive snapshots. * The architecture separates memory into two distinct modules: an attention mechanism for precise short-term context and the MLP for summarizing long-term information. ## The Gradient-Based Surprise Metric * Titans employs a "surprise metric" to decide which information is important enough to store, mirroring the human brain's tendency to remember unexpected events. * The model calculates an internal error signal (gradient); a high gradient indicates that the new input is anomalous or context-breaking, signaling it should be prioritized for long-term storage. * The system incorporates "Momentum" to track the flow of context over time, ensuring that subsequent relevant information is captured even if individual tokens are not surprising. * To manage memory capacity during extremely long sequences, an adaptive weight decay mechanism acts as a forgetting gate to discard information that is no longer useful. ## MIRAS: A Unified Framework for Sequence Modeling * MIRAS provides a theoretical blueprint that views all major sequence models—including Transformers and linear RNNs—as different forms of associative memory modules. * The framework defines sequence models through four key design choices: memory architecture (e.g., MLP vs. vector), attentional bias, and the internal learning objectives used to combine new and old data. * This approach shifts AI modeling toward real-time adaptation, where the model actively learns and incorporates specific new details into its core knowledge as data streams in. These advancements suggest a shift away from static context windows toward dynamic systems capable of lifelong learning. For developers working with large-scale data, the Titans architecture provides a practical tool for scaling performance, while the MIRAS framework offers a roadmap for designing next-generation models that adapt instantly to new information.

Transformer AI Sequence Modeling Recurrent Neural Networks+5

googleNov 18, 2025Original article

Real-time speech-to-speech translation (opens in new tab)

Google DeepMind and Google Core ML have developed an innovative end-to-end speech-to-speech translation (S2ST) model that enables real-time, voice-preserved communication with only a two-second delay. By replacing traditional cascaded pipelines with a streaming architecture trained on time-synchronized data, the system overcomes long-standing issues of high latency and accumulated errors. This advancement represents a significant shift toward natural, fluid cross-language dialogue that retains the original speaker's personality. ## Limitations of Cascaded S2ST Traditional real-time translation systems typically rely on a cascaded chain of three distinct AI models: Automatic Speech Recognition (ASR), Automatic Speech Translation (AST), and Text-to-Speech (TTS). This approach suffers from several critical drawbacks: * **High Latency:** Processing through three separate stages results in a 4–5 second delay, forcing users into unnatural, turn-based interactions. * **Error Propagation:** Inaccuracies in the initial transcription or translation phase accumulate, often leading to garbled or incorrect final audio output. * **Loss of Identity:** General-purpose TTS engines generate generic voices, stripping the communication of the original speaker’s unique vocal characteristics. ## Time-Synced Data Acquisition Pipeline To train an end-to-end model capable of low-latency output, researchers created a scalable pipeline that transforms raw audio into a specialized time-synchronized dataset. * **Alignment Multi-mapping:** The process uses forced alignment algorithms to map source audio to source text, source text to translated text, and finally, translated text to generated speech. * **Voice Preservation:** A custom TTS engine generates the target language audio while intentionally preserving the vocal characteristics of the original speaker. * **Strict Validation:** Automated filters discard any segments where alignments fail or where the translated audio cannot meet specific real-time delay requirements. * **Data Augmentation:** The training set is further refined using techniques such as sample rate reduction, denoising, and reverberation to ensure the model performs well in real-world environments. ## End-to-End Streaming Architecture The model’s architecture is designed for continuous audio streams, leveraging the AudioLM framework and fundamental transformer blocks to make real-time decisions. * **Streaming Encoder:** This component summarizes source audio data by focusing on the preceding 10-second window of input. * **Streaming Decoder:** This module predicts translated audio autoregressively, utilizing compressed encoder states and previous predictions to maintain flow. * **RVQ Audio Tokens:** The system represents audio as a 2D set of Residual Vector Quantization (RVQ) tokens, where the X-axis represents time and the Y-axis represents audio quality/fidelity. * **SpectroStream Integration:** By using SpectroStream codec technology, the model manages hierarchical audio representations, allowing it to prioritize the sequential output of audio segments for immediate playback. This technology effectively bridges the gap between high-quality translation and real-time responsiveness. For developers and researchers in the field, the transition from modular cascaded systems to end-to-end streaming architectures—supported by rigorous time-aligned datasets—is the recommended path for achieving truly seamless human-to-human cross-language communication.

Transformer AI Machine Learning Text To Speech+5

googleNov 6, 2025Original article

Introducing Nested Learning: A new ML paradigm for continual learning (opens in new tab)

Google Research has introduced Nested Learning, a paradigm that treats machine learning models as systems of interconnected, multi-level optimization problems rather than separate architectures and training rules. By unifying structure and optimization through varying update frequencies, this approach aims to mitigate "catastrophic forgetting," the tendency for models to lose old knowledge when acquiring new skills. The researchers validated this framework through "Hope," a self-modifying architecture that outperforms current state-of-the-art models in long-context memory and language modeling. ### The Nested Learning Paradigm This framework shifts the view of machine learning from a single continuous process to a set of coherent, nested optimization problems. Each component within a model is characterized by its own "context flow"—the specific set of information it learns from—and its own update frequency. * The paradigm argues that architecture (structure) and optimization (training rules) are fundamentally the same concept, differing only by their level of computational depth and update rates. * Associative memory is used as a core illustrative concept, where the training process (backpropagation) is modeled as a system mapping data points to local error values. * By defining an update frequency rate for each component, researchers can order these problems into "levels," allowing for a more unified and efficient learning system inspired by the human brain's neuroplasticity. ### Deep Optimizers and Refined Objectives Nested Learning provides a principled way to improve standard optimization algorithms by viewing them through the lens of associative memory modules. * Existing momentum-based optimizers often rely on simple dot-product similarity, which fails to account for how different data samples relate to one another. * By replacing these simple similarities with standard loss metrics, such as L2 regression loss, the researchers derived new formulations for momentum that are more resilient to imperfect or noisy data. * This approach turns the optimizer itself into a deeper learning component with its own internal optimization objective. ### Continuum Memory Systems and the "Hope" Architecture The paradigm addresses the limitations of Large Language Models (LLMs), which are often restricted to either their immediate input window or static pre-trained knowledge. * The researchers developed "Hope," a proof-of-concept architecture that utilizes multi-time-scale updates for its internal components. * While standard Transformers act primarily as short-term memory, the Nested Learning approach allows for "continuum memory" that manages long-context information more effectively. * Experimental results show that this self-modifying architecture achieves superior performance in language modeling compared to existing state-of-the-art models. By recognizing that every part of a model is essentially an optimizer operating at a different frequency, Nested Learning offers a path toward AI that can adapt to new experiences in real-time. This structural shift moves away from the "static pre-training" bottleneck and toward systems capable of true human-like neuroplasticity and lifelong learning.

Transformer Machine Learning Deep Learning Optimization Algorithms+4