lineJul 16, 2026

Developing a Model to Assess Harmfulness from Open Chat Names and Descriptions (opens in new tab)

The AI Services Lab developed a model to automatically detect harmful LINE OpenChat names and descriptions, reducing the need for manual review. The project improved an existing moderation system by cleaning inconsistent labels, selecting a lightweight safety-tuned decoder model, and adapting it to predict both penalty levels and reasons. Granite Guardian 3.1 2B was ultimately fine-tuned with LoRA and deployed using token-probability-based inference. ## OpenChat Monitoring - Users must provide an OpenChat name and may add a description. - Names and descriptions are reviewed whenever they are created or modified. - LINE processes a large volume of global OpenChats, making fully manual moderation impractical. - The project aimed to: - Expand automated moderation to countries requiring more detailed judgments. - Improve accuracy in regions already using automation. - Reduce the amount of content requiring human review. ## Data Cleansing - Training data consisted of previously manually reviewed OpenChat names and descriptions. - Only records reviewed under the current moderation guidelines were used. - Identical name-description pairs sometimes had conflicting penalty outcomes. - Labels were consolidated using these rules: - Select the most severe penalty if it appeared at least twice. - If it appeared only once, treat it as possible noise and select the second-most-severe penalty. - Choose the most frequent penalty reason. - If reasons were tied, choose the globally rarer reason, following a TF-IDF-like principle that rarer reasons may be more specific. - This process produced a single, consistent label for each identical input. ## Selecting the Pretrained Model The team evaluated models according to four requirements: - Decoder-based architecture. - Fine-tuned for safety moderation. - Approximately 2 billion parameters. - Apache license for commercial use. Granite Guardian 3.1 2B was selected because: - It is designed to classify harmfulness through the probabilities of “Yes” and “No” tokens. - Restricting predictions to predefined tokens avoids unpredictable free-form responses. - Token probabilities provide confidence scores that can be thresholded for operational needs. - Its relatively small size supports lower serving costs and faster responses. ## Fine-Tuning for Penalty Prediction - A simple harmful/not-harmful classification was insufficient because moderation decisions include different penalty levels and reasons. - The model was trained to produce structured responses containing: - An `Action` penalty code. - A `Reason` penalty reason. - Cross-entropy loss was calculated only over the assistant’s response tokens, not the entire prompt. - This focuses training on predicting moderation decisions rather than reproducing the input text. - LoRA was used instead of full-parameter fine-tuning: - The base model parameters remained frozen. - Only small trainable matrices representing parameter updates were optimized. - This reduced memory and training costs while preserving pretrained capabilities. ## Inference Design - During inference, the model calculates logits for all possible next tokens. - The system extracts only the logits corresponding to valid penalty-code tokens, converts them to probabilities, and selects the highest-scoring code. - It then predicts the penalty reason in a second step. - Existing operational codes consisted of arbitrary letters and numbers that tokenized into multiple pieces. - To simplify probability calculations, penalty codes and reasons were mapped to meaningful natural-language tokens, each represented by a single tokenizer token. - KV caching was used between the penalty-code and penalty-reason predictions to improve efficiency. The resulting approach combines cleaned moderation labels, lightweight decoder-model fine-tuning, structured output targets, and constrained token-level inference. It is intended to broaden automated OpenChat moderation while maintaining the accuracy and response speed required for real-time LINE operations.

pytorchtransformerloratokenization+4

metaJun 30, 2026

10 Years of Meta’s Commitment to Python (opens in new tab)

Meta marks its 10th consecutive year sponsoring the Python Software Foundation (PSF), emphasizing that Python is central to its infrastructure, products, and AI work. The company views sponsorship as both a responsibility to the open-source community and a strategic investment in the long-term health, security, and innovation of the technology it relies on. ## Python’s Role at Meta - Python is Meta’s most widely used programming language. - It supports infrastructure for products including Instagram and Threads, as well as AI research and data-driven initiatives. - Meta engineers contribute directly to Python’s development, including core maintenance and Python Enhancement Proposals. - Meta’s open-source contributions include: - PyTorch, originally developed at Meta before becoming an independent foundation. - Pyrefly, a fast Python type checker and language server. - Meta expects Python to remain important as it expands AI capabilities and scales its infrastructure. ## Why Meta Supports the PSF - Open-source adoption creates a shared responsibility to maintain a healthy, secure, and sustainable ecosystem. - PSF funding supports the Developer-in-Residence program, enabling full-time developers to work on Python improvements that might otherwise be neglected or left to volunteers. - Sponsorship helps strengthen PyPI, including critical security improvements that protect package distribution and consumption. - Funding also supports education and community development through: - PyCon US workshops, summits, and discounted or free passes. - Fundraising and support for groups such as PyLadies. - Meta considers these efforts an investment in the tools, infrastructure, and people behind its own technology stack. ## Ways to Support the Python Software Foundation - Individuals can make one-time donations or become PSF members. - Membership may include voting rights and can be supported through financial contributions or volunteer time. - Organizations can become annual sponsors at different contribution levels. - Sponsorship offers public recognition, community engagement opportunities, event participation, and—in higher tiers—greater visibility and invitations to special initiatives. Meta concludes by thanking Python’s maintainers, contributors, educators, and advocates, while encouraging other individuals and organizations to help sustain the language through PSF donations, membership, or sponsorship.

pytorchaimachine-learningopen-source+3

googleJun 3, 2026

The next chapter in flood resilience: Open sourcing Google’s hydrology framework (opens in new tab)

Google Research has open-sourced the hydrology framework behind its Flood Hub river forecasts. The Python/PyTorch package lets researchers and national forecasting agencies train AI models with global and local data while retaining control over their information. Google argues that open access, local expertise, and interoperable tools can make advanced flood warnings more accurate, affordable, and widely deployable. ## The Open-Source Hydrology Framework - The framework is available on GitHub under an Apache 2.0 license. - It provides model architectures, training pipelines, documentation, and tutorials. - Users can train models with climate, soil, topography, land-cover, and weather data. - Historical river observations come from the open Caravan dataset, which agencies can extend with local measurements. - The package is built with PyTorch and is intended for both researchers and operational forecasters. ## Model Versions and Improvements - The release includes: - The original model used in Google’s 2024 benchmarking study. - An upgraded v2 model currently used for real-time global forecasts in Flood Hub. - The v2 model uses a multi-input ME-LSTM architecture. - Separate networks embed different meteorological products before combining them in an LSTM. - Inputs include GraphCast, ECMWF forecasts, NASA IMERG satellite rainfall estimates, and NOAA CPC precipitation data. - Benchmarking showed the newer model extends the reliable forecast horizon by: - Six days in gauged river basins. - One day in ungauged basins. ## Local Data and Operational Forecasting - Agencies can fine-tune models for specific watersheds using local observations and expert knowledge. - The approach supports the integration of Indigenous and Local Knowledge, which the World Meteorological Organization says is still rarely incorporated systematically. - Models are designed to be relatively inexpensive and easier to train than traditional conceptual hydrological systems. - Local organizations can preserve control over their data while adapting the models to regional conditions. ## Partnership with the Czech Hydrometeorological Institute - Google worked with CHMI to validate the model against locally calibrated traditional forecasting models. - CHMI created an adapter connecting the framework to Delft-FEWS, a widely used operational forecasting platform. - This integration demonstrates how machine-learning forecasts can fit into existing workflows used by government agencies, NGOs, and private organizations. - The partnership provides a practical model for other national hydrological services. ## Broader Flood-Resilience Goals - Open-source distribution could help resource-constrained regions access advanced forecasting without expensive infrastructure. - The framework is intended to support capacity building for early-warning systems worldwide. - Google presents the release as a way to let the global hydrology community reproduce, improve, and localize its research. National hydrological agencies and researchers should evaluate the open-source framework using their own watershed data, integrate it with existing forecasting systems, and validate its predictions against established local models before operational deployment.

pytorchmachine-learningpythonflood-forecasting+3

metaMay 26, 2026

SilverTorch: Index as Model — A New Retrieval Paradigm for Recommendation Systems (opens in new tab)

SilverTorch is a unified, GPU-based recommendation retrieval system designed to replace fragmented microservices with one integrated neural network. Its “Index as Model” architecture represents retrieval components—including item indices, filtering, reranking, and user modeling—as PyTorch modules. The system reportedly delivers up to 23.7× higher throughput and 20.9× better compute-cost efficiency than comparable CPU-based or traditional multi-service systems, while improving recommendation quality. ## Limits of Microservice-Based Retrieval - Traditional retrieval pipelines use separate services for: - Computing user embeddings - Finding similar content - Applying eligibility rules - Scoring and reranking candidates - An orchestrator coordinates these services before passing thousands of candidates to downstream ranking, all within roughly 100 milliseconds. - This architecture creates several structural problems: - **Data movement:** Network calls, serialization, and service coordination consume latency that could otherwise support more computation. - **Version inconsistency:** User models, item indices, and filtering rules may be updated independently, causing mismatches between user and item representations. - **Siloed engineering:** ML teams typically work in PyTorch while infrastructure teams work in C++, making improvements difficult to translate, test, and deploy. - GPU optimizations such as Faiss-GPU can accelerate individual services but do not eliminate the architectural overhead or enable deep coordination between components. ## Index as Model - SilverTorch replaces the service mesh with a single neural network. - Its central design principle, **Index as Model**, turns traditional retrieval artifacts into model components: - Item indices become tensors. - Eligibility filters become operators. - User towers, scoring layers, and rerankers become modules. - A single request passes through the integrated model, which: - Finds content relevant to the user’s interests - Applies language, geography, and policy constraints - Predicts multiple engagement outcomes - Produces a combined score for the final candidate set - This integration enables more complex models and larger candidate evaluations without exceeding the sub-100-millisecond latency target. ## Unified Retrieval Components - SilverTorch incorporates multiple functional regions within one model: - Approximate nearest-neighbor search identifies relevant items efficiently. - Eligibility filtering removes content that cannot be shown to a user. - Multi-task reranking predicts actions such as likes, shares, and comments. - Composite scoring combines these predictions into a final ranking signal. - Some components are hand-engineered, while others can be trained end-to-end through backpropagation. - From the runtime’s perspective, every component is a standard PyTorch `nn.Module`, regardless of whether it performs search, filtering, or learned prediction. ## Pure PyTorch Implementation - SilverTorch reimplements ANN search, Bloom-filter indexing, eligibility checks, neural reranking, and composite scoring as pure PyTorch modules. - The unified design requires: - Tensor-based data representation - Tensor-in, tensor-out operations - A consistent `nn.Module` interface - This allows modules to share memory, execution graphs, and compilation steps. - Engineers can co-design stages—for example, selecting promising clusters, filtering within them, and scoring only surviving candidates—instead of treating each operation as an isolated service. - The approach reduces the separation between ML and infrastructure engineering, allowing both groups to work within the same programmable layer. ## Performance and Scale - In an 80-million-item end-to-end evaluation, SilverTorch achieved: - **23.7× higher requests per second** than a strong traditional multi-service baseline using the same model architecture. - **20.9× better estimated total-cost-of-ownership efficiency** than a CPU-based solution. - The system is intended to support retrieval across multiple applications and large-scale feeds and video products. - Its increased efficiency makes neural reranking and multi-task engagement scoring practical within strict production latency budgets. SilverTorch’s main recommendation is architectural: consolidate retrieval into a single, composable model rather than optimizing disconnected services. Representing every retrieval stage as a PyTorch module can reduce overhead, improve consistency, enable deeper cross-stage optimization, and make more sophisticated recommendations feasible at scale.

pytorchmicroservicesrecommendation-systemsneural-networks+3

lineMar 6, 2026

On-Device Image Model (opens in new tab)

The post describes building an on-device image understanding system for messaging apps, with semantic image search as the first focus. Its central strategy was knowledge distillation: a multilingual student text encoder learned to reproduce the embedding space of a strong, English-only teacher model. This preserved most English retrieval quality while enabling Japanese, Traditional Chinese, Thai, and Korean search, achieving an average Recall@5 above 78%. ## Why Messaging Apps Need On-Device Image Understanding - Images are often treated simply as “a photo,” unlike text messages, which can support search, summaries, and notification previews. - Image understanding could improve: - Notifications: “Sent one photo” → “Sent a photo of a dog” - Search: queries such as “dog,” “puppy,” or “a cat inside a box” - Recommendations: automatic image classification and organization - Shared image-text embeddings allow semantically equivalent phrases—such as “dog,” “puppy,” and “개”—to retrieve the same images. ## Why the Model Had to Run On-Device - **Latency:** Network round trips make notifications and search less predictable and responsive. - **Privacy:** Sending photos, captions, or embeddings to a server increases privacy risks. - **Offline support:** The feature should work in subways, airplanes, roaming environments, and unstable networks. - **Mobile constraints:** The model needed to run on both Android and iOS with limited memory and compute resources. - The project targeted a model under **200 MB**, response times within a few hundred milliseconds including cold start, and LiteRT compatibility. ## Project Goals and Evaluation - The image search system needed to: - Retrieve images by semantic meaning rather than keyword matching. - Support English, Japanese, Traditional Chinese, Thai, and Korean. - A separate captioning system was designed to generate short, natural descriptions of roughly eight words or fewer. - Search quality was measured using: - Image-to-Text Recall@5 - Text-to-Image Recall@5 - Caption quality was evaluated with CIDEr, CLIPScore, and an LLM-based acceptance ratio designed to detect repetition, typos, and grammatical problems. ## Why Translation Was Not Enough The initial approach translated each query into English before using an English-only image-text model: ```text Query → Language detection → Translation → English text encoder → Embedding → Search ``` This approach introduced several problems: - **Quality loss:** Informal terms or short queries could be mistranslated. For example, “멍멍이” might be interpreted as “barking” instead of “dog.” - **Additional latency:** Translation adds a fixed cost before text encoding. - **Inconsistent results:** Translation quality varies by language pair and wording. - **Operational complexity:** Each additional language requires more models, updates, and failure handling. Training a multilingual image-text model from scratch would require substantial data and compute. Instead, the project retained the proven English image embedding space and expanded only the text encoder. ## Knowledge Distillation for Multilingual Search - The original English text encoder served as the frozen **teacher**. - A copied text encoder served as the trainable **student**. - English text was passed to the teacher, while corresponding multilingual text was passed to the student. - The student was trained to match the teacher’s embeddings using mean squared error (MSE). ```text teacher_embedding = teacher(English text) student_embedding = student(Multilingual text) loss = MSE(teacher_embedding, student_embedding) ``` The image encoder remained frozen so that the established image-text embedding space would not be disrupted. Important implementation considerations included: - Ensuring the tokenizer handled multilingual characters correctly. - Defining consistent case-insensitivity rules. - Balancing training samples across languages. - Matching training-time preprocessing and tokenization with mobile inference behavior. ## Retrieval Results - English performance declined slightly: - Image-to-Text Recall@5: **79.58% → 76.56%** - Text-to-Image Recall@5: **75.89% → 74.47%** - Multilingual performance improved from below **10% average Recall@5** to above **78%**, roughly a sevenfold improvement. - Japanese achieved **81.94%**, exceeding the original English model in the reported evaluation. - Traditional Chinese, Thai, and Korean also reached practically usable retrieval quality. The trade-off—slightly lower English performance in exchange for four additional languages—provided substantially greater overall product value. ## Converting the Model to LiteRT - LiteRT was selected because it officially supports both Android and iOS and provides mobile-oriented operators, quantization, and optimization tools. - Core ML was rejected because it is iOS-specific and introduced conversion and long-term cross-platform maintenance concerns. - Conversion required addressing unsupported PyTorch operators. - For example, LiteRT did not support `erf`, so the model’s implementation had to replace it with a compatible pseudo-`erf` operation. The resulting approach demonstrates that knowledge distillation can efficiently extend an existing English image-text model to multiple languages while preserving its on-device deployment advantages.

pytorchknowledge-distillationlitertembedding-model+4

netflixFeb 13, 2026

Scaling LLM Post-Training at Netflix (opens in new tab)

Netflix argues that LLM post-training at production scale is as much an infrastructure challenge as a modeling challenge. Its internal framework abstracts distributed data processing, model sharding, GPU orchestration, checkpointing, and complex training workflows so developers can focus on experimentation. The result is a flexible system supporting SFT, DPO, reinforcement learning, and knowledge distillation across hundreds of GPUs. ## Why Post-Training Becomes an Engineering Problem - Pre-training provides general language ability, but post-training adapts models to Netflix’s catalog, member histories, recommendation tasks, personalization, and search. - Production-scale training introduces challenges involving: - Large proprietary datasets - Multi-node GPU coordination - Distributed model state - Workflows that combine training and inference - Failure recovery and experiment tracking - A simple Hugging Face fine-tuning script is insufficient for reliable, large-scale jobs. ## Preparing Data Correctly - Chat templates serialize conversations but do not determine which tokens should contribute to the loss. - Netflix applies explicit loss masking so training focuses on assistant responses rather than prompts or other non-target text. - Variable-length examples can waste GPU memory through padding and create synchronization overhead across FSDP workers. - Sequence packing combines multiple samples into fixed-length sequences. - A document mask prevents attention across separately packed samples while improving GPU utilization. ## Loading and Optimizing Large Models - Models that do not fit on one GPU require sharding strategies such as FSDP or tensor parallelism. - Partial weights should be loaded directly onto the device mesh rather than materializing the entire checkpoint on a single device. - Developers can choose full fine-tuning or LoRA and use: - Activation checkpointing - Compilation - Appropriate precision settings - Reinforcement learning requires compatible precision between rollout generation and policy training. - Large vocabularies create memory pressure because logits have dimensions `[batch, seq_len, vocab]`. - The framework reduces peak memory by removing ignored tokens before projection and computing logits and loss in sequence chunks. ## Distributed Training and Workflow Management - The framework supports standard forward/backward training for SFT as well as workflows that interleave: - Rollout generation - Reward-model and reference-model inference - Policy updates - Ray actors orchestrate distributed jobs while keeping hardware concerns separate from modeling code. - Experiment tracking covers both quality metrics, such as loss, and efficiency metrics, such as Model FLOPS Utilization (MFU). - Standardized checkpointing allows jobs to resume after failures. ## Netflix’s Post-Training Framework - The stack is built on: - Mako for AWS GPU provisioning - PyTorch, Ray, and vLLM - Netflix’s framework library for reusable utilities and training recipes - Jobs are generally defined through configuration files that select a recipe and provide task-specific components. - Unlike narrower fine-tuning systems, the framework supports: - Custom output heads - Expanded vocabularies and semantic IDs - Special tokens - Transformer models trained on non-natural-language sequences - This flexibility is important for Netflix-specific recommendation and personalization use cases. ## Four Core Abstractions ### Data - Dataset abstractions cover SFT, reward modeling, and RL. - Streaming supports datasets larger than local disk capacity. - Asynchronous sequence packing overlaps CPU preprocessing with GPU execution to reduce idle time. ### Model - The framework supports architectures such as Qwen3 and Gemma3, including Mixture-of-Experts variants. - LoRA is integrated into model definitions. - High-level sharding APIs distribute models across device meshes without requiring developers to write low-level distributed code. ### Compute - A unified job interface scales from one node to hundreds of GPUs. - MFU measurement remains accurate for custom architectures and LoRA configurations. - Checkpoints include parameters, optimizer state, dataloader state, and data-mixer state, enabling exact resumption. ### Workflow - The system supports SFT, DPO, RL, and knowledge distillation. - Online RL uses a hybrid architecture combining a single controller with Single Program, Multiple Data (SPMD) workers. - This extends conventional SPMD training to multi-stage workflows that cannot be represented as a simple training loop. Netflix’s approach is to standardize the difficult operational parts of post-training while preserving enough flexibility for unconventional models and objectives. A framework built around reusable data, model, compute, and workflow abstractions can help teams iterate faster and scale experiments without repeatedly rebuilding distributed infrastructure.

pytorchdatabase-designllmcloud-computing+4

pinterestFeb 2, 2026

Beyond Two Towers: Re-architecting the Serving Stack for Next-Gen Ads Lightweight Ranking Models… (opens in new tab)

Two-Tower models make retrieval and lightweight ranking highly efficient by scoring user and item embeddings with a dot product, but they cannot represent rich user-item interactions or deep feature crossings. This post describes an ads-serving redesign that introduces general-purpose GPU models while preserving end-to-end latency. The main strategy is to reduce data movement, move filtering logic onto the GPU, and optimize inference from an initial 4-second p90 latency to about 20 milliseconds. ## Why Move Beyond Two-Tower Models - Two-Tower architectures independently encode users and items, enabling fast scoring across millions of candidates. - Their decoupled structure limits: - User-item interaction features - Target attention - Early feature crossing - Deep architectures requiring simultaneous access to user and candidate data - More expressive models require GPU-based general-purpose inference rather than specialized dot-product or ANN retrieval. - The existing retrieval stack was not designed to transfer large candidate and feature sets to a GPU, creating a major latency challenge. ## Restructuring the Serving Funnel The traditional funnel consisted of: - Feature expansion for thousands of candidates - Retrieval and Two-Tower lightweight ranking - Heavy ranking and auction processing for the top documents Adding GPU inference directly to this flow would require fetching, serializing, transferring, and returning features for tens of thousands of documents. The authors therefore redesigned the entire early-stage serving pipeline instead of optimizing the model alone. ## Segmenting the Inventory for Feature Fetching Feature retrieval was a major latency source, often taking longer than model inference for workloads ranging from 10,000 to 100,000 documents. - **High-value inventory:** Roughly 1 million documents responsible for a substantial share of revenue have their features embedded in the PyTorch model as registered buffers. - Features become part of the model state, similar to weights. - They remain in GPU high-bandwidth memory. - Requests avoid remote feature-service calls and host-to-device transfers. - The model file must be periodically updated to refresh features. - Future work may include GPU-based caching. - **Long-tail inventory:** The remaining roughly 1 billion documents use a high-performance key-value store with in-host caching. - The post focuses on the first strategy, which is already running in production. ## Moving Business Logic onto the GPU Previously, the model returned scores for approximately 100,000 candidates, while CPU-side code handled utility calculation, filtering, diversity, deduplication, and top-k selection. - The new PyTorch model performs these operations directly: - Combines pCTR, pCVR, bid, and other signals into utility scores. - Applies diversity and filtering rules. - Performs top-k selection. - The GPU returns only the final winners—typically around 1,000 documents—instead of all candidate scores. - This reduces device-to-host data transfer and takes advantage of GPU parallelism. - The approach works because lightweight-ranking business rules are sufficiently simple to express with tensor operations. ## Reducing GPU Inference Latency Initial GPU inference measured roughly 4,000 ms at p90, far too slow for real-time serving. Several systems optimizations reduced this to approximately 20 ms: - **Multiple CUDA streams:** Separate streams for workers allow host-to-device transfers, computation, and device-to-host transfers to overlap. - **Worker alignment:** Worker threads are matched and pinned to physical CPU cores to reduce context switching and lock contention. - **Kernel fusion:** Triton kernels combine operations such as linear layers and activations, reducing memory traffic. - **BF16 computation:** Brain Floating Point 16 lowers memory usage and accelerates arithmetic compared with FP32. - **Profiling tools:** PyTorch Profiler and NVIDIA Nsight Systems were used to identify bottlenecks. ## Practical Recommendation Deploying more expressive ranking models requires rethinking the serving architecture around data movement and execution placement. Embedding frequently used features, executing business logic on the GPU, and applying low-level CUDA and kernel optimizations can make complex neural ranking feasible without increasing end-to-end latency.

pytorchmachine-learningrecommendation-systemsgpu-inference+4