slackMay 28, 2026

Slack AI: The Path to Multi-Cloud (opens in new tab)

Slack’s AI infrastructure evolved from self-managed SageMaker deployments to managed Amazon Bedrock as Slack sought enterprise-grade security, reliability, lower operational overhead, and faster access to new models. SageMaker provided strong isolation and compliance but required extensive capacity and regional management, while Bedrock simplified operations through managed throughput and on-demand options. Slack’s carefully staged migration achieved zero customer-facing incidents and established a practice of measuring parity, shifting traffic gradually, and monitoring continuously. ## Phase 1: The SageMaker Era - SageMaker offered: - Security and FedRAMP compliance - Control over model availability - An escrow VPC strategy that kept Slack data private while preventing access to providers’ model weights - Slack deployed model containers across multiple AWS regions to support global availability. - Engineering teams had to manage: - Cross-region IAM roles - Routing across model endpoints - Capacity planning - Auto-scaling - The main operational challenges were: - **Scaling latency:** New instances could not start instantly. - **GPU scarcity:** A100 and H100 capacity was often unavailable. - **Over-provisioning:** Idle resources had to be maintained for peak demand. - On-Demand Capacity Reservations and scheduled scaling reduced some problems, but required substantial manual coordination. - SageMaker also created model feature lag because AWS prioritized releasing newer Anthropic models and optimizations through Bedrock. ## Phase 2: Migrating to Amazon Bedrock By mid-2024, Bedrock had matured enough to meet Slack’s security and FedRAMP requirements. - The migration provided: - Managed infrastructure instead of individual GPU instances - Faster access to newly released LLMs - Flexible capacity options for different workload patterns - Slack used: - **Provisioned Throughput (PT)** for predictable, latency-sensitive features such as channel summaries - **On Demand (OD)** for bursty scheduled workloads such as Recap - Bedrock measured capacity in **Model Units (MUs)**, letting Slack plan around token throughput rather than hardware details. ## Executing a Zero-Incident Migration Slack used a gradual, validation-heavy process: - Obtained Legal, Security, and FedRAMP approval before moving production traffic. - Performed load testing to determine the number of MUs needed to match SageMaker capacity. - Compared model quality and latency through A/B tests and evaluation frameworks. - Used feature flags for incremental traffic shifting and immediate rollback. - Ran shadow requests and extensive tests to verify behavioral and performance parity. This approach allowed Slack to move live production traffic without customer-facing incidents. ## Operational Improvements and Remaining Gaps - Bedrock reduced infrastructure maintenance, allowing engineers to focus more on model quality and product features. - Slack could adopt new models weeks or months earlier, improving features such as AI Search with higher-reasoning models. - Capacity planning shifted from reactive scaling to forecasting demand several weeks ahead. - Provisioned Throughput still required maintaining a high baseline of MUs to handle large regional workday surges, creating an over-provisioning challenge for traffic that varies significantly throughout the day. Slack’s migration demonstrates that managed AI infrastructure can improve agility and reliability, but success depends on careful capacity modeling, comprehensive parity testing, gradual rollout, and continuous monitoring.

gpullmmodel-servingaws-bedrock+3

metaMay 26, 2026

SilverTorch: Index as Model — A New Retrieval Paradigm for Recommendation Systems (opens in new tab)

SilverTorch is a unified, GPU-based recommendation retrieval system designed to replace fragmented microservices with one integrated neural network. Its “Index as Model” architecture represents retrieval components—including item indices, filtering, reranking, and user modeling—as PyTorch modules. The system reportedly delivers up to 23.7× higher throughput and 20.9× better compute-cost efficiency than comparable CPU-based or traditional multi-service systems, while improving recommendation quality. ## Limits of Microservice-Based Retrieval - Traditional retrieval pipelines use separate services for: - Computing user embeddings - Finding similar content - Applying eligibility rules - Scoring and reranking candidates - An orchestrator coordinates these services before passing thousands of candidates to downstream ranking, all within roughly 100 milliseconds. - This architecture creates several structural problems: - **Data movement:** Network calls, serialization, and service coordination consume latency that could otherwise support more computation. - **Version inconsistency:** User models, item indices, and filtering rules may be updated independently, causing mismatches between user and item representations. - **Siloed engineering:** ML teams typically work in PyTorch while infrastructure teams work in C++, making improvements difficult to translate, test, and deploy. - GPU optimizations such as Faiss-GPU can accelerate individual services but do not eliminate the architectural overhead or enable deep coordination between components. ## Index as Model - SilverTorch replaces the service mesh with a single neural network. - Its central design principle, **Index as Model**, turns traditional retrieval artifacts into model components: - Item indices become tensors. - Eligibility filters become operators. - User towers, scoring layers, and rerankers become modules. - A single request passes through the integrated model, which: - Finds content relevant to the user’s interests - Applies language, geography, and policy constraints - Predicts multiple engagement outcomes - Produces a combined score for the final candidate set - This integration enables more complex models and larger candidate evaluations without exceeding the sub-100-millisecond latency target. ## Unified Retrieval Components - SilverTorch incorporates multiple functional regions within one model: - Approximate nearest-neighbor search identifies relevant items efficiently. - Eligibility filtering removes content that cannot be shown to a user. - Multi-task reranking predicts actions such as likes, shares, and comments. - Composite scoring combines these predictions into a final ranking signal. - Some components are hand-engineered, while others can be trained end-to-end through backpropagation. - From the runtime’s perspective, every component is a standard PyTorch `nn.Module`, regardless of whether it performs search, filtering, or learned prediction. ## Pure PyTorch Implementation - SilverTorch reimplements ANN search, Bloom-filter indexing, eligibility checks, neural reranking, and composite scoring as pure PyTorch modules. - The unified design requires: - Tensor-based data representation - Tensor-in, tensor-out operations - A consistent `nn.Module` interface - This allows modules to share memory, execution graphs, and compilation steps. - Engineers can co-design stages—for example, selecting promising clusters, filtering within them, and scoring only surviving candidates—instead of treating each operation as an isolated service. - The approach reduces the separation between ML and infrastructure engineering, allowing both groups to work within the same programmable layer. ## Performance and Scale - In an 80-million-item end-to-end evaluation, SilverTorch achieved: - **23.7× higher requests per second** than a strong traditional multi-service baseline using the same model architecture. - **20.9× better estimated total-cost-of-ownership efficiency** than a CPU-based solution. - The system is intended to support retrieval across multiple applications and large-scale feeds and video products. - Its increased efficiency makes neural reranking and multi-task engagement scoring practical within strict production latency budgets. SilverTorch’s main recommendation is architectural: consolidate retrieval into a single, composable model rather than optimizing disconnected services. Representing every retrieval stage as a PyTorch module can reduce overhead, improve consistency, enable deeper cross-stage optimization, and make more sophisticated recommendations feasible at scale.

gpumicroservicesrecommendation-systemsneural-networks+3

kakaoApr 21, 2026

Kanana Scala 1st Seminar On-site Sketch (opens in new tab)

Kakao’s first Kanana Scholar seminar brought together seven leading AI professors and Kakao researchers to discuss the company’s independent AI strategy. Kakao presented its from-scratch Kanana foundation models, emphasizing data efficiency, Korean-language capability, and multimodal processing. The discussion concluded that Kakao should focus less on generic benchmark scores and more on technology sovereignty, personalized agents, and practical execution in real services. ## Kanana Foundation Models - Kakao is developing its own foundation-model lineup to strengthen competitiveness and reduce dependence on overseas providers. - Kanana reportedly achieved strong performance using 11 trillion training tokens, compared with 23 trillion tokens for a similarly sized global-target model. - Kakao attributed this efficiency to the quality and refinement of its training data. - The company also demonstrated **Kanana-o**, an omni model capable of processing text, images, and audio in real time. - The model handled emotional speech and multi-speaker conversations naturally, receiving praise for its Korean fluency. ## Technology Sovereignty and Customization - Kakao argued that proprietary models protect it from external risks such as changing licensing policies and closed technologies. - Owning the technology enables Kakao to build efficient, customized models optimized for its services. - Participating professors agreed that control over Korean cultural context and local issues is essential for technological sovereignty. - They viewed an independent model as a strategic asset for long-term service stability. ## Digital World Models and Personalized Agents - Kakao aims to understand users’ behavioral context within KakaoTalk and provide highly personalized assistance. - On-device AI could protect private conversations while allowing agents to respond immediately to user needs. - The professors suggested expanding the idea of “physical AI” into a **digital world model** that predicts interactions and causal relationships across a platform. - This direction could create an area of AI differentiation uniquely suited to Kakao’s ecosystem. ## Evaluating Practical Agentic Intelligence - Kakao is prioritizing AI systems that can create multi-step plans, call necessary tools, and complete tasks independently. - It plans to use an internally developed orchestration benchmark to evaluate real-world problem-solving ability. - The professors cited Claude as an example of how users perceive intelligence through successful completion of complex requests, not merely high benchmark scores. - They recommended competing through practical execution in real service environments rather than focusing only on text-generation performance. ## Industry-Academic Cooperation - Kakao plans to explore GPU support for university research labs and undergraduate AI clubs. - Possible support could include credits, project-based resources, and other forms of infrastructure assistance. - The seminar marked the beginning of a broader collaboration aimed at advancing Korea’s AI ecosystem and developing future talent. Kakao’s recommended path is to combine proprietary, efficient models with privacy-preserving personalization and strong agentic execution. Success will depend on how effectively Kanana turns technical depth into useful intelligence that users can experience in everyday services.

gpuaiai-agentmultimodal-ai+4

netflixFeb 13, 2026

Scaling LLM Post-Training at Netflix (opens in new tab)

Netflix argues that LLM post-training at production scale is as much an infrastructure challenge as a modeling challenge. Its internal framework abstracts distributed data processing, model sharding, GPU orchestration, checkpointing, and complex training workflows so developers can focus on experimentation. The result is a flexible system supporting SFT, DPO, reinforcement learning, and knowledge distillation across hundreds of GPUs. ## Why Post-Training Becomes an Engineering Problem - Pre-training provides general language ability, but post-training adapts models to Netflix’s catalog, member histories, recommendation tasks, personalization, and search. - Production-scale training introduces challenges involving: - Large proprietary datasets - Multi-node GPU coordination - Distributed model state - Workflows that combine training and inference - Failure recovery and experiment tracking - A simple Hugging Face fine-tuning script is insufficient for reliable, large-scale jobs. ## Preparing Data Correctly - Chat templates serialize conversations but do not determine which tokens should contribute to the loss. - Netflix applies explicit loss masking so training focuses on assistant responses rather than prompts or other non-target text. - Variable-length examples can waste GPU memory through padding and create synchronization overhead across FSDP workers. - Sequence packing combines multiple samples into fixed-length sequences. - A document mask prevents attention across separately packed samples while improving GPU utilization. ## Loading and Optimizing Large Models - Models that do not fit on one GPU require sharding strategies such as FSDP or tensor parallelism. - Partial weights should be loaded directly onto the device mesh rather than materializing the entire checkpoint on a single device. - Developers can choose full fine-tuning or LoRA and use: - Activation checkpointing - Compilation - Appropriate precision settings - Reinforcement learning requires compatible precision between rollout generation and policy training. - Large vocabularies create memory pressure because logits have dimensions `[batch, seq_len, vocab]`. - The framework reduces peak memory by removing ignored tokens before projection and computing logits and loss in sequence chunks. ## Distributed Training and Workflow Management - The framework supports standard forward/backward training for SFT as well as workflows that interleave: - Rollout generation - Reward-model and reference-model inference - Policy updates - Ray actors orchestrate distributed jobs while keeping hardware concerns separate from modeling code. - Experiment tracking covers both quality metrics, such as loss, and efficiency metrics, such as Model FLOPS Utilization (MFU). - Standardized checkpointing allows jobs to resume after failures. ## Netflix’s Post-Training Framework - The stack is built on: - Mako for AWS GPU provisioning - PyTorch, Ray, and vLLM - Netflix’s framework library for reusable utilities and training recipes - Jobs are generally defined through configuration files that select a recipe and provide task-specific components. - Unlike narrower fine-tuning systems, the framework supports: - Custom output heads - Expanded vocabularies and semantic IDs - Special tokens - Transformer models trained on non-natural-language sequences - This flexibility is important for Netflix-specific recommendation and personalization use cases. ## Four Core Abstractions ### Data - Dataset abstractions cover SFT, reward modeling, and RL. - Streaming supports datasets larger than local disk capacity. - Asynchronous sequence packing overlaps CPU preprocessing with GPU execution to reduce idle time. ### Model - The framework supports architectures such as Qwen3 and Gemma3, including Mixture-of-Experts variants. - LoRA is integrated into model definitions. - High-level sharding APIs distribute models across device meshes without requiring developers to write low-level distributed code. ### Compute - A unified job interface scales from one node to hundreds of GPUs. - MFU measurement remains accurate for custom architectures and LoRA configurations. - Checkpoints include parameters, optimizer state, dataloader state, and data-mixer state, enabling exact resumption. ### Workflow - The system supports SFT, DPO, RL, and knowledge distillation. - Online RL uses a hybrid architecture combining a single controller with Single Program, Multiple Data (SPMD) workers. - This extends conventional SPMD training to multi-stage workflows that cannot be represented as a simple training loop. Netflix’s approach is to standardize the difficult operational parts of post-training while preserving enough flexibility for unconventional models and objectives. A framework built around reusable data, model, compute, and workflow abstractions can help teams iterate faster and scale experiments without repeatedly rebuilding distributed infrastructure.

gpudatabase-designllmcloud-computing+4