Knowledge Distillation

kakaoJul 27, 20264 min readCurated summary

Smaller and More Powerful Kanana SLM Development

Kanana-2 is Kakao’s second Small Language Model series, designed to deliver strong performance under the memory and compute constraints of on-device environments. It includes 3B, 1.3B, and 0.9B models, using improved pre-training, distillation, pruning, tokenizer efficiency, and Sliding Window Attention. Kakao reports that the resulting models outperform earlier Kanana models and compare favorably with similarly sized open-source models. ## Motivation and Model Lineup - On-device services require models that are small and fast because smartphones have limited memory and compute. - Kakao uses its own SLMs in the “Kanana in KakaoTalk” service. - The Kanana-2 series consists of: - Kanana-2-3B - Kanana-2-1.3B - Kanana-2-0.9B - Base and Instruct versions of the 3B and 1.3B models are being released. - The development builds on techniques from Kanana-2-30B-A3B and earlier Kanana Nano models. ## Efficiency Improvements - **Kanana-2 Tokenizer** - Improves Korean tokenization efficiency by more than 30% compared with the previous tokenizer. - Reduces the number of tokens required for Korean text, improving processing efficiency. - **Sliding Window Attention** - Reduces KV cache size during inference. - Improves memory efficiency and helps reduce decoding bottlenecks on devices. ## 3B Pre-Training ### TPU-Based Training from Scratch - Kanana-2-3B-Base was initially trained from scratch on a TPU v5e cluster. - Kakao used a MaxText-based internal training framework. - The team developed infrastructure allowing training to transition between TPU and GPU clusters: - Pre-training was completed on TPU. - Distillation was subsequently performed on GPUs using Megatron-LM. - Pre-training used two stages: - Stage 1: 7.5 trillion tokens - Stage 2: 2 trillion tokens - The Muon optimizer was used throughout pre-training. ### Learning-Rate Scaling - Directly searching learning rates at multi-trillion-token scale would be too expensive. - Kakao instead tested learning-rate candidates using a 100-billion-token proxy dataset while preserving the Stage 1 data distribution. - The selected learning rate was scaled to the full 7.5-trillion-token training run using the Token Horizon scaling rule: `LR_target ≈ LR_proxy × (D_target / D_proxy)^−β` - The experiments used: - `D_proxy = 100B` - `D_target = 7.5T` - `β = 0.32` - This approach enabled stable hyperparameter selection with a smaller exploration budget. ## Teacher-Based Distillation - The Kanana-2-30B-A3B-Instruct-2601 model was used as the teacher. - Kakao compared Base, Instruct, and Thinking versions of the teacher model. - The Instruct teacher consistently produced the strongest student-model performance. - The result supports recent findings that post-trained teachers can be especially effective for transferring mathematical and coding capabilities. ## Long-Context Training - The model’s context length was expanded from 4K to 32K using YaRN. - Additional mid-training data was introduced during the learning-rate decay phase. - The resulting Kanana-2-3B-Base reportedly surpassed earlier Kanana 3B models across Korean and English knowledge, mathematics, and coding. - It also exceeded many similarly sized open-source SOTA base models. ## Building the 1.3B and 0.9B Models - Kanana-2-1.3B-Base and Kanana-2-0.9B-Base were progressively derived from Kanana-2-3B-Base. - The process extended the Minitron-based structured pruning and knowledge-distillation approach used for Kanana Nano. - The smaller models were trained with SWA-aware long-context procedures to preserve efficiency in on-device decoding. ### Improved Hidden-Dimension Pruning - Traditional hidden-dimension pruning scores each dimension independently using activation statistics from calibration data. - This is efficient but may overlook information represented jointly across multiple dimensions. - Kanana-2 applies PCA-based pruning inspired by Ministral 3: - Collect activation statistics from Attention RMSNorm, MLP RMSNorm, and Final RMSNorm inputs. - Compute a global rotation matrix with PCA. - Apply the rotation consistently to token embeddings and attention/MLP projection weights. - Reduce hidden dimensions after rotation, aiming to preserve more meaningful shared representations. Kakao’s approach combines large-scale pre-training, teacher distillation, structured compression, better Korean tokenization, and attention-level inference optimization. For practical on-device deployment, the smaller Kanana-2 models are intended to offer a better balance of speed, memory usage, and capability than simply deploying a larger general-purpose LLM.

Read original

Knowledge Distillation Kanana Tpu Small Language Model+4

lineMar 6, 20263 min readCurated summary

On-Device Image Model Training for Mess

This post describes an on-device image captioning system for mobile messenger apps. Because autoregressive vision-language models took more than five seconds to generate captions, the team replaced them with a non-autoregressive decoder, reducing latency to roughly 200–400 ms. They then used LLM-based acceptance evaluation, caption re-generation, and multi-stage knowledge distillation to improve quality while keeping the model at 172 MB. ## Why Conventional Captioning Was Unsuitable - Models such as BLIP-2, MobileVLM, PaliGemma, and MiniCPM were too large or slow for mobile deployment. - BLIP-1 was selected as a practical baseline because of its smaller size and clear licensing, but still required more than five seconds after quantization. - Autoregressive decoding generates tokens sequentially, requiring one decoder pass per token. - On a Samsung Galaxy Fold 4, the initial model required about 142 ms per token, or approximately 2.8 seconds for 20 tokens. - Mobile UX required stable latency in the hundreds of milliseconds, including cold-start and variable-device conditions, so simple model compression was insufficient. ## Non-Autoregressive Caption Generation - The system predicts all caption tokens in parallel using a fixed set of learnable query tokens. - This changes the decoding cost from roughly O(T) for autoregressive generation to near O(1) through parallel processing. - The architecture consists of: - An image encoder reused from the previous system - Image embeddings injected as a prefix, following the ClipCap approach - A 66.4-million-parameter Transformer-based text decoder - Twenty learnable query tokens for short captions - Query-CTC loss addresses the alignment problem caused by predicting tokens simultaneously. - The resulting model generated captions in about 200 ms, achieving the required speed improvement. ## Speed Improved, but Caption Quality Declined - Standard metrics such as CIDEr and CLIPScore appeared acceptable. - Manual inspection revealed frequent: - Repeated words, such as “a desk with a computer on a desk” - Spelling errors, such as “a people ons” - Grammatical problems - Incomplete captions, such as “a” - These defects made the model unsuitable for direct use in a messaging product. ## LLM-Based Acceptance Evaluation - The team introduced an “accept ratio” based on GPT-4o mini judgments. - Captions were classified as either `accept` or `non-accept`. - The evaluation checked for: - Duplicate content - Errors - Clarity and grammatical correctness - This better reflected production usability than conventional image-captioning benchmarks. - The low acceptance rate confirmed that CIDEr and CLIPScore alone could not measure whether captions were appropriate for users. ## Data Quality and Knowledge Distillation - Analysis showed that the training data contained inconsistent and noisy captions: - A mixture of very short and overly long descriptions - Unnecessary OCR-like attempts to describe text in images - Uneven language quality - The smaller 66.4-million-parameter model also had less representational capacity than BLIP-1’s 110 million parameters. - Generating an entire sentence in one pass was especially difficult for the compact non-autoregressive model. - The training pipeline was redesigned as an iterative quality-improvement loop: - Train a baseline using the original data - Identify failures with LLM-based acceptance evaluation - Re-caption poor-quality training examples - Distill knowledge from a larger teacher model into the student model - Replace or refine rejected samples and repeat - Architectural scaling and metric optimization did not consistently improve acceptance rates, while re-captioning and knowledge distillation produced more meaningful gains. The practical recommendation is to design on-device captioning around the actual product experience: prioritize parallel decoding for latency, measure quality with production-oriented acceptance criteria, and use carefully curated data plus knowledge distillation to make compact models reliable.

Read original

Knowledge Distillation Database Design Large Language Models Non Autoregressive+3

lineMar 6, 20264 min readCurated summary

On-Device Image Model

The post describes building an on-device image understanding system for messaging apps, with semantic image search as the first focus. Its central strategy was knowledge distillation: a multilingual student text encoder learned to reproduce the embedding space of a strong, English-only teacher model. This preserved most English retrieval quality while enabling Japanese, Traditional Chinese, Thai, and Korean search, achieving an average Recall@5 above 78%. ## Why Messaging Apps Need On-Device Image Understanding - Images are often treated simply as “a photo,” unlike text messages, which can support search, summaries, and notification previews. - Image understanding could improve: - Notifications: “Sent one photo” → “Sent a photo of a dog” - Search: queries such as “dog,” “puppy,” or “a cat inside a box” - Recommendations: automatic image classification and organization - Shared image-text embeddings allow semantically equivalent phrases—such as “dog,” “puppy,” and “개”—to retrieve the same images. ## Why the Model Had to Run On-Device - **Latency:** Network round trips make notifications and search less predictable and responsive. - **Privacy:** Sending photos, captions, or embeddings to a server increases privacy risks. - **Offline support:** The feature should work in subways, airplanes, roaming environments, and unstable networks. - **Mobile constraints:** The model needed to run on both Android and iOS with limited memory and compute resources. - The project targeted a model under **200 MB**, response times within a few hundred milliseconds including cold start, and LiteRT compatibility. ## Project Goals and Evaluation - The image search system needed to: - Retrieve images by semantic meaning rather than keyword matching. - Support English, Japanese, Traditional Chinese, Thai, and Korean. - A separate captioning system was designed to generate short, natural descriptions of roughly eight words or fewer. - Search quality was measured using: - Image-to-Text Recall@5 - Text-to-Image Recall@5 - Caption quality was evaluated with CIDEr, CLIPScore, and an LLM-based acceptance ratio designed to detect repetition, typos, and grammatical problems. ## Why Translation Was Not Enough The initial approach translated each query into English before using an English-only image-text model: ```text Query → Language detection → Translation → English text encoder → Embedding → Search ``` This approach introduced several problems: - **Quality loss:** Informal terms or short queries could be mistranslated. For example, “멍멍이” might be interpreted as “barking” instead of “dog.” - **Additional latency:** Translation adds a fixed cost before text encoding. - **Inconsistent results:** Translation quality varies by language pair and wording. - **Operational complexity:** Each additional language requires more models, updates, and failure handling. Training a multilingual image-text model from scratch would require substantial data and compute. Instead, the project retained the proven English image embedding space and expanded only the text encoder. ## Knowledge Distillation for Multilingual Search - The original English text encoder served as the frozen **teacher**. - A copied text encoder served as the trainable **student**. - English text was passed to the teacher, while corresponding multilingual text was passed to the student. - The student was trained to match the teacher’s embeddings using mean squared error (MSE). ```text teacher_embedding = teacher(English text) student_embedding = student(Multilingual text) loss = MSE(teacher_embedding, student_embedding) ``` The image encoder remained frozen so that the established image-text embedding space would not be disrupted. Important implementation considerations included: - Ensuring the tokenizer handled multilingual characters correctly. - Defining consistent case-insensitivity rules. - Balancing training samples across languages. - Matching training-time preprocessing and tokenization with mobile inference behavior. ## Retrieval Results - English performance declined slightly: - Image-to-Text Recall@5: **79.58% → 76.56%** - Text-to-Image Recall@5: **75.89% → 74.47%** - Multilingual performance improved from below **10% average Recall@5** to above **78%**, roughly a sevenfold improvement. - Japanese achieved **81.94%**, exceeding the original English model in the reported evaluation. - Traditional Chinese, Thai, and Korean also reached practically usable retrieval quality. The trade-off—slightly lower English performance in exchange for four additional languages—provided substantially greater overall product value. ## Converting the Model to LiteRT - LiteRT was selected because it officially supports both Android and iOS and provides mobile-oriented operators, quantization, and optimization tools. - Core ML was rejected because it is iOS-specific and introduced conversion and long-term cross-platform maintenance concerns. - Conversion required addressing unsupported PyTorch operators. - For example, LiteRT did not support `erf`, so the model’s implementation had to replace it with a compatible pseudo-`erf` operation. The resulting approach demonstrates that knowledge distillation can efficiently extend an existing English image-text model to multiple languages while preserving its on-device deployment advantages.

Read original

Knowledge Distillation Pytorch Litert Embedding Model+4

metaJan 14, 2026Original article

Adapting the Facebook Reels RecSys AI Model Based on User Feedback (opens in new tab)

Meta has enhanced the Facebook Reels recommendation engine by shifting focus from traditional engagement signals, like watch time and likes, to direct user feedback. By implementing the User True Interest Survey (UTIS) model, the system now prioritizes content that aligns with genuine user preferences rather than just short-term interactions. This shift has resulted in significant improvements in recommendation relevance, high-quality content delivery, and long-term user retention. **Limitations of Engagement-Based Metrics** * Traditional signals like "likes" and "watch time" are often noisy and may not reflect a user’s actual long-term interests. * Models optimized solely for engagement tend to favor short-term value over the long-term utility of the product. * Internal research found that previous heuristic-based interest models only achieved 48.3% precision in identifying what users truly care about. * Effective interest matching requires understanding nuanced factors such as production style, mood, audio, and motivation, which implicit signals often miss. **The User True Interest Survey (UTIS) Model** * Meta collects direct feedback via randomized, single-question surveys asking users to rate video interest on a 1–5 scale. * The raw survey data is binarized to denoise responses and weighted to correct for sampling and nonresponse bias. * The UTIS model functions as a lightweight "alignment model layer" built on top of the main multi-task ranking system. * The architecture uses existing model predictions as input features, supplemented by engineered features that capture content attributes and user behavior. **Integration into the Ranking Funnel** * **Late Stage Ranking (LSR):** The UTIS score is used as an additional input feature in the final value formula, allowing the system to boost high-interest videos and demote low-interest ones. * **Early Stage Ranking (Retrieval):** The model aggregates survey data to reconstruct user interest profiles, helping the system source more relevant candidates during the initial retrieval phase. * **Knowledge Distillation:** Large sequence-based retrieval models are aligned using UTIS predictions as labels through distillation objectives. **Performance and Impact** * The deployment of UTIS has led to a measurable increase in the delivery of niche, high-quality content. * Generic, popularity-based recommendations that often lack depth have been reduced. * Meta observed robust improvements across core metrics, including higher follow rates, more shares, and increased user retention. * The system now offers better interpretability, allowing engineers to understand which specific factors contribute to a user’s sense of "interest match." To continue improving the Reels ecosystem, Meta is focusing on doubling down on personalization by tackling challenges related to sparse data and sampling bias while exploring more advanced AI architectures to further diversify recommendations.

Knowledge Distillation AI Machine Learning Recommendation Systems+5

googleSep 30, 2025Original article

Introducing interactive on-device segmentation in Snapseed (opens in new tab)

Google has introduced a new "Object Brush" feature in Snapseed that enables intuitive, real-time selective photo editing through a novel on-device segmentation technology. By leveraging a high-performance interactive AI model, users can isolate complex subjects with simple touch gestures in under 20 milliseconds, bridging the gap between professional-grade editing and mobile convenience. This breakthrough is achieved through a sophisticated teacher-student training architecture that prioritizes both pixel-perfect accuracy and low-latency performance on consumer hardware. ### High-Performance On-Device Inference * The system is powered by the Interactive Segmenter model, which is integrated directly into the Snapseed "Adjust" tool to facilitate immediate object-based modifications. * To ensure a fluid user experience, the model utilizes the MediaPipe framework and LiteRT’s GPU acceleration to process selections in less than 20ms. * The interface supports dynamic refinement, allowing users to provide real-time feedback by tracing lines or tapping to add or subtract specific areas of an image. ### Teacher-Student Model Distillation * The development team first created "Interactive Segmenter: Teacher," a large-scale model fine-tuned on 30,000 high-quality, pixel-perfect manual annotations across more than 350 object categories. * Because the Teacher model’s size and computational requirements are prohibitive for mobile use, researchers developed "Interactive Segmenter: Edge" through knowledge distillation. * This distillation process utilized a dataset of over 2 million weakly annotated images, allowing the smaller Edge model to inherit the generalization capabilities of the Teacher model while maintaining a footprint suitable for mobile devices. ### Training via Synthetic User Prompts * To make the model universally capable across all object types, the training process uses a class-agnostic approach based on the Big Transfer (BiT) strategy. * The model learns to interpret user intent through "prompt generation," which simulates real-world interactions such as random scribbles, taps, and lasso (box) selections. * During training, both the Teacher and Edge models receive identical prompts—such as red foreground scribbles and blue background scribbles—to ensure the student model learns to produce high-quality masks even from imprecise user input. This advancement significantly lowers the barrier to entry for complex photo manipulation by moving heavy-duty AI processing directly onto the mobile device. Users can expect a more responsive and precise editing experience that handles everything from fine-tuning a subject's lighting to isolating specific environmental elements like clouds or clothing.

Knowledge Distillation AI Machine Learning Computer Vision+5

googleAug 20, 2025Original article

From massive models to mobile magic: The tech behind YouTube real-time generative AI effects (opens in new tab)

YouTube has successfully deployed over 20 real-time generative AI effects by distilling the capabilities of massive cloud-based models into compact, mobile-ready architectures. By utilizing a "teacher-student" training paradigm, the system overcomes the computational bottlenecks of high-fidelity generative AI while ensuring the output remains responsive on mobile hardware. This approach allows for complex transformations, such as cartoon style transfer and makeup application, to run frame-by-frame on-device without sacrificing the user’s identity. ### Data Curation and Diversity * The foundation of the effects pipeline relies on high-quality, properly licensed face datasets. * Datasets are meticulously filtered to ensure a uniform distribution across different ages, genders, and skin tones. * The Monk Skin Tone Scale is used as a benchmark to ensure the effects work equitably for all users. ### The Teacher-Student Framework * **The Teacher:** A large, powerful pre-trained model (initially StyleGAN2 with StyleCLIP, later transitioning to Google DeepMind’s Imagen) acts as the "expert" that generates high-fidelity visual effects. * **The Student:** A lightweight UNet-based architecture designed for mobile efficiency. It utilizes a MobileNet backbone for both the encoder and decoder to ensure fast frame-by-frame processing. * The distillation process narrows the scope of the massive teacher model into a student model focused on a single, specific task. ### Iterative Distillation and Training * **Data Generation:** The teacher model processes thousands of images to create "before and after" pairs. These are augmented with synthetic elements like AR glasses, sunglasses, and hand occlusions to improve real-world robustness. * **Optimization:** The student model is trained using a sophisticated combination of loss functions, including L1, LPIPS, Adaptive, and Adversarial loss, to balance numerical accuracy with aesthetic quality. * **Architecture Search:** Neural architecture search is employed to tune "depth" and "width" multipliers, identifying the most efficient model structure for different mobile hardware constraints. ### Addressing the Inversion Problem * A major challenge in real-time effects is the "inversion problem," where the model struggles to represent a real face in latent space, leading to a loss of the user's identity (e.g., changes in skin tone or clothing). * YouTube uses Pivotal Tuning Inversion (PTI) to ensure that the user's specific features are preserved during the generative process. * By editing images in the latent space—a compressed numerical representation—the system can apply stylistic changes while maintaining the core characteristics of the original video stream. By combining advanced model distillation with on-device optimization via MediaPipe, YouTube demonstrates a practical path for bringing heavy generative AI research into consumer-facing mobile applications.

Knowledge Distillation AI Generative AI Computer Vision+5