kakao·Jul 27, 2026·4 min readCurated summary

Smaller and More Powerful Kanana SLM Development

Kanana-2 is Kakao’s second Small Language Model series, designed to deliver strong performance under the memory and compute constraints of on-device environments. It includes 3B, 1.3B, and 0.9B models, using improved pre-training, distillation, pruning, tokenizer efficiency, and Sliding Window Attention. Kakao reports that the resulting models outperform earlier Kanana models and compare favorably with similarly sized open-source models. ## Motivation and Model Lineup - On-device services require models that are small and fast because smartphones have limited memory and compute. - Kakao uses its own SLMs in the “Kanana in KakaoTalk” service. - The Kanana-2 series consists of: - Kanana-2-3B - Kanana-2-1.3B - Kanana-2-0.9B - Base and Instruct versions of the 3B and 1.3B models are being released. - The development builds on techniques from Kanana-2-30B-A3B and earlier Kanana Nano models. ## Efficiency Improvements - **Kanana-2 Tokenizer** - Improves Korean tokenization efficiency by more than 30% compared with the previous tokenizer. - Reduces the number of tokens required for Korean text, improving processing efficiency. - **Sliding Window Attention** - Reduces KV cache size during inference. - Improves memory efficiency and helps reduce decoding bottlenecks on devices. ## 3B Pre-Training ### TPU-Based Training from Scratch - Kanana-2-3B-Base was initially trained from scratch on a TPU v5e cluster. - Kakao used a MaxText-based internal training framework. - The team developed infrastructure allowing training to transition between TPU and GPU clusters: - Pre-training was completed on TPU. - Distillation was subsequently performed on GPUs using Megatron-LM. - Pre-training used two stages: - Stage 1: 7.5 trillion tokens - Stage 2: 2 trillion tokens - The Muon optimizer was used throughout pre-training. ### Learning-Rate Scaling - Directly searching learning rates at multi-trillion-token scale would be too expensive. - Kakao instead tested learning-rate candidates using a 100-billion-token proxy dataset while preserving the Stage 1 data distribution. - The selected learning rate was scaled to the full 7.5-trillion-token training run using the Token Horizon scaling rule: `LR_target ≈ LR_proxy × (D_target / D_proxy)^−β` - The experiments used: - `D_proxy = 100B` - `D_target = 7.5T` - `β = 0.32` - This approach enabled stable hyperparameter selection with a smaller exploration budget. ## Teacher-Based Distillation - The Kanana-2-30B-A3B-Instruct-2601 model was used as the teacher. - Kakao compared Base, Instruct, and Thinking versions of the teacher model. - The Instruct teacher consistently produced the strongest student-model performance. - The result supports recent findings that post-trained teachers can be especially effective for transferring mathematical and coding capabilities. ## Long-Context Training - The model’s context length was expanded from 4K to 32K using YaRN. - Additional mid-training data was introduced during the learning-rate decay phase. - The resulting Kanana-2-3B-Base reportedly surpassed earlier Kanana 3B models across Korean and English knowledge, mathematics, and coding. - It also exceeded many similarly sized open-source SOTA base models. ## Building the 1.3B and 0.9B Models - Kanana-2-1.3B-Base and Kanana-2-0.9B-Base were progressively derived from Kanana-2-3B-Base. - The process extended the Minitron-based structured pruning and knowledge-distillation approach used for Kanana Nano. - The smaller models were trained with SWA-aware long-context procedures to preserve efficiency in on-device decoding. ### Improved Hidden-Dimension Pruning - Traditional hidden-dimension pruning scores each dimension independently using activation statistics from calibration data. - This is efficient but may overlook information represented jointly across multiple dimensions. - Kanana-2 applies PCA-based pruning inspired by Ministral 3: - Collect activation statistics from Attention RMSNorm, MLP RMSNorm, and Final RMSNorm inputs. - Compute a global rotation matrix with PCA. - Apply the rotation consistently to token embeddings and attention/MLP projection weights. - Reduce hidden dimensions after rotation, aiming to preserve more meaningful shared representations. Kakao’s approach combines large-scale pre-training, teacher distillation, structured compression, better Korean tokenization, and attention-level inference optimization. For practical on-device deployment, the smaller Kanana-2 models are intended to offer a better balance of speed, memory usage, and capability than simply deploying a larger general-purpose LLM.

Read original(opens in new tab)

TpuKnowledge DistillationKananaSmall Language Model+4

google·Jul 9, 2025Original article

Graph foundation models for relational data (opens in new tab)

Google researchers have introduced Graph Foundation Models (GFMs) as a solution to the limitations of traditional tabular machine learning, which often ignores the rich connectivity of relational databases. By representing tables as interconnected graphs where rows are nodes and foreign keys are edges, this approach enables a single model to generalize across entirely different schemas and feature sets. This shift allows for transferable graph representations that can perform inference on unseen tasks without the costly need for domain-specific retraining. ### Transforming Relational Schemas into Graphs The core methodology involves a scalable data preparation step that converts standard relational database structures into a single heterogeneous graph. This process preserves the underlying logic of the data while making it compatible with graph-based learning: * **Node Mapping:** Each unique table is treated as a node type, and every individual row within that table is converted into a specific node. * **Edge Creation:** Foreign key relationships are transformed into typed edges that connect nodes across different tables. * **Feature Integration:** Standard columns containing numerical or categorical data are converted into node features, while temporal data can be preserved as features on either nodes or edges. ### Overcoming the Generalization Gap A primary hurdle in developing GFMs is the lack of a universal tokenization method, unlike the word pieces used in language models or patches used in vision models. Traditional Graph Neural Networks (GNNs) are typically locked to the specific graph they were trained on, but GFMs solve this through several technical innovations: * **Schema Agnosticism:** The model avoids hard-coded embedding tables for specific node types, allowing it to interpret database schemas it has never encountered during training. * **Feature Interaction Learning:** Instead of training on "absolute" features (like specific price distributions), the model captures how different features interact with one another across diverse tasks. * **Generalizable Encoders:** The architecture uses transferable methods to derive fixed-size representations for nodes, whether they contain three continuous float features or dozens of categorical values. ### Scaling and Real-World Application To handle the requirements of enterprise-level data, the GFM framework is built to operate on a massive scale using Google’s specialized infrastructure: * **Massive Throughput:** The system utilizes JAX and TPU infrastructure to process graphs containing billions of nodes and edges. * **Internal Validation:** The model has been tested on complex internal Google tasks, such as spam detection in advertisements, which requires analyzing dozens of interconnected relational tables simultaneously. * **Performance Benefits:** By considering the connections between rows—a factor traditional tabular baselines like decision trees often ignore—the GFM provides superior downstream performance in high-stakes prediction services. Transitioning from domain-specific models to Graph Foundation Models allows organizations to leverage relational data more holistically. By focusing on the connectivity of data rather than just isolated table features, GFMs provide a path toward a single, generalist model capable of handling diverse enterprise tasks.

TpuAITransformerJax+5