Smaller and More Powerful Kanana SLM Development
Kanana-2 is Kakao’s second Small Language Model series, designed to deliver strong performance under the memory and compute constraints of on-device environments. It includes 3B, 1.3B, and 0.9B models, using improved pre-training, distillation, pruning, tokenizer efficiency, and Sliding Window Attention. Kakao reports that the resulting models outperform earlier Kanana models and compare favorably with similarly sized open-source models. ## Motivation and Model Lineup - On-device services require models that are small and fast because smartphones have limited memory and compute. - Kakao uses its own SLMs in the “Kanana in KakaoTalk” service. - The Kanana-2 series consists of: - Kanana-2-3B - Kanana-2-1.3B - Kanana-2-0.9B - Base and Instruct versions of the 3B and 1.3B models are being released. - The development builds on techniques from Kanana-2-30B-A3B and earlier Kanana Nano models. ## Efficiency Improvements - **Kanana-2 Tokenizer** - Improves Korean tokenization efficiency by more than 30% compared with the previous tokenizer. - Reduces the number of tokens required for Korean text, improving processing efficiency. - **Sliding Window Attention** - Reduces KV cache size during inference. - Improves memory efficiency and helps reduce decoding bottlenecks on devices. ## 3B Pre-Training ### TPU-Based Training from Scratch - Kanana-2-3B-Base was initially trained from scratch on a TPU v5e cluster. - Kakao used a MaxText-based internal training framework. - The team developed infrastructure allowing training to transition between TPU and GPU clusters: - Pre-training was completed on TPU. - Distillation was subsequently performed on GPUs using Megatron-LM. - Pre-training used two stages: - Stage 1: 7.5 trillion tokens - Stage 2: 2 trillion tokens - The Muon optimizer was used throughout pre-training. ### Learning-Rate Scaling - Directly searching learning rates at multi-trillion-token scale would be too expensive. - Kakao instead tested learning-rate candidates using a 100-billion-token proxy dataset while preserving the Stage 1 data distribution. - The selected learning rate was scaled to the full 7.5-trillion-token training run using the Token Horizon scaling rule: `LR_target ≈ LR_proxy × (D_target / D_proxy)^−β` - The experiments used: - `D_proxy = 100B` - `D_target = 7.5T` - `β = 0.32` - This approach enabled stable hyperparameter selection with a smaller exploration budget. ## Teacher-Based Distillation - The Kanana-2-30B-A3B-Instruct-2601 model was used as the teacher. - Kakao compared Base, Instruct, and Thinking versions of the teacher model. - The Instruct teacher consistently produced the strongest student-model performance. - The result supports recent findings that post-trained teachers can be especially effective for transferring mathematical and coding capabilities. ## Long-Context Training - The model’s context length was expanded from 4K to 32K using YaRN. - Additional mid-training data was introduced during the learning-rate decay phase. - The resulting Kanana-2-3B-Base reportedly surpassed earlier Kanana 3B models across Korean and English knowledge, mathematics, and coding. - It also exceeded many similarly sized open-source SOTA base models. ## Building the 1.3B and 0.9B Models - Kanana-2-1.3B-Base and Kanana-2-0.9B-Base were progressively derived from Kanana-2-3B-Base. - The process extended the Minitron-based structured pruning and knowledge-distillation approach used for Kanana Nano. - The smaller models were trained with SWA-aware long-context procedures to preserve efficiency in on-device decoding. ### Improved Hidden-Dimension Pruning - Traditional hidden-dimension pruning scores each dimension independently using activation statistics from calibration data. - This is efficient but may overlook information represented jointly across multiple dimensions. - Kanana-2 applies PCA-based pruning inspired by Ministral 3: - Collect activation statistics from Attention RMSNorm, MLP RMSNorm, and Final RMSNorm inputs. - Compute a global rotation matrix with PCA. - Apply the rotation consistently to token embeddings and attention/MLP projection weights. - Reduce hidden dimensions after rotation, aiming to preserve more meaningful shared representations. Kakao’s approach combines large-scale pre-training, teacher distillation, structured compression, better Korean tokenization, and attention-level inference optimization. For practical on-device deployment, the smaller Kanana-2 models are intended to offer a better balance of speed, memory usage, and capability than simply deploying a larger general-purpose LLM.
Read original(opens in new tab)