Smaller and More Powerful Kanana SLM Development

Kanana-2 is Kakao’s second Small Language Model series, designed to deliver strong performance under the memory and compute constraints of on-device environments. It includes 3B, 1.3B, and 0.9B models, using improved pre-training, distillation, pruning, tokenizer efficiency, and Sliding Window Attention. Kakao reports that the resulting models outperform earlier Kanana models and compare favorably with similarly sized open-source models.

Motivation and Model Lineup

• On-device services require models that are small and fast because smartphones have limited memory and compute.
• Kakao uses its own SLMs in the “Kanana in KakaoTalk” service.
• The Kanana-2 series consists of:
  • Kanana-2-3B
  • Kanana-2-1.3B
  • Kanana-2-0.9B
• Base and Instruct versions of the 3B and 1.3B models are being released.
• The development builds on techniques from Kanana-2-30B-A3B and earlier Kanana Nano models.

Efficiency Improvements

• Kanana-2 Tokenizer
  • Improves Korean tokenization efficiency by more than 30% compared with the previous tokenizer.
  • Reduces the number of tokens required for Korean text, improving processing efficiency.
• Sliding Window Attention
  • Reduces KV cache size during inference.
  • Improves memory efficiency and helps reduce decoding bottlenecks on devices.

3B Pre-Training

TPU-Based Training from Scratch

• Kanana-2-3B-Base was initially trained from scratch on a TPU v5e cluster.
• Kakao used a MaxText-based internal training framework.
• The team developed infrastructure allowing training to transition between TPU and GPU clusters:
  • Pre-training was completed on TPU.
  • Distillation was subsequently performed on GPUs using Megatron-LM.
• Pre-training used two stages:
  • Stage 1: 7.5 trillion tokens
  • Stage 2: 2 trillion tokens
• The Muon optimizer was used throughout pre-training.

Learning-Rate Scaling

• Directly searching learning rates at multi-trillion-token scale would be too expensive.

• Kakao instead tested learning-rate candidates using a 100-billion-token proxy dataset while preserving the Stage 1 data distribution.

• The selected learning rate was scaled to the full 7.5-trillion-token training run using the Token Horizon scaling rule:

  LR_target ≈ LR_proxy × (D_target / D_proxy)^−β

• The experiments used:

  • D_proxy = 100B
  • D_target = 7.5T
  • β = 0.32

• This approach enabled stable hyperparameter selection with a smaller exploration budget.

Teacher-Based Distillation

• The Kanana-2-30B-A3B-Instruct-2601 model was used as the teacher.
• Kakao compared Base, Instruct, and Thinking versions of the teacher model.
• The Instruct teacher consistently produced the strongest student-model performance.
• The result supports recent findings that post-trained teachers can be especially effective for transferring mathematical and coding capabilities.

Long-Context Training

• The model’s context length was expanded from 4K to 32K using YaRN.
• Additional mid-training data was introduced during the learning-rate decay phase.
• The resulting Kanana-2-3B-Base reportedly surpassed earlier Kanana 3B models across Korean and English knowledge, mathematics, and coding.
• It also exceeded many similarly sized open-source SOTA base models.

Building the 1.3B and 0.9B Models

• Kanana-2-1.3B-Base and Kanana-2-0.9B-Base were progressively derived from Kanana-2-3B-Base.
• The process extended the Minitron-based structured pruning and knowledge-distillation approach used for Kanana Nano.
• The smaller models were trained with SWA-aware long-context procedures to preserve efficiency in on-device decoding.

Improved Hidden-Dimension Pruning

• Traditional hidden-dimension pruning scores each dimension independently using activation statistics from calibration data.
• This is efficient but may overlook information represented jointly across multiple dimensions.
• Kanana-2 applies PCA-based pruning inspired by Ministral 3:
  • Collect activation statistics from Attention RMSNorm, MLP RMSNorm, and Final RMSNorm inputs.
  • Compute a global rotation matrix with PCA.
  • Apply the rotation consistently to token embeddings and attention/MLP projection weights.
  • Reduce hidden dimensions after rotation, aiming to preserve more meaningful shared representations.

Kakao’s approach combines large-scale pre-training, teacher distillation, structured compression, better Korean tokenization, and attention-level inference optimization. For practical on-device deployment, the smaller Kanana-2 models are intended to offer a better balance of speed, memory usage, and capability than simply deploying a larger general-purpose LLM.