From AI That Speaks Well to AI That Speaks Exactly as Desired: Advancing Kanana-o Voice Generation

Kanana-o’s latest speech-generation improvements target two goals: faster, more efficient synthesis and more precise adherence to user instructions. Kakao addresses these through LM-SPT, a speech tokenizer that separates semantic and acoustic information while reducing the token rate from 25 Hz to 12.5 Hz, and through online reinforcement learning. Together, these changes enable Kanana-o to generate natural speech more efficiently and control characteristics such as speed, pitch, tone, and volume more reliably.

Goals for Kanana-o’s Speech Generation

• Improve real-time performance by shortening speech-token sequences and simplifying decoding.
• Move beyond merely natural speech toward speech that follows explicit user preferences.
• Support control over:
  • Speaking speed
  • Voice quality and tone
  • Pitch and intonation
  • Volume
  • Emotional and conversational style
• Combine better speech representation with instruction-following training.

Limitations of the Original System

• The original Kanana-o represented speech with 25 discrete tokens per second.
• Its Voice Token LM generated these tokens sequentially based on text responses and conversation context.
• Speech reconstruction required two stages:
  • Token-to-Mel: Convert speech tokens into a mel-spectrogram.
  • Mel-to-Waveform: Convert the mel-spectrogram into a waveform.
• The tokenizer captured linguistic content effectively but did not explicitly represent acoustic properties such as voice timbre, pitch, intonation, speed, or emotion.
• Long token sequences increased generation latency and computational cost.
• The two-stage decoder, often involving iterative diffusion or flow-matching inference, made the pipeline difficult to optimize for real-time services.

LM-SPT: A More Efficient Speech Tokenizer

• LM-SPT stands for LM-aligned SPeech Tokenizer.
• It compresses speech to 12.5 frames per second—half the original rate—reducing the number of sequential prediction steps.
• It represents both:
  • Semantic speech tokens: The spoken content aligned with text and conversational context.
  • Acoustic speech tokens: Voice-specific details such as timbre, pitch, intonation, and speaking rate.
• This separation allows the language model to generate content and acoustic characteristics more independently and controllably.
• LM-SPT uses:
  • Two encoders for semantic and acoustic information
  • One semantic codebook
  • Multiple acoustic codebooks
  • A Split Residual Vector Quantization structure

Semantic Speech-Resynthesis Distillation

• Training only for waveform reconstruction does not guarantee that semantic and acoustic information remain separated.
• Earlier systems commonly distilled representations from self-supervised models such as HuBERT or WavLM.
• That approach can suffer from:
  • Misalignment between phonetic representations and higher-level language-model semantics
  • Loss of information when matching models with different frame rates
• LM-SPT instead uses a Semantic Speech-Resynthesis Distillation method:
  • Reconstruct speech using only semantic tokens.
  • Compare the original and reconstructed speech with a pretrained speech encoder aligned to language-model representations.
  • Train the semantic tokens to preserve the same meaning without requiring exact frame-by-frame teacher alignment.
• This approach helps retain meaningful content even at the lower 12.5 Hz token rate.

Simplified Speech Decoding

• During normal tokenization and reconstruction, the system does not require a heavy pretrained speech encoder.
• A lightweight learned encoder is sufficient.
• The final decoder uses semantic and acoustic tokens together to reconstruct the waveform directly.
• This removes the intermediate mel-spectrogram stage and replaces the previous two-stage process with a lighter single-decoder structure.
• As a result, the system reduces both language-model generation length and waveform reconstruction complexity.

Instruction Following Through Online Reinforcement Learning

• LM-SPT provides the representation needed to control acoustic features at the token level.
• Kanana-o also applies online reinforcement learning to teach the speech-generation module to follow diverse vocal instructions.
• The objective is to balance:
  • Accurate compliance with requested speaking styles
  • Natural and high-quality audio output

Kakao’s approach combines a lower-rate, semantically and acoustically structured tokenizer with reinforcement learning for instruction adherence. The result is intended to make Kanana-o faster and more suitable for real-time use while allowing users to specify not only what the system says, but how it says it.