From AI That Speaks Well to AI That Speaks Exactly as Desired: Advancing Kanana-o Voice Generation
Kanana-o’s latest speech-generation improvements target two goals: faster, more efficient synthesis and more precise adherence to user instructions. Kakao addresses these through LM-SPT, a speech tokenizer that separates semantic and acoustic information while reducing the token rate from 25 Hz to 12.5 Hz, and through online reinforcement learning. Together, these changes enable Kanana-o to generate natural speech more efficiently and control characteristics such as speed, pitch, tone, and volume more reliably. ## Goals for Kanana-o’s Speech Generation - Improve real-time performance by shortening speech-token sequences and simplifying decoding. - Move beyond merely natural speech toward speech that follows explicit user preferences. - Support control over: - Speaking speed - Voice quality and tone - Pitch and intonation - Volume - Emotional and conversational style - Combine better speech representation with instruction-following training. ## Limitations of the Original System - The original Kanana-o represented speech with 25 discrete tokens per second. - Its Voice Token LM generated these tokens sequentially based on text responses and conversation context. - Speech reconstruction required two stages: - **Token-to-Mel:** Convert speech tokens into a mel-spectrogram. - **Mel-to-Waveform:** Convert the mel-spectrogram into a waveform. - The tokenizer captured linguistic content effectively but did not explicitly represent acoustic properties such as voice timbre, pitch, intonation, speed, or emotion. - Long token sequences increased generation latency and computational cost. - The two-stage decoder, often involving iterative diffusion or flow-matching inference, made the pipeline difficult to optimize for real-time services. ## LM-SPT: A More Efficient Speech Tokenizer - LM-SPT stands for **LM-aligned SPeech Tokenizer**. - It compresses speech to 12.5 frames per second—half the original rate—reducing the number of sequential prediction steps. - It represents both: - **Semantic speech tokens:** The spoken content aligned with text and conversational context. - **Acoustic speech tokens:** Voice-specific details such as timbre, pitch, intonation, and speaking rate. - This separation allows the language model to generate content and acoustic characteristics more independently and controllably. - LM-SPT uses: - Two encoders for semantic and acoustic information - One semantic codebook - Multiple acoustic codebooks - A Split Residual Vector Quantization structure ## Semantic Speech-Resynthesis Distillation - Training only for waveform reconstruction does not guarantee that semantic and acoustic information remain separated. - Earlier systems commonly distilled representations from self-supervised models such as HuBERT or WavLM. - That approach can suffer from: - Misalignment between phonetic representations and higher-level language-model semantics - Loss of information when matching models with different frame rates - LM-SPT instead uses a **Semantic Speech-Resynthesis Distillation** method: - Reconstruct speech using only semantic tokens. - Compare the original and reconstructed speech with a pretrained speech encoder aligned to language-model representations. - Train the semantic tokens to preserve the same meaning without requiring exact frame-by-frame teacher alignment. - This approach helps retain meaningful content even at the lower 12.5 Hz token rate. ## Simplified Speech Decoding - During normal tokenization and reconstruction, the system does not require a heavy pretrained speech encoder. - A lightweight learned encoder is sufficient. - The final decoder uses semantic and acoustic tokens together to reconstruct the waveform directly. - This removes the intermediate mel-spectrogram stage and replaces the previous two-stage process with a lighter single-decoder structure. - As a result, the system reduces both language-model generation length and waveform reconstruction complexity. ## Instruction Following Through Online Reinforcement Learning - LM-SPT provides the representation needed to control acoustic features at the token level. - Kanana-o also applies online reinforcement learning to teach the speech-generation module to follow diverse vocal instructions. - The objective is to balance: - Accurate compliance with requested speaking styles - Natural and high-quality audio output Kakao’s approach combines a lower-rate, semantically and acoustically structured tokenizer with reinforcement learning for instruction adherence. The result is intended to make Kanana-o faster and more suitable for real-time use while allowing users to specify not only what the system says, but how it says it.
Read original(opens in new tab)