The provided content does not include the blog post’s body. It only contains the title, author links, navigation, and search interface, so the article’s technical argument and conclusions cannot be summarized reliably.
## Available Information
- **Title:** “Beyond AI That Speaks Well: Making Kanana-o Speak the Way Users Want”
- **Topic indicated by the title:** Improving Kanana-o’s voice-generation capabilities to produce speech according to user preferences.
- **Authors:** martin.gale, abigail.r, and edwin.ai
- **Missing:** The article’s main sections, implementation details, experiments, and conclusions.
Please provide the full article text or its URL content for a detailed summary.
Kanana-o’s latest speech-generation improvements target two goals: faster, more efficient synthesis and more precise adherence to user instructions. Kakao addresses these through LM-SPT, a speech tokenizer that separates semantic and acoustic information while reducing the token rate from 25 Hz to 12.5 Hz, and through online reinforcement learning. Together, these changes enable Kanana-o to generate natural speech more efficiently and control characteristics such as speed, pitch, tone, and volume more reliably.
## Goals for Kanana-o’s Speech Generation
- Improve real-time performance by shortening speech-token sequences and simplifying decoding.
- Move beyond merely natural speech toward speech that follows explicit user preferences.
- Support control over:
- Speaking speed
- Voice quality and tone
- Pitch and intonation
- Volume
- Emotional and conversational style
- Combine better speech representation with instruction-following training.
## Limitations of the Original System
- The original Kanana-o represented speech with 25 discrete tokens per second.
- Its Voice Token LM generated these tokens sequentially based on text responses and conversation context.
- Speech reconstruction required two stages:
- **Token-to-Mel:** Convert speech tokens into a mel-spectrogram.
- **Mel-to-Waveform:** Convert the mel-spectrogram into a waveform.
- The tokenizer captured linguistic content effectively but did not explicitly represent acoustic properties such as voice timbre, pitch, intonation, speed, or emotion.
- Long token sequences increased generation latency and computational cost.
- The two-stage decoder, often involving iterative diffusion or flow-matching inference, made the pipeline difficult to optimize for real-time services.
## LM-SPT: A More Efficient Speech Tokenizer
- LM-SPT stands for **LM-aligned SPeech Tokenizer**.
- It compresses speech to 12.5 frames per second—half the original rate—reducing the number of sequential prediction steps.
- It represents both:
- **Semantic speech tokens:** The spoken content aligned with text and conversational context.
- **Acoustic speech tokens:** Voice-specific details such as timbre, pitch, intonation, and speaking rate.
- This separation allows the language model to generate content and acoustic characteristics more independently and controllably.
- LM-SPT uses:
- Two encoders for semantic and acoustic information
- One semantic codebook
- Multiple acoustic codebooks
- A Split Residual Vector Quantization structure
## Semantic Speech-Resynthesis Distillation
- Training only for waveform reconstruction does not guarantee that semantic and acoustic information remain separated.
- Earlier systems commonly distilled representations from self-supervised models such as HuBERT or WavLM.
- That approach can suffer from:
- Misalignment between phonetic representations and higher-level language-model semantics
- Loss of information when matching models with different frame rates
- LM-SPT instead uses a **Semantic Speech-Resynthesis Distillation** method:
- Reconstruct speech using only semantic tokens.
- Compare the original and reconstructed speech with a pretrained speech encoder aligned to language-model representations.
- Train the semantic tokens to preserve the same meaning without requiring exact frame-by-frame teacher alignment.
- This approach helps retain meaningful content even at the lower 12.5 Hz token rate.
## Simplified Speech Decoding
- During normal tokenization and reconstruction, the system does not require a heavy pretrained speech encoder.
- A lightweight learned encoder is sufficient.
- The final decoder uses semantic and acoustic tokens together to reconstruct the waveform directly.
- This removes the intermediate mel-spectrogram stage and replaces the previous two-stage process with a lighter single-decoder structure.
- As a result, the system reduces both language-model generation length and waveform reconstruction complexity.
## Instruction Following Through Online Reinforcement Learning
- LM-SPT provides the representation needed to control acoustic features at the token level.
- Kanana-o also applies online reinforcement learning to teach the speech-generation module to follow diverse vocal instructions.
- The objective is to balance:
- Accurate compliance with requested speaking styles
- Natural and high-quality audio output
Kakao’s approach combines a lower-rate, semantically and acoustically structured tokenizer with reinforcement learning for instruction adherence. The result is intended to make Kanana-o faster and more suitable for real-time use while allowing users to specify not only what the system says, but how it says it.
Kakao has significantly advanced its integrated multimodal model, Kanana-o, by enhancing its ability to process complex instructions across text, image, and audio inputs while enriching its emotional vocal expression. By developing specialized datasets and sophisticated training techniques for prosody, the team has bridged the performance gap between text and audio modalities. The result is a more natural, human-like AI capable of nuanced interaction and high-performance instruction following, particularly within the Korean linguistic context.
## Advancing Multimodal Instruction Following
* Addressed the "modality gap" where multimodal models often show decreased reasoning and reasoning performance when processing audio inputs compared to text.
* Constructed a structured, high-quality dataset featuring complex, multi-step instructions such as summarizing a context and then translating it into a specific language or style.
* Leveraged the Speech-KoMT-Bench to evaluate performance, showing that Kanana-o significantly outperforms global competitors of similar scale in Korean-specific tasks.
* Focused on "Domain-generalization" to ensure the model's core intelligence remains stable regardless of whether the input is text, audio, or a combination of both.
## Image-Audio-Text Modality Alignment
* Developed integrated datasets to ensure that reasoning capabilities learned in text-image or text-audio contexts generalize to complex image-audio scenarios.
* Trained the model to handle tasks where users ask questions about visual information via voice, requiring the simultaneous alignment of three different data types.
* Prioritized the maintenance of "World Knowledge" during multimodal training so that the addition of new modalities does not degrade the model’s factual accuracy.
## Enhancing Vocal Expressiveness and Prosody
* Focused on "prosody"—the rhythm, pitch, and stress of speech—to move beyond robotic, flat text-to-speech (TTS) outputs.
* Implemented a system of descriptive tokens and emotion tags (e.g., "warm voice," "excited tone") during training to give the model fine-grained control over its vocal persona.
* Incorporated natural human speech elements, such as realistic breathing patterns and contextual variations in speech speed, to make interactions feel more intuitive and less synthetic.
* Refined the model's ability to interpret the user's emotional state from their voice and respond with a matching emotional intensity.
The evolution of Kanana-o highlights a shift from simply maximizing generic benchmarks to optimizing real-world user experiences through multimodal alignment and emotional intelligence. The success of this model underscores the necessity of high-quality, structured instruction data and fine-grained control over output styles to create truly conversational AI that feels natural to the user.
