cloudflareAug 3, 2026

Smaller, faster, safer: running Kimi and GLM at scale (opens in new tab)

Cloudflare improves the efficiency of serving large, long-context models by optimizing both GPU memory usage and request concurrency. Its approach combines FP8 KV-cache quantization, INT4 weight compression, and integrity checks for shared caches, while using separate prefill and decode pools to apply each optimization where it helps most. These techniques increase throughput and reduce costs without materially affecting model accuracy. ## Quantizing the KV Cache - Long-context models store attention keys and values in a KV cache, which often exhausts GPU memory before model weights do. - Cloudflare stores the cache in FP8 instead of BF16, cutting its size in half. - On Kimi K2.6, this increases available context from roughly 686,000 to 1.37 million tokens. - FP8 is slightly slower at the same concurrency because values must be converted during attention. - However, FP8 supports more concurrent requests: - BF16 runs out of memory at 32 requests. - FP8 reaches 2,192 tokens per second at 64 requests. - Peak throughput improves by about 41%, with roughly 30% lower cost per token. - Cloudflare keeps BF16 for prefill, where workloads are compute-bound. - Evaluation results show FP8 and BF16 produce effectively indistinguishable quality across reasoning, knowledge, tool-calling, and internal benchmarks. ## Compressing Model Weights - GLM 5.2 weights are compressed from FP8 to INT4 for the decode phase. - The checkpoint shrinks from 705 GB to 421 GB, while per-GPU memory in an eight-way deployment falls from about 88 GB to 52 GB. - The freed memory supports approximately 1.18 million tokens of KV cache. - INT4 improves decode performance because generation is memory-bandwidth-bound: - Single-request throughput rises from 60 to 92 tokens per second, a 55% gain. - Gains range from 16% to 27% at higher concurrency. - Prefill becomes slower with INT4 because compressed weights must be expanded before computation: - FP8 prefill: about 10,160 tokens per second. - INT4 prefill: about 8,660 tokens per second. - Cloudflare therefore uses FP8 for prefill and INT4 for decode. - Accuracy remains within 0.8 percentage points of the FP8 model across tested benchmarks. ## Protecting a Shared KV Cache - Greater memory efficiency allows hundreds of requests to share physical KV-cache pages, increasing the risk of page-allocation or bookkeeping errors. - Cloudflare assigns each cache page a changing tag whenever it is reallocated. - Requests record the pages and tags they expect, and the server validates these mappings before supported decode operations. - If a mismatch occurs, the request is aborted instead of reading incorrect data. - In production-style tests, integrity checking caused: - Less than 1% throughput reduction. - Less than 1% increase in p95 latency. - Validation runs as a separate batch check rather than inside the attention kernel, avoiding GPU synchronization races. - The feature is enabled per deployment, while deployments that do not use it incur no measurable overhead. ## Future Work - Cloudflare is expanding FP8 KV caches across its fleet. - It is testing NVFP4 weight compression on NVIDIA Blackwell GPUs. - The company is also working toward making cache integrity checks universally enabled at negligible cost. Together, these optimizations let Cloudflare serve larger models with more concurrent users, lower inference costs, and essentially unchanged model quality. Separating prefill and decode workloads is central to applying each precision choice where it delivers the best trade-off.

continuous-batchinggpu-inferencemixture-of-expertskv-cache+4

kakaoMay 5, 2026

Bringing a Voice AI Model to Production: The Journey of Optimizing Kanana-O Serving (opens in new tab)

Kanana-O is a multimodal model that understands text, images, and audio, then responds with text and speech. Deploying it for real-time voice conversations required solving problems that do not arise during model training, including low first-response latency, concurrent users, streaming across multiple models, and uneven GPU memory demands. Kakao built the specialized Kanana-Omni Server, achieving 1.6× the throughput of vLLM-Omni at 64 concurrent users. ## Kanana-O’s Three-Stage Architecture - **Thinker** processes multimodal inputs and generates text. - **Talker** converts Thinker’s text embeddings into sequential speech tokens. - **VoiceBox** combines speech tokens into audible audio waveforms. - In production, these components must operate concurrently rather than sequentially to deliver audio within hundreds of milliseconds. ## Why a Specialized Serving Server Was Needed - Thinker passes hidden-state embeddings directly to Talker rather than ordinary token IDs. - These high-dimensional tensors must be transferred continuously, making serialization or CPU copies too expensive. - Talker produces speech tokens step by step, while VoiceBox waits for enough tokens to form larger audio chunks. - Talker also combines speaker embeddings, Thinker outputs, and its own accumulated audio embeddings, creating an input structure unlike standard autoregressive decoding. - These constraints made a custom server more suitable than general-purpose frameworks. ## Zero-Copy Data Transfer - The server preallocates shared-memory blocks during startup. - Thinker writes tensors into an available block, while Talker receives only metadata such as the block identifier and byte size. - This avoids repeated allocation, copying, and serialization. - For GPU tensors on the same node, CUDA IPC transfers data directly between GPU processes, avoiding Device→Host→Device movement. ## Cascaded Streaming Pipeline - Thinker, Talker, and VoiceBox run as overlapping asynchronous stages. - Thinker can send its first output chunk while Talker processes earlier chunks and VoiceBox synthesizes audio from still earlier ones. - Talker buffers speech tokens until VoiceBox has enough data to create an audio chunk. - This pipelining significantly reduces the time before the user hears the first response. ## Process Isolation and Fault Containment - Thinker and Talker each run their own vLLM engine in separate processes. - This avoids conflicts between CUDA contexts, model memory, KV caches, and schedulers. - Processes are started with `spawn` rather than `fork`, preventing inherited CUDA state from causing corruption. - If one component fails, such as Thinker running out of memory, the other components and the API server can continue operating and be restarted independently. ## Continuous Batching with vLLM - Manually batching requests is difficult because multimodal inputs and accumulated Talker embeddings vary in size. - The server submits requests rapidly and delegates batch construction to vLLM’s continuous-batching scheduler. - Each request runs as an independent asynchronous generation task. - vLLM combines requests internally during forward passes, while request IDs ensure each task receives only its own streamed output. - This improves GPU utilization without requiring custom synchronization and padding logic. ## Single FastAPI Worker and Asynchronous Execution - Multiple Uvicorn workers would load separate copies of the vLLM engines, multiplying GPU memory usage and model-loading costs. - Therefore, the server uses `workers=1`. - Since a blocking operation would otherwise stall every connected user, the entire request path—from the API endpoint through final audio generation—is designed around `async`/`await`. - Keeping the pipeline non-blocking allows one worker to accept and progress many concurrent requests. Kakao’s main recommendation is to design serving infrastructure around the model’s actual dataflow rather than forcing it into a generic framework. For complex multimodal pipelines, zero-copy transfers, asynchronous cascaded streaming, process isolation, and engine-level continuous batching can be more important than simply scaling API workers.

continuous-batchingmultimodal-aivllmfastapi+3