Smaller, faster, safer: running Kimi and GLM at scale (opens in new tab)
Cloudflare improves the efficiency of serving large, long-context models by optimizing both GPU memory usage and request concurrency. Its approach combines FP8 KV-cache quantization, INT4 weight compression, and integrity checks for shared caches, while using separate prefill and decode pools to apply each optimization where it helps most. These techniques increase throughput and reduce costs without materially affecting model accuracy. ## Quantizing the KV Cache - Long-context models store attention keys and values in a KV cache, which often exhausts GPU memory before model weights do. - Cloudflare stores the cache in FP8 instead of BF16, cutting its size in half. - On Kimi K2.6, this increases available context from roughly 686,000 to 1.37 million tokens. - FP8 is slightly slower at the same concurrency because values must be converted during attention. - However, FP8 supports more concurrent requests: - BF16 runs out of memory at 32 requests. - FP8 reaches 2,192 tokens per second at 64 requests. - Peak throughput improves by about 41%, with roughly 30% lower cost per token. - Cloudflare keeps BF16 for prefill, where workloads are compute-bound. - Evaluation results show FP8 and BF16 produce effectively indistinguishable quality across reasoning, knowledge, tool-calling, and internal benchmarks. ## Compressing Model Weights - GLM 5.2 weights are compressed from FP8 to INT4 for the decode phase. - The checkpoint shrinks from 705 GB to 421 GB, while per-GPU memory in an eight-way deployment falls from about 88 GB to 52 GB. - The freed memory supports approximately 1.18 million tokens of KV cache. - INT4 improves decode performance because generation is memory-bandwidth-bound: - Single-request throughput rises from 60 to 92 tokens per second, a 55% gain. - Gains range from 16% to 27% at higher concurrency. - Prefill becomes slower with INT4 because compressed weights must be expanded before computation: - FP8 prefill: about 10,160 tokens per second. - INT4 prefill: about 8,660 tokens per second. - Cloudflare therefore uses FP8 for prefill and INT4 for decode. - Accuracy remains within 0.8 percentage points of the FP8 model across tested benchmarks. ## Protecting a Shared KV Cache - Greater memory efficiency allows hundreds of requests to share physical KV-cache pages, increasing the risk of page-allocation or bookkeeping errors. - Cloudflare assigns each cache page a changing tag whenever it is reallocated. - Requests record the pages and tags they expect, and the server validates these mappings before supported decode operations. - If a mismatch occurs, the request is aborted instead of reading incorrect data. - In production-style tests, integrity checking caused: - Less than 1% throughput reduction. - Less than 1% increase in p95 latency. - Validation runs as a separate batch check rather than inside the attention kernel, avoiding GPU synchronization races. - The feature is enabled per deployment, while deployments that do not use it incur no measurable overhead. ## Future Work - Cloudflare is expanding FP8 KV caches across its fleet. - It is testing NVFP4 weight compression on NVIDIA Blackwell GPUs. - The company is also working toward making cache integrity checks universally enabled at negligible cost. Together, these optimizations let Cloudflare serve larger models with more concurrent users, lower inference costs, and essentially unchanged model quality. Separating prefill and decode workloads is central to applying each precision choice where it delivers the best trade-off.
