Building the foundation for running extra-large language models (opens in new tab)

Cloudflare is building infrastructure for serving extra-large open-source language models efficiently, especially for agentic applications with long prompts and frequent tool calls. Its approach combines specialized hardware configurations, prefill/decode disaggregation, prompt caching, distributed KV-cache management, and speculative decoding. These optimizations substantially improve latency, throughput, and cost efficiency without requiring more GPUs.

Hardware Configurations for Agent Workloads

• Different applications stress models differently:
  • Content generation sends fewer input tokens but produces many output tokens.
  • Summarization sends very large inputs and generates relatively short outputs.
• Agent workloads typically involve:
  • Large system prompts
  • Tool and MCP definitions
  • Accumulated conversation history
  • Generated code and previous interactions
• Workers AI therefore prioritizes fast input processing and tool-calling performance.

Prefill-Decode Disaggregation

• LLM inference has two stages:
  • Prefill: Processes input tokens and populates the KV cache; generally compute-bound.
  • Decode: Generates output tokens; generally memory-bound.
• Running both stages on one server can underutilize GPUs because they stress different resources.
• Cloudflare separates them across dedicated inference servers:
  • A prefill server processes the request and stores its KV cache.
  • A decode server retrieves the cache and generates the response.
• This enables independent tuning, scaling for input- or output-heavy traffic, and use of heterogeneous hardware.
• The architecture requires a sophisticated load balancer that:
  • Transfers KV-cache metadata between stages.
  • Rewrites streaming SSE responses.
  • Handles different inference-server protocols.
  • Balances traffic based on estimated in-flight prefill and decode tokens.
• After adopting this design, Cloudflare saw:
  • Lower p90 time to first token and reduced tail-latency variance.
  • Intertoken latency fall from roughly 100 ms to 20–30 ms.
  • About a threefold improvement while using the same number of GPUs.

Prompt Caching and Session Affinity

• Long agent conversations repeatedly reuse the same context, making prompt caching essential.
• The x-session-affinity header routes requests toward regions containing previously computed input tensors.
• Cloudflare added support for this header to agent harnesses such as OpenCode.
• Cached prompts improve:
  • Overall throughput
  • Interactive response times
  • Pricing, with discounted cached tokens
  • GPU efficiency
• Adoption by heavy internal users increased peak input-token cache hit rates from 60% to 80%.

Distributed KV-Cache Optimization

• Larger models span multiple GPUs, requiring KV caches to be shared across devices and nodes.
• For Kimi, Cloudflare uses Moonshot AI’s:
  • Mooncake Transfer Engine for high-speed memory transfers using RDMA technologies such as NVLink and NVMe over Fabric.
  • Mooncake Store to extend cache storage beyond GPU VRAM onto NVMe.
• Combined with LMCache or SGLang HiCache, the system can:
  • Reuse cached prompts from any node in a cluster.
  • Reduce reliance on session-aware routing.
  • Balance traffic more evenly.
  • Keep sessions cached longer.
  • Increase cache hit rates and supported throughput.

Speculative Decoding

• The post begins introducing speculative decoding as another optimization.
• It describes the basic LLM process of predicting successive tokens, but the provided text ends before explaining the technique or its results.

Cloudflare’s overall strategy is to match infrastructure to real usage patterns rather than rely on a single hardware configuration. Separating inference stages, maximizing cache reuse, and distributing KV caches are practical ways to make large-model hosting faster and more economical.