model-routing

Unifying Workers AI and AI Gateway into a single AI control plane (opens in new tab)

AI Gateway and Workers AI are converging into a unified control plane for accessing models across Cloudflare and external providers. A single Workers binding or REST API can now provide inference, observability, logging, security, and billing without requiring users to choose a product upfront. Cloudflare’s longer-term goal is model-first routing, where applications request capabilities or models while the gateway handles provider selection, failover, and load balancing. ## Unified Bindings and API - The Workers AI binding and AI Gateway now share the same entrypoint. - Requests can use the built-in `default` gateway or a named gateway for separate applications and customized policies. - The unified REST API routes requests through `/ai/` endpoints, using the `cf-aig-gateway-id` header. - This removes the need to decide between Workers AI and AI Gateway before building an application. ## Automatic Observability for Workers AI - Passing `default` as the gateway ID automatically creates an AI Gateway on the first authenticated request. - Requests receive built-in: - Full request and response logging - Token tracking by model - Cost attribution - Latency and error metrics - Developers can begin with the default gateway and later switch to a named gateway for features such as custom caching or application-specific traffic separation. - The AI Gateway dashboard provides detailed visibility into prompts, responses, latency, token usage, and failures. ## Unified Billing with AI Gateway Credits - AI Gateway credits can now pay for Workers AI usage in addition to providers such as OpenAI and Anthropic. - Users can maintain one prepaid credit balance across supported providers. - Workers AI users who use unified billing receive elevated rate limits, subject to current Cloudflare policies and documentation. ## Model-First Routing - Cloudflare plans to route requests based on the desired model rather than requiring users to select a specific provider. - The gateway could handle: - Provider selection - Failover - Load balancing - Capacity management - For example, a request for a model such as Kimi K2.7 Code could be served by Workers AI, the model’s original provider, or another vetted provider hosting the same weights. - Applications could remain available if one provider is overloaded or unavailable. - Users will still be able to restrict traffic to a single provider when necessary. - Routing is intended to preserve requirements such as Zero Data Retention and maintain model quality. Cloudflare recommends using the unified binding or REST API with the default gateway to gain observability and centralized billing immediately. As model-first routing develops, applications can rely less on provider-specific infrastructure and gain greater resilience through automatic provider management.

Getting more from each token: How Copilot improves context handling and model routing (opens in new tab)

GitHub is improving Copilot’s efficiency by optimizing both how context is handled and which model performs each task. Prompt caching and deferred tool loading reduce repeated overhead in long agentic sessions, while Auto selects models based on task complexity and real-time system conditions. The goal is to preserve quality while using tokens, compute, and cost more intelligently. ## Prompt Caching and Deferred Tools - Copilot sessions repeatedly include instructions, repository context, conversation history, tool definitions, and task state. - Prompt caching allows repeated prompt prefixes to reuse model state instead of being recomputed on every request. - Tool search loads tool definitions only when needed, rather than placing every full schema into context on every turn. - This is especially valuable for sessions involving MCP tools, terminal commands, file operations, workspace search, and other actions. - The approach reduces fixed per-turn overhead while preserving access to a broad toolset. ## Auto Model Selection - Copilot Auto chooses a model based on the task rather than requiring developers to select one manually. - Quick explanations, focused edits, and complex multi-file changes often benefit from different levels of reasoning. - Evaluations found that no single model consistently performs best across all tasks. - Auto uses more efficient models when they can achieve comparable results and selects stronger reasoning models when the task demands them. - The objective is not simply lower cost, but matching model capability to the work. ## How Auto Routes Tasks Auto combines two signals: - **Real-time model health** - Considers availability, utilization, response speed, error rates, and cost. - Avoids selecting a model that is technically capable but currently overloaded or unreliable. - **Task-aware routing with HyDRA** - Evaluates reasoning depth, code complexity, debugging difficulty, and tool-orchestration requirements. - Identifies models capable of meeting the task’s quality threshold, then chooses the best fit among them. - The system can be tuned toward maximum quality or greater cost savings. ## Cache-Aware Routing - Switching models during every turn can reduce efficiency by invalidating cached prompt prefixes. - Auto therefore keeps the same model during a conversation so the cache can continue building. - It changes models at natural cache boundaries: - The first turn, when no cache exists yet. - After context compaction, when older turns are summarized and the prompt prefix resets. ## Multilingual Routing - Copilot’s routing system was trained on conversations spanning 16 language families, including CJK and European languages. - Evaluations covered production VS Code chat data across 19 languages. - Routing accuracy remained within four points of the English baseline across language groups. - No statistically significant quality gap was observed between the language groups. Copilot’s efficiency strategy combines smarter context management with adaptive model selection. Caching and on-demand tools reduce repeated work, while cache-aware Auto routing chooses an appropriate model without unnecessarily sacrificing quality or session efficiency.