Inference Engine | Techlist.io

awsApr 16, 20263 min readCurated summary

Introducing Anthropic’s Claude Opus 4.7 model in Amazon Bedrock | Amazon Web Services

Claude Opus 4.7 is now available in Amazon Bedrock as Anthropic’s most capable Opus model for coding, professional knowledge work, long-running agents, and visual tasks. It runs on Bedrock’s new inference engine, which dynamically schedules capacity, improves availability, queues requests during demand spikes, and provides zero operator access to prompts and responses. AWS positions the model as a production-ready upgrade to Opus 4.6, though teams may need to adjust prompts and agent harnesses. ## Model Improvements - **Agentic coding:** Stronger long-horizon autonomy, systems engineering, and complex code reasoning. - 64.3% on SWE-bench Pro - 87.6% on SWE-bench Verified - 69.4% on Terminal-Bench 2.0 - **Knowledge work:** Better document creation, financial analysis, and multi-step research. - Handles underspecified requests by making and explaining reasonable assumptions. - Self-verifies responses to improve first-pass quality. - 64.4% on Finance Agent v1.1. - **Long-running tasks:** Maintains coherence across its full 1-million-token context window while handling ambiguity and verifying results. - **Vision:** High-resolution image support improves analysis of charts, dense documents, and detailed screen interfaces. - **Adaptive thinking:** Dynamically allocates reasoning-token budgets based on request complexity. ## Bedrock Infrastructure and Scaling - The new inference engine uses updated scheduling and scaling logic to allocate capacity dynamically. - It prioritizes steady-state workloads while rapidly scaling capacity for changing demand. - During high demand, requests are queued instead of rejected. - Up to 10,000 requests per minute per account and Region are available immediately, with additional capacity available by request. - Zero operator access prevents AWS and Anthropic operators from viewing customer prompts and responses. ## Ways to Invoke the Model - Test Claude Opus 4.7 in the Amazon Bedrock console through **Playground**. - Use the Anthropic Messages API through: - Anthropic SDK integrations with Bedrock - `bedrock-mantle` endpoints - Use Bedrock-native APIs: - **Converse API** for multi-turn conversations and Guardrails integration - **Invoke API** for direct invocation and lower-level control - Invoke it with the AWS CLI using model ID `anthropic.claude-opus-4-7`. - The Anthropic Bedrock SDK can authenticate with AWS Signature Version 4 and submit Messages API requests programmatically. ## Availability - Initially available in: - US East (N. Virginia) - Asia Pacific (Tokyo) - Europe (Ireland) - Europe (Stockholm) - AWS directs users to the Bedrock model and pricing pages for regional availability updates. Teams seeking stronger coding agents, complex research workflows, or high-volume production inference can try Claude Opus 4.7 in Bedrock, while validating prompt and harness changes before migrating existing Opus 4.6 workloads.

Read original

Inference Engine Python Amazon Bedrock API+4

cloudflareMar 19, 20263 min readCurated summary

Powering the agents: Workers AI now runs large models, starting with Kimi K2.5

Cloudflare is expanding Workers AI beyond smaller models by adding Moonshot AI’s Kimi K2.5, a frontier open-source model designed for agentic workloads. With a 256k context window, tool calling, vision, and structured outputs, Kimi can power an agent’s full lifecycle directly on Cloudflare’s platform. Cloudflare argues that its price-performance makes open-source models essential as personal and enterprise agents dramatically increase inference demand. ## Kimi K2.5’s Price-Performance Advantage - Cloudflare uses Kimi internally for: - Agentic coding through OpenCode - Automated code review via the Bonk public code review agent - Security analysis of Cloudflare codebases - A security-review agent processes more than 7 billion tokens daily and has found over 15 confirmed issues in one codebase. - Compared with a mid-tier proprietary model, switching to Kimi reduced the estimated cost of this workload by 77%, from roughly $2.4 million annually. - As employees increasingly run multiple agents continuously, inference costs become a major barrier to scaling. - Cloudflare positions open-source, frontier-quality models as a more economical alternative to proprietary systems. ## Serving Large Models on Workers AI - Supporting Kimi required upgrades to Workers AI’s inference stack, which historically focused on smaller models. - Cloudflare uses its proprietary Infire inference engine and custom kernels to improve: - Model performance - GPU utilization - Throughput - The platform applies advanced serving strategies such as: - Data, tensor, and expert parallelization - Disaggregated prefill, separating input processing from generation across machines - Workers AI handles these infrastructure optimizations so developers do not need specialized machine learning, DevOps, or reliability engineering expertise. ## Prefix Caching for Agent Workloads - Agents frequently resend large prompts containing: - System instructions - Tool definitions - MCP server tools - Conversation history - Entire codebases - Prefix caching avoids reprocessing unchanged input tokens during multi-turn interactions. - This reduces prefill work, improving: - Time to First Token (TTFT) - Tokens Per Second (TPS) - Overall inference cost - Workers AI now exposes cached tokens as a usage metric and charges less for them than regular input tokens. - Cloudflare has also introduced techniques to improve cache hit rates. ## Session Affinity - Workers AI provides an `x-session-affinity` header to route requests from the same session or agent to the same model instance. - Keeping requests on the same instance increases prefix-cache reuse. - Higher cache hit rates lead to faster responses, greater throughput, and lower costs. - Clients should provide a unique session or agent identifier with the header. Cloudflare’s recommendation is to use Workers AI when building agents that need frontier-level reasoning without the cost and operational burden of proprietary models or self-hosted infrastructure.

Read original

Inference Engine Database Design Large Language Models Kubernetes+4