cloudflareAug 4, 2026

How Cloudflare enforces engineering standards using AI (opens in new tab)

Cloudflare built the Codex to turn scattered engineering knowledge into governed, machine-readable standards that both engineers and AI agents can apply consistently. It now supports code reviews, technical design reviews, and incident reviews, with AI systems flagging nearly 230,000 violations and blocking about 16,000 merges. The central approach is to combine human-owned RFCs with structured extraction, staged enforcement, and context-aware agents. ## Why Cloudflare Built the Codex - Engineering guidance previously existed across formal documentation, repositories, chat, and individual experience. - Engineers struggled to determine whether guidance was current, authoritative, or relevant. - Growth made it difficult for anyone to know every standard or for reviewers to check every requirement. - The Codex provides a shared source of truth that can be retrieved and applied at the point of work. ## Governance and RFC Workflow - The Codex is divided into domains such as: - Architecture and control plane systems - Security and reliability - Programming languages including TypeScript and Rust - Each domain has an owner responsible for content quality and consistency. - Standards follow an RFC format using RFC 2119 terminology: - **SHOULD** for recommendations - **MUST** for mandatory requirements - Employees can propose RFCs through structured merge requests. - Proposals undergo increasingly broad review before domain-owner approval. - Approved RFCs are published to an internal Astro-powered site. - Enforcement is deliberately separated from approval: - Approved standards can generate findings. - Only enforced standards can block merges. - This gives teams time to adopt requirements and implement enforcement mechanisms. ## Structured Standards for Agents - Feeding all 60-plus RFCs directly into an LLM would consume too much context and reduce accuracy. - A dedicated agent extracts SHOULD and MUST statements into structured JSON. - Each statement includes: - A stable slug - RFC and domain metadata - Requirement level - Section and source link - Stable identifiers allow Cloudflare to track requirements across RFC revisions, systems, monitoring, and exception handling. - Cloudflare moved from concise Markdown extraction to JSON to enable filtering and progressive disclosure. - Future metadata may identify which SDLC stage applies, such as design, implementation, or runtime. ## AI Code Review - The AI code reviewer retrieves relevant statements first and loads complete RFCs only when more context is needed. - Approved-RFC findings are non-blocking recommendations. - Violations of MUST requirements in enforced RFCs can withhold approval or block a merge. - Since launch, the reviewer has: - Flagged nearly 230,000 violations - Withheld approval for almost 16,000 violations ## Faster Code Review Alternatives - Full AI reviews generally take several minutes because they use coordinators and multiple agents. - To reduce remediation delays, Cloudflare is also developing mechanically verifiable checks. - Language-specific Codex requirements can be distributed through custom linter configuration packages. - TypeScript was the first language to receive Codex linter support, alongside standardization on oxlint. The Codex’s practical value comes from connecting governed human standards to automated enforcement. Cloudflare’s staged RFC lifecycle, stable statement identifiers, and combination of AI review with fast linters provide a scalable way to preserve engineering knowledge while reducing review inconsistency.

openapillmtypescriptrust+4

tossJul 30, 2026

Building Toss’s Device Farm (opens in new tab)

Nebula is Toss’s centralized device farm, turning real-device testing into a simple API call instead of a team-specific infrastructure project. It grew from 15 devices and one developer into a 24/7 platform with more than 100 devices, shared across the company. The team replaced Appium with a faster, stateless custom driver and invested heavily in real-time streaming, reliability, security, and compliance. ## From Team-Owned Farms to a Central Platform - Before Nebula, teams managed their own small farms using MacBooks or Mac minis. - Each team repeatedly solved the same problems: - Appium setup and maintenance - Device detection and recovery - OS-version compatibility - USB and hardware failures - Security and compliance - Most teams could manage only five to ten devices, and resources remained isolated. - Nebula centralized device operations so product teams could focus on testing rather than infrastructure. ## One API for Real Devices Nebula’s core goal is to let anyone control a physical device from anywhere through a single API. - Clients reserve a device and invoke actions such as clicks or text input. - Users do not need to know which host owns the device or configure ADB, Xcode, cables, or test servers. - The same API supports frontend tools, SDKs, CLIs, direct API clients, and automated systems. ## Four-Layer Architecture - **Clients:** Web interfaces, SDKs, CLIs, and direct API calls. - **Server:** Orchestrates device discovery, allocation, and test execution. - Kafka distributes execution requests. - Multiple runners consume work horizontally as demand grows. - `occupy`, `assign`, and `release` provide distributed locking so tests cannot interfere with one another. - **Agents:** Run on Android/Linux and iOS/Mac hosts, discover local devices, and forward server requests. - **Devices:** Each device has a controller server and runner that execute actions on the physical phone. ## Why Nebula Replaced Appium ### Faster execution - Nebula’s click and input operations were more than ten times faster than Appium in common cases. - Much of Appium’s latency comes from `waitForIdle`, which waits for the screen to stabilize before acting. - Disabling that wait narrows the difference to roughly two or three times. - Appium prioritizes robustness against changing screens, while Nebula prioritizes immediate interaction for live remote control. ### Stateless operation - Appium requires sessions that can take 15–40 seconds to initialize. - Session startup becomes increasingly fragile and difficult to manage at scale. - Nebula keeps device controllers pre-warmed and accepts stateless HTTP requests, eliminating session setup and reducing failure points. ### Company-specific customization Because Nebula owns its driver specification, it can provide: - A custom IME that preserves Korean and emoji input. - Toss-specific signal triggers. - App Center integration for installing pre-release builds. - Built-in enforcement of internal security policies. The driver uses Android ADB and UiAutomation, and Swift/XCTest on iOS. Its OpenAPI specification generates Go and TypeScript interfaces. ## Real-Time Interaction and Screen Mirroring Nebula needed users to watch and control remote devices simultaneously, not merely replay predefined test steps. ### Android - Instead of using `scrcpy` directly, the team adapted its encoding approach. - Android’s `SurfaceControl` creates a virtual display. - `MediaCodec` encodes the output as H.264. - A broadcaster distributes the stream to multiple browser viewers. ### iOS - iOS screen capture is constrained by USB access and cannot expose the display as freely as Android. - Existing approaches such as QVH and Appium MJPEG did not support simultaneous viewing and interaction. - Nebula developed a capture path inspired by QuickTime’s iOS capture mechanism without exclusively claiming the USB connection. - Both platforms now use H.264 streaming and broadcasting, allowing the entire device farm to be viewed in a browser. ## Security and Compliance - Centralization made it possible to apply consistent security requirements across every device. - The team worked with Toss’s security organization to define mobile-device compliance standards. - Policies are enforced at the platform and driver levels rather than relying on individual developers. - Teams can therefore test on devices that already meet the company’s security requirements. ## Operating Hundreds of Devices Nebula must keep hardware and software running continuously. - **Hardware operations** - USB cables, hubs, and power delivery had to be tested and designed for large-scale use. - Physical failures still require human intervention. - Redundancy is being introduced to reduce service interruptions. - **Software operations** - Controllers and mirroring processes are orchestrated across Mac mini and Linux hosts. - Dead processes are automatically recovered. - Server, agent, controller, and mirroring deployments are performed without interrupting user tests. - Monitoring and observability cover devices, processes, and server performance. ## An Internal Ecosystem Built on the API Nebula has evolved from a device provider into shared testing infrastructure. - Web console for no-code interaction and test-step creation. - SDKs for writing E2E tests. - CLI support for terminals, CI/CD, and local AI agents. - Automated log verification during device interaction. - AI agents that plan and execute tests dynamically. - Direct API access for teams with specialized needs. Reported benefits include faster Appium migration, lower barriers to regression testing, and reducing manual verification from 30–40 minutes to under 10 minutes. Nebula’s main lesson is that a stable, simple API can become the foundation for a much broader testing ecosystem. Centralizing device infrastructure, replacing unsuitable abstractions, and treating real-time operation, compliance, and reliability as first-class requirements enabled Toss to scale physical-device testing across the company.

openapiandroidioskafka+4

lineJun 25, 2026

In the AI Era, Development Ability Is Determined by Verification Skills: Strategies for Rapid Validation and Local Environment Setup Learned While Developing the Flava API Gateway (opens in new tab)

AI coding agents iterate quickly, but their output can be inconsistent, make incorrect design decisions, or generate code that does not compile. Because CI runs, environment provisioning, and human review are slower, the article argues that reliable agent-assisted development requires three practices: spec-driven development, automated verification, and fast, self-contained local environments. ## Flava API Gateway and the Development Challenge - Flava API Gateway is part of LY Corporation’s private Flava cloud. - It provides a multi-tenant RESTful control-plane API for creating, deploying, and monitoring web APIs. - Kong serves as the data plane. - The team adopted agent-based coding while building the product and focused on preserving software reliability without sacrificing AI-driven speed. ## Spec-Driven Development The team found that agents became more unpredictable when implementation began before the design was settled. They use explicit specifications to reduce ambiguity and constrain implementation decisions. - OpenAPI is written before code to define the control-plane API. - Features are divided into smaller units and implemented with OpenSpec. - Specifications serve both as implementation guidance and as a standard for detecting deviations. ### Managing OpenAPI with Nickel - Raw OpenAPI YAML is repetitive and difficult to maintain manually. - Nickel is used to describe API resources declaratively and generate complete CRUD specifications. - A resource definition can specify: - Description and parent resource - Whether updates are allowed - Automatic timestamps - Property schemas - Required fields - Sorting and filtering behavior - The generator produces consistent endpoints such as `listPaths`, `createPath`, `getPath`, and `deletePath`. - Generated endpoints include pagination, sorting, filtering, ETags for optimistic locking, and standardized error responses. ### OpenSpec Workflow OpenSpec structures each change into four artifacts: - **Proposal:** Why the change is needed and what will change - **Design:** Technical decisions and trade-offs - **Delta specifications:** Behavioral requirements written as Given-When-Then scenarios - **Task list:** A step-by-step implementation checklist The developer and agent review the feature together, the agent creates these artifacts, and then implements the checklist incrementally. Once complete, the delta specification is archived into the main specification library, creating a versioned, evolving record of the system’s behavior. ## Automated Verification The team initially tried adding lists of pitfalls to prompts, but found this ineffective and potentially harmful. Instead, they made tests and tools reveal errors progressively so the agent could diagnose and correct them. - Automated tests, linters, and formatters provide precise feedback. - Failed tests identify what went wrong, allowing the agent to fix one issue before moving to the next. - Project-specific skills bundle these checks together. - `AGENTS.md` tells the agent when to load the relevant skills, avoiding unnecessary instructions on every turn. - Testing and linting are treated as essential infrastructure rather than optional activities, since agents frequently make errors during implementation. ## Fast, Independent Local Environments Relying on CI and shared test environments is too slow for agent-driven iteration. Long waits can disrupt the agent’s context and make repeated experimentation impractical. - A complete local environment provides immediate feedback. - Local dependencies make logs and state easier to inspect. - Developers avoid sending every failed attempt through a remote pipeline. - The local test suite contains 2,754 tests across three layers: - **Unit tests:** Isolated business logic - **Integration tests:** Real PostgreSQL, database constraints, triggers, soft-delete cascades, transactions, in-process HTTP, and OpenAPI compliance - **End-to-end tests:** Athenz authentication, Kong, API keys, and multi-tenant isolation - The full suite completes in roughly 15 seconds on a developer machine. - Parallel execution and strong test isolation are critical to achieving this speed. ## Practical Recommendation Agent-assisted development works best when agents are given clear behavioral contracts, immediate automated feedback, and a fast local loop. Teams should invest in specifications, comprehensive tests and linting, and realistic local dependencies so agents can correct mistakes continuously without waiting for CI.

openapipostgresqlcoding-agentsautomated-testing+4

cloudflareMar 9, 2026

Active defense: introducing a stateful vulnerability scanner for APIs (opens in new tab)

Cloudflare is launching a beta Web and API Vulnerability Scanner to actively detect API logic flaws that defensive tools often miss. Its first target is Broken Object Level Authorization (BOLA), where authenticated users can access or modify another user’s resources through valid requests. The scanner combines Cloudflare’s existing API visibility with stateful DAST to test authorization across sequences of API calls. ## Why Defensive Security Misses API Logic Flaws - Traditional WAFs are effective against recognizable attacks such as SQL injection, XSS, malformed requests, and suspicious payloads. - API vulnerabilities often involve valid requests that violate business rules rather than protocol or schema requirements. - In the food delivery example: - User A sends a valid `PATCH` request for User B’s order. - User A’s token, headers, and request schema are all legitimate. - The vulnerability exists because the API fails to verify that User A owns the order. - A basic authorization check could prevent the issue: ```js if (order.userID != user.ID) throw Unauthorized; ``` ## Passive Detection and the Limits of Traditional DAST - Cloudflare’s existing API Shield BOLA detection passively analyzes customer traffic for abnormal usage patterns. - Effective passive detection requires context about: - Valid API calls - Variable parameters - Normal user behavior - API responses when parameters are manipulated - Passive analysis may be insufficient in development environments with little traffic or production systems without observed attacks. - DAST creates new traffic specifically for security testing and can operate in environments without relevant user activity. - Traditional DAST tools often: - Require extensive configuration - Depend on manually maintained Swagger/OpenAPI files - Struggle with modern authentication flows - Lack API-specific tests such as BOLA detection ## Cloudflare’s API Scanning Advantage - Scan results will appear in Security Insights alongside other Cloudflare security findings. - API Shield customers already benefit from Cloudflare’s API Discovery and Schema Learning, which catalog endpoints and learn traffic patterns. - The initial release requires an uploaded OpenAPI specification, though future versions are expected to work without one. - Cloudflare can use passive traffic inspection to identify possible BOLA issues and actively verify them with new HTTP requests. - Customers provide API credentials, while Cloudflare uses API schemas to construct a scan plan automatically. ## Stateful API Scanning - Conventional scanners often evaluate requests independently, making it difficult to test vulnerabilities that require a sequence of related actions. - BOLA testing may require: - Creating a resource as one user - Attempting to access or modify it as another user - Comparing the resulting behavior - Cloudflare’s scanner builds an API call graph from the OpenAPI document. - It walks that graph using separate owner and attacker contexts: - Owners create resources. - Attackers use their own valid credentials to attempt access. - This stateful approach is designed to test authorization across realistic API workflows rather than isolated requests. Cloudflare’s scanner is intended to complement—not replace—passive monitoring and edge defenses. Organizations should use the beta to actively test APIs, especially authorization controls, in environments where normal traffic provides insufficient security context.

openapicloudflareapi-securityvulnerability-scanning+3

cloudflareFeb 20, 2026

Code Mode: give agents an entire API in 1,000 tokens (opens in new tab)

Code Mode addresses a central MCP problem: giving agents broad tool access without overwhelming their context windows. Instead of exposing thousands of individually described tools, it provides a typed SDK and lets agents write sandboxed JavaScript to search specifications and execute API calls. Cloudflare’s new MCP server applies this approach to its entire API using only `search()` and `execute()`, reducing context usage by 99.9% to roughly 1,000 tokens. ## The Context-Window Problem - MCP makes external tools available to AI agents, but each tool’s description consumes model context. - Large APIs can require enormous token budgets merely to describe their endpoints. - A conventional MCP server for the Cloudflare API would consume approximately 1.17 million tokens. - This can exceed the context window of even advanced foundation models. ## Code Mode as a Compact API Interface - Agents write code against a typed SDK rather than selecting from thousands of separate tools. - Generated code serves as a compact, composable plan. - Agents can: - Explore available operations. - Chain multiple API calls. - Handle pagination and errors. - Return only the data needed for the task. - The tool footprint remains fixed as the underlying API grows. ## Cloudflare’s Two-Tool MCP Server - The server exposes only: - `search()` — searches the Cloudflare OpenAPI specification. - `execute()` — runs JavaScript against the Cloudflare API. - The full OpenAPI specification stays outside the model context. - `$refs` are pre-resolved, allowing agents to inspect endpoint metadata directly through code. - Agents can filter endpoints by product, path, tags, or other metadata. ## Sandboxed Execution - Both tools execute code inside a Dynamic Worker isolate based on lightweight V8 sandboxes. - The environment has: - No filesystem access. - No environment variables exposed to generated code. - External fetches disabled by default. - Outbound requests can be explicitly enabled through controlled fetch handlers. ## Example: Finding DDoS Protection Endpoints - For a request to protect an origin from DDoS attacks, an agent can first consult documentation or skills. - It then uses `search()` to inspect the OpenAPI specification. - A JavaScript query can locate zone endpoints related to WAF and rulesets. - The search narrows more than 2,500 Cloudflare endpoints to relevant operations such as: - Listing and updating WAF packages and rules. - Listing and creating zone rulesets. - Reading and updating entry-point rulesets. - Creating and updating rules within a ruleset. - Once the relevant endpoints are identified, `execute()` can perform coordinated API requests in one sandboxed run. ## SDK Availability - Cloudflare is open-sourcing a Code Mode SDK as part of the Cloudflare Agents SDK. - The same pattern can be used to build more context-efficient MCP servers and AI agents. Code Mode is most useful for large, evolving APIs: expose a small, stable tool surface, let agents discover capabilities programmatically, and execute only the focused operations required for each task.

openapiai-agentmodel-context-protocoljavascript+3