Database Design

191 posts

toss4 min readCurated summary

Applying Post-Quantum Cryptography for the Quantum Computing Era: Why Implement It 10 Years Early?

Toss Payments’ hardest legacy-modernization challenge was not replacing old systems, but improving security across tens of thousands of merchants with diverse, outdated environments. Over four years, it gradually introduced HTTP/3, removed vulnerable cipher suites, deployed TLS 1.3, and ultimately adopted post-quantum cryptography (PQC) in April 2026. The central lesson is that security upgrades must begin early and be introduced gradually, with merchant support and backward compatibility built in. ## Breaking the Inertia of Legacy Systems - Mission-critical payment services tend to follow the principle: “If it works, don’t touch it.” - Security protocol changes are particularly difficult because they can affect every merchant integration and may be hard to troubleshoot or roll back. - Many merchants still operate decades-old server-side systems that cannot support modern security policies. - Documentation alone is often insufficient, especially for merchants without dedicated development teams. - Because every API, SDK, payment window, and server connection is part of the security boundary, Toss Payments could not improve security independently of its merchants. ## Why Existing Encryption Must Evolve - Modern HTTPS, banking, and payment systems rely heavily on RSA and ECDSA. - These algorithms are considered secure because factoring large numbers and solving elliptic-curve problems is impractical for classical computers. - Quantum computers could solve these problems efficiently, making current public-key cryptography vulnerable. - The anticipated point at which quantum computers can break these systems is often called “Q-Day.” - The “Harvest Now, Decrypt Later” threat means attackers can collect encrypted payment data today and decrypt it years later when quantum computers become practical. ## A Four-Year Security Upgrade Program Toss Payments chose a gradual migration strategy to improve security without abruptly disrupting merchant payments: - **2022:** Introduced HTTP/3, which requires TLS 1.3. - **2022–2025:** Removed vulnerable TLS cipher suites. - **2022–2025:** Enabled TLS 1.3 across all endpoints. - **April 2026:** Introduced post-quantum cryptography. ## HTTP/3 as a Low-Impact Starting Point - HTTP/3 improves speed and reliability on unstable networks. - Because it requires TLS 1.3, enabling HTTP/3 also raised security standards. - Modern browsers automatically select HTTP/3, so merchants required no configuration changes. - This made HTTP/3 an effective first step with minimal migration risk. ## Gradual Cipher Suite Removal - A cipher suite defines the algorithms used by a client and server to establish encrypted communication. - Some legacy merchant servers supported only vulnerable suites, such as `TLS_RSA_WITH_AES_128_CBC_SHA`. - Removing them immediately could stop payments for affected merchants, while delaying removal would leave the wider ecosystem exposed. - Toss Payments used: - Merchant-by-merchant compatibility analysis - Individual notifications six months to a year in advance - Environment-specific documentation and configuration guidance - Technical consulting where necessary - The Technical Account Manager team was essential in coordinating these changes and communicating with merchants in accessible language. ## TLS 1.3 Deployment - TLS 1.2 remained the minimum supported version, while TLS 1.3 was added alongside it. - Clients capable of TLS 1.3 automatically use the stronger protocol. - Older clients continue using TLS 1.2 without forced changes. - TLS 1.3 was enabled endpoint by endpoint from 2022 and supported across all endpoints by 2025. - The process demonstrated that ecosystem-wide security improvements require more time helping merchants migrate than technically changing the servers. ## Post-Quantum Cryptography - Toss Payments began preparing for PQC in 2025 and completed deployment in April 2026. - Modern browsers and clients that support PQC automatically use stronger quantum-resistant channels. - Unsupported environments continue using established encryption methods, preserving compatibility. - Merchants do not need to change configurations or update their integrations. - The approach provides stronger protection against future quantum attacks while minimizing present-day disruption. ## Cross-Team Collaboration - **Infra Team:** Applied PQC within Toss Payments’ private data-center infrastructure and physical hardware. - **Server Platform Team:** Integrated PQC into live traffic paths in AWS. - **TAM Team:** Used its experience from the cipher-suite migration to guide merchants and assess integration environments. - The result was a large-scale, proactive security deployment across the private payment ecosystem. Toss Payments’ experience suggests that organizations should start security migrations well before threats become immediate. Compatibility layers, staged enforcement, and sustained technical support allow legacy ecosystems to adopt stronger security without sacrificing availability.

Read original(opens in new tab)
cloudflare3 min readCurated summary

Making Rust Workers reliable: panic and abort recovery in wasm‑bindgen

Rust Workers historically treated Rust panics and aborts as fatal WebAssembly failures, potentially poisoning a Worker instance and causing unrelated requests to fail. Cloudflare’s latest work upstreamed into `wasm-bindgen` adds comprehensive recovery: `panic=unwind` preserves application state after recoverable panics, while abort handling ensures Rust code cannot run again after an unrecoverable abort. ## Initial Recovery Mitigations - Early Rust Workers used a custom panic handler to track failures and reinitialize the entire application before serving later requests. - JavaScript bindings were wrapped with Proxy-based indirection so every Rust entry point passed through recovery logic. - Generated bindings were modified to reinitialize the WebAssembly module after failures. - This approach shipped by default in `workers-rs` 0.6 and prevented persistent failure modes, but reinitialization could discard in-memory state. ## Panic Unwinding with WebAssembly Exception Handling - WebAssembly’s `wasm32-unknown-unknown` target traditionally defaults to `panic=abort`, turning panics into traps and `WebAssembly.RuntimeError` exceptions. - With WebAssembly Exception Handling support, Rust can be compiled using: ```bash RUSTFLAGS='-Cpanic=unwind' cargo build -Zbuild-std ``` - Unwinding allows Rust destructors to run, preserving state and cleaning up resources instead of terminating the entire instance. - `std::panic::catch_unwind` can translate a Rust panic into a recoverable `Result`. ## Changes to wasm-bindgen - The Walrus WebAssembly parser was updated to understand `try`/`catch` exception-handling instructions. - The descriptor interpreter was updated to evaluate code containing exception blocks. - Generated exports now catch Rust panics at the Rust–JavaScript boundary and expose them as `PanicError` exceptions. - Async exports reject their JavaScript promises with `PanicError`. - Exported functions use `extern "C-unwind"` so unwinding is explicitly permitted across the boundary. - A `MaybeUnwindSafe` trait checks `UnwindSafe` requirements only when compiling with `panic=unwind`. - For closures that cannot safely unwind, `Closure::new_aborting` provides an explicit alternative that terminates on panic rather than risking invalid state. ## Results of `panic=unwind` - Panics in exported Rust functions are caught by `wasm-bindgen`. - JavaScript receives a `PanicError`. - Async calls reject their promises instead of poisoning the Worker. - Rust destructors execute correctly. - The WebAssembly instance remains valid and reusable. - Stateful applications, including Durable Objects, can recover without losing all in-memory state. ## Abort Recovery - `panic=unwind` cannot handle aborts such as out-of-memory failures because aborts do not unwind. - The remaining recovery mechanism prevents Rust code from being re-entered after an abort, avoiding repeated execution in a corrupted WebAssembly state. - Together, unwinding and abort recovery prevent one failed request from poisoning sibling or future requests. The recommended approach is to use the latest `wasm-bindgen` and Rust Workers releases, enabling `panic=unwind` where state preservation matters while using explicit aborting closures when unwind safety cannot be guaranteed.

Read original(opens in new tab)
gitlab3 min readCurated summary

GitLab AI Hackathon 2026: Meet the winners

Nearly 7,000 developers participated in GitLab’s 2026 AI Hackathon, creating more than 600 agents and workflows for the GitLab Duo Agent Platform. The winning projects focused on practical software delivery challenges—including organizational knowledge loss, security, compliance, migrations, and sustainability—rather than simple chatbot interactions. The results suggest that agentic AI is becoming most valuable when integrated directly into development workflows and given richer project context. ## Hackathon Scope and Evaluation - The hackathon ran from February 9 to March 25, 2026, on Devpost. - Google Cloud and Anthropic co-sponsored the event, contributing judges, prizes, and cloud resources. - Nineteen judges evaluated projects on: - Technical execution - Design - Potential impact - Quality of the idea - Total prizes reached $65,000. ## Grand Prize: LORE - LORE, or Living Organizational Record Engine, addresses the loss of institutional knowledge when engineers leave. - It combines: - Eight specialized agents - A router that directs questions to the appropriate agent - Protections against circular loops in its knowledge graph - A visual dashboard - Carbon tracking - Its command-line tool includes 43 tests, leading judges to describe it as a polished product rather than a typical hackathon prototype. ## Google Cloud and Anthropic Winners - **Gitdefender**, the Google Cloud Grand Prize winner, detects security issues during code review, writes fixes, and opens the review automatically. - **Aegis**, the Google Cloud Runner Up, explains the reasoning behind its AI decisions and is deployed on Google Cloud. - **GraphDev**, the Anthropic Grand Prize winner, maps code relationships and shows how systems evolve, helping developers understand the impact of changes. - **DocSync**, the Anthropic Runner Up, uses Detector, Writer, and Reviewer agents to update documentation. It opens a review when confident and creates an issue for human review when uncertain. ## Category Winners - **Time-Traveler**, winner for technical achievement, creates a safe copy of a production environment and runs database migrations against it using five connected agents, PostgreSQL, real data, and Google Cloud deployment. - **RedAgent**, the most impactful project, verifies AI-generated security findings before developers act on them, addressing distrust in automated reports. - **Launch Control**, recognized for ease of use, combined polished user experience, strong infrastructure, and sustainability considerations. ## Sustainability-Focused Projects - Five projects received sustainability prizes or bonuses as the organizers highlighted the growing energy cost of CI/CD systems and large language models. - **GreenPipe** analyzes CI/CD pipelines and generates carbon-footprint reports. - Sustainable Design bonuses recognized projects including: - **BugFlow**, which generated 10 fixes from one bug report in 20 minutes - **DELTA Cyber Reasoning**, an automated fuzz-testing tool - **CarbonLint**, which applies code analysis to energy consumption - **TFGuardian**, which includes carbon-footprint analysis - One project reduced monthly costs from $556 to $18, representing a reported 96% carbon reduction. ## Honorable Mentions - **SecurityMonkey** tests security scanners by injecting known vulnerabilities. - **stregent** enables CI/CD investigation and fixes through WhatsApp. - **Compliance Sentinel** evaluates merge requests for compliance risk and blocks critical violations. - **Carbon Tracker** measures the carbon footprint of individual pipeline jobs and suggests improvements. - **RepoWarden** captures the rationale behind code, not only its behavior. - **MR Compliance Auditor** maps merge-request evidence to SOC 2 controls and displays compliance scores in real time. ## What Comes Next The projects operated within a single GitLab project, but many teams supplemented their agents with local knowledge graphs to understand code relationships and dependencies. GitLab plans to build on this approach in future hackathons by providing agents with richer context. GitLab’s hackathon demonstrates that the strongest AI agents are workflow-integrated tools that can investigate, make decisions, execute changes, and involve humans when needed. Developers can explore the 600-plus projects in the gallery or build their own agents on the GitLab Duo Agent Platform.

Read original(opens in new tab)
spotify3 min readCurated summary

Building a Natural Language Interface to the Spotify Ads API with Claude Code Plugins | Spotify Engineering

The post describes an open-source Claude Code plugin that lets users manage Spotify advertising campaigns through natural-language requests. It translates high-level intent into validated, multi-step Spotify Ads API calls, handling targeting lookups, budget conversions, entity dependencies, and missing information. The authors favor a Markdown- and CLI-based design over MCP because it keeps the system transparent, lightweight, customizable, and grounded in Spotify’s OpenAPI specification. ## Natural-Language Campaign Creation - Users can request campaigns conversationally, such as creating an audio campaign targeting a specific age range and location with a daily budget. - The agent orchestrates the campaign lifecycle: - Creates the campaign. - Creates an ad set with targeting and budget. - Creates the ad and attaches creative assets. - It handles implementation details automatically: - Looks up geographic targeting IDs. - Converts dollar amounts into API micro-units. - Validates audience size. - Passes entity IDs between dependent API calls. - Prompts for missing required information. ## Claude Code Plugin Architecture - **Skills** provide slash commands, with each command defined in Markdown alongside its endpoints, request formats, and output behavior. - **Agents** interpret freeform requests and decompose them into the necessary API operations. - **Hooks** intercept tool calls to refresh OAuth tokens and inject HTTP headers. - **Settings** store local user configuration, including credentials, ad accounts, and environment preferences. - Because all components are human-readable Markdown, the plugin has no compilation, bundling, or package-management step. - API behavior can often be corrected by updating documentation or instructions rather than changing compiled code. ## CLI and OpenAPI Instead of MCP - The authors avoided MCP because the Spotify Ads API has more than 30 resource types and complex nested schemas. - Defining every endpoint as an MCP tool would create a large static registry and consume context even when most tools were irrelevant. - The plugin loads only the API documentation needed for a particular task. - API calls are issued as visible `curl` commands, allowing users to inspect, copy, modify, and reproduce them. - Spotify’s roughly 8,600-line OpenAPI v3 specification serves as the single source of truth. - Updating one bundled specification file is simpler than maintaining a separate translation into MCP schemas. ## Domain-Specific Agent Behavior - The request-builder agent is defined in `agents/spotify-ads-request-builder.md`. - It teaches the model Spotify-specific conversions, including: - Dollar values to micro-amounts. - Natural-language dates to ISO 8601. - Platform names to API enum values. - It performs multi-step orchestration for campaign, ad set, and ad creation. - It resolves locations such as “Connecticut” through geo-targeting search endpoints and builds the required `geo_targets` structure. - It performs pre-flight audience estimates to ensure targeting meets minimum size requirements. - It is designed to control execution carefully and validate requests before making changes that could affect advertising budgets. ## Practical Implication The plugin demonstrates that a large advertising API can be made approachable without hiding its mechanics. A Markdown-based Claude Code integration, backed by the official OpenAPI specification and transparent CLI requests, offers a practical balance between natural-language convenience, developer control, auditability, and maintainability.

Read original(opens in new tab)
meta2 min readCurated summary

Modernizing the Facebook Groups Search to Unlock the Power of Community Knowledge

Facebook has re-architected Groups Search to make community knowledge easier to discover, understand, and validate. Its new hybrid retrieval system combines keyword matching with semantic search, while automated model-based evaluation measures relevance at scale. The result is improved search engagement and relevance without increasing error rates. ## Friction in Community Search - **Discovery:** Traditional lexical search depends on exact words, so a query for “small individual cakes with frosting” might miss posts discussing “cupcakes.” Semantic matching helps connect different phrasing with the same intent. - **Consumption:** Users often must read dozens of comments to identify consensus or useful advice, creating an “effort tax.” - **Validation:** Relevant expertise is frequently scattered across group discussions, making it difficult to evaluate purchases or decisions using community knowledge. ## Hybrid Retrieval Architecture - Queries are tokenized, normalized, and rewritten before retrieval. - The **lexical path**, powered by Facebook’s Unicorn inverted index, retrieves exact or closely matching terms and preserves precision for proper nouns and quotations. - In parallel, the **semantic path** uses a 12-layer, 200-million-parameter Search Semantic Retriever to encode queries into dense vectors. - Approximate nearest-neighbor search over a Faiss index retrieves conceptually similar posts, even when they use different words. ## Multi-Task Ranking - Results from lexical and semantic retrieval are merged for ranking. - The ranking model combines traditional signals such as TF-IDF and BM25 with semantic cosine-similarity scores. - A multi-task, multi-label model jointly optimizes for clicks, shares, and comments. - This approach balances theoretical relevance with the likelihood of meaningful community engagement. ## Automated Relevance Evaluation - Semantic similarity scores can be difficult to interpret, so evaluation was integrated into build verification testing. - Llama 3 with multimodal capabilities acts as an automated judge of search results. - Evaluation recognizes nuanced outcomes, including “somewhat relevant” results that share a broader domain or theme. - This enables scalable measurement of conceptual matching and result diversity without relying entirely on human labeling. ## Results and Future Work - The hybrid system outperformed the keyword-only baseline in offline quality and search-engagement metrics. - Facebook reports improved relevance without higher error rates. - Future plans include using LLMs directly during ranking and dynamically adapting retrieval parameters to query complexity. The approach demonstrates that combining lexical precision with neural semantic understanding can make community search more effective. Further LLM integration may help the system interpret post content and tailor retrieval more intelligently.

Read original(opens in new tab)
line3 min readCurated summary

ODW #3: Boosting Development Efficiency by Safely Utilizing MCP Servers

LY Corporation is expanding AI use across its engineering organization through MCP servers, which connect AI assistants with internal and external tools through a common protocol. The company combines this flexibility with allowlists, automated security checks, and internal standards to reduce risk. Its Orchestration Development Workshop demonstrates practical applications such as Jira ticket automation and multi-agent code reviews, while emphasizing shared learning and experimentation as AI practices evolve. ## MCP Servers and Their Benefits - MCP servers act as translators between AI assistants and external systems. - Before MCP, each assistant required a separate integration for every tool. - With MCP, a tool can implement one standardized interface and work with multiple compatible assistants. - This improves interoperability, scalability, and the ability to combine different AI tools. ## Security Risks and LY Corporation’s Controls - A 2025 Astrix Security report found that: - More than 5,200 public MCP servers were analyzed. - 53% relied on long-lived static API keys or personal access tokens. - Only 8.5% used newer authentication methods such as OAuth. - LY Corporation manages externally developed MCP servers through: - An allowlist permitting only approved servers. - Automated security verification based on internal standards. - Internal MCP servers for groupware and business systems are built to meet the company’s security requirements. - Centralized infrastructure lets teams focus on applying AI rather than independently rebuilding integrations and controls. ## Workshop Applications The Orchestration Development Workshop taught participants how to understand, configure, and safely apply MCP servers with AI assistants. - Topics included MCP fundamentals, security risks, internal policies, development rules, and configuration in Claude and Cline. - The internal plugin marketplace was introduced as a way to distribute MCP configurations. - Participants practiced using Claude Code with the internal groupware MCP server to: - Generate a Jira ticket title and summary. - Create the ticket automatically. - The exercise showed how AI can remove repetitive administrative work and free time for higher-value tasks. ## Multi-Agent Code Review Demonstration - A demonstration combined Claude Code, Codex CLI, Context7 MCP, and Codex MCP. - A Sonnet-based agent first analyzed a pull request, including: - Technical stack and relevant documentation. - Code changes and repository context. - Security, performance, and code-quality concerns. - GPT-5 then validated the initial review, identifying missed issues and checking the prioritization of findings. - Using different models provided more varied and potentially objective perspectives on the same code. ## Results and Organizational Learning - Around 1,600 people attended the workshop in real time. - 31.5% had already applied related techniques before the event. - Another 55.7% planned to try them soon. - LY also created “Help LY MCP,” a GPTs-based tool that explains internal MCP rules and helps teams assess whether proposed uses are suitable, including for global subsidiaries. - The workshop’s broader purpose was to create a shared understanding of: - What AI and MCP can currently do. - What risks and pitfalls exist. - How to use the technology meaningfully. ## Continuing to Experiment The article concludes that rapidly changing AI technology makes shared experimentation more valuable than simply announcing new tools. MCP may eventually be surpassed by other approaches, such as skills, so teams should continually reassess the best solution. LY recommends creating a culture where employees can safely try small ideas, learn together, and adapt as new practices emerge.

Read original(opens in new tab)
google2 min readCurated summary

ReasoningBank: Enabling agents to learn from experience

ReasoningBank is an agent-memory framework designed to help deployed agents learn continuously from both successful and failed task attempts. Rather than storing exhaustive action histories or only successful workflows, it distills reusable reasoning strategies, decision rationales, and preventative lessons. Evaluations on WebArena and SWE-Bench-Verified show higher success rates and fewer execution steps, especially when combined with memory-aware test-time scaling. ## Distilling Generalizable Reasoning - Each memory contains: - A concise title - A brief description - Detailed reasoning steps, rationales, or operational insights - The agent retrieves relevant memories before acting. - After completing a task, an LLM judge evaluates the trajectory and identifies useful success or failure signals. - The agent converts those signals into new memories and appends them to the ReasoningBank. - Failure analysis is central: mistakes become counterfactual guidance and strategic guardrails, such as verifying the current page before repeatedly clicking “Load More.” ## Memory-Aware Test-Time Scaling - Memory-aware test-time scaling (MaTTS) connects inference-time exploration with long-term memory. - **Parallel scaling:** Multiple trajectories are generated and compared, allowing the agent to distinguish robust strategies from flawed reasoning. - **Sequential scaling:** The agent progressively refines a single trajectory, preserving useful intermediate insights from trial and error. - This creates a feedback loop: better memories guide exploration, while richer exploration produces better memories. ## Benchmark Results and Strategic Maturity - Against memory-free ReAct agents using Gemini-2.5-Flash: - Success rates improved by 8.3% on WebArena. - Success rates improved by 4.6% on SWE-Bench-Verified. - SWE-Bench-Verified tasks required nearly three fewer execution steps on average. - Adding MaTTS with parallel scaling factor **k=5** produced further gains: - A 3% success-rate increase over ReasoningBank alone on WebArena. - 0.4 fewer steps per task. - Over repeated tasks, simple procedural checklists evolved into more sophisticated memories containing compositional and preventative logic. ReasoningBank suggests that effective agent scaling requires more than additional inference compute or stored trajectories. Agents should systematically learn from both outcomes and mistakes, using structured reasoning memories to become more capable and efficient after deployment.

Read original(opens in new tab)
cloudflare4 min readCurated summary

The AI engineering stack we built internally — on the platform we ship

Cloudflare built an internal AI engineering stack that now supports AI coding tools for 93% of its R&D organization. The system combines centralized authentication and model routing with internal knowledge, automated review, and sandboxed agent execution. Cloudflare argues that building these capabilities on its own platform improved security, visibility, cost control, and developer velocity, while also serving as a proving ground for products it ships publicly. ## Adoption and Impact - Over the previous 30 days: - 3,683 employees used AI coding tools, representing 60% of the company and 93% of R&D. - AI tools generated 47.95 million requests. - 295 teams used agentic AI tools or coding assistants. - AI Gateway handled 20.18 million requests and routed 241.37 billion tokens. - Workers AI processed 51.83 billion tokens. - The four-week rolling average of merge requests increased from roughly 5,600 per week to more than 8,700. - The week of March 23 reached 10,952 merge requests, nearly twice the Q4 baseline. - Cloudflare’s initial focus was MCP servers, but the effort expanded to standards, code review, onboarding, and propagating changes across thousands of repositories. ## Architecture at a Glance Cloudflare organized the stack into three layers: - **Platform layer:** Authentication, model routing, inference, MCP access, and code execution. - **Knowledge layer:** System context and repository guidance through Backstage and `AGENTS.md`. - **Enforcement layer:** Automated quality controls using AI Code Reviewer and the Engineering Codex. The stack uses Cloudflare products including: - **Cloudflare Access** for zero-trust authentication. - **AI Gateway** for centralized LLM routing, cost tracking, bring-your-own-key support, and zero-data-retention controls. - **Workers AI** for open-weight model inference. - **Workers and Access** for an MCP Server Portal with single OAuth. - **Dynamic Workers** for sandboxed agent-generated code execution. - **Agents SDK and Durable Objects** for stateful, long-running agent sessions. - **Sandbox SDK** for isolated cloning, building, and testing environments. - **Workflows** for durable, multi-step processes. - **Backstage** for a 16,000-plus-entity knowledge graph. ## Centralized Authentication and AI Routing - Cloudflare Access authenticates users and enforces zero-trust policies. - Every LLM request passes through AI Gateway, providing a single control point for: - Provider credentials - Usage and cost attribution - Model selection - Data-retention policies - Provider permissions - In the past month, frontier providers handled 91.16% of requests, while Workers AI handled 8.84%. - Cloudflare routes requests through a proxy Worker rather than connecting clients directly to AI Gateway. - The proxy enables later additions such as per-user attribution, model catalogs, permission enforcement, and support for new coding tools without changing client configurations. ## Workers AI and Open-Weight Models - Workers AI runs open-source models on GPUs distributed across Cloudflare’s global network. - Keeping inference on the same network as Workers, Durable Objects, and storage reduces latency, network failures, and cross-cloud configuration. - Kimi K2.5, with a 256,000-token context window, tool calling, and structured outputs, processes more than 7 billion tokens per day for a Cloudflare security agent. - Cloudflare estimates that running this workload on Workers AI is 77% cheaper than using a mid-tier proprietary model. - Workers AI is also used for: - Documentation review in CI - Generating `AGENTS.md` files - Lightweight inference where latency matters more than maximum model capability - Cloudflare expects open-source models to handle an increasing proportion of its internal workloads. ## One-Command Client Configuration - Engineers begin setup with: ```bash opencode auth login https://opencode.internal.domain ``` - The command uses an OpenCode discovery endpoint at: ```text https://opencode.internal.domain/.well-known/opencode ``` - The Worker-hosted endpoint provides authentication and configuration information. - This mechanism is designed to configure providers, models, MCP servers, agents, commands, and permissions without requiring engineers to edit configuration files manually. ## Overall Recommendation Cloudflare’s experience suggests that organizations adopting AI coding tools should build a centralized control plane early: authenticate users consistently, route model traffic through one managed gateway, maintain shared system knowledge, and enforce quality through automated review and isolated execution. Using the same production platform for internal tooling can also expose product gaps and accelerate improvements to the platform itself.

Read original(opens in new tab)
cloudflare3 min readCurated summary

Orchestrating AI Code Review at scale

Cloudflare built a CI-native AI code review system to reduce review bottlenecks without overwhelming engineers with noisy or generic model feedback. Instead of using one large prompt, it orchestrates up to seven specialized agents for areas such as security, performance, compliance, and documentation, then uses a coordinator to deduplicate and assess findings. The system now reviews tens of thousands of merge requests, approving clean changes and blocking serious bugs or vulnerabilities. ## Why Naive AI Review Wasn’t Enough - Traditional code review can leave merge requests waiting for hours and creates repeated context switching. - Commercial AI review tools provided useful functionality but lacked the flexibility and customization required across Cloudflare’s organization. - A basic “send the Git diff to an LLM” approach produced: - Vague recommendations - Hallucinated syntax errors - Repetitive advice such as adding error handling where it already existed - Complex codebases required specialized analysis rather than generic summarization. ## Specialized Agents and Coordination - The system launches up to seven focused reviewers covering: - Security - Performance - Code quality - Documentation - Release management - Internal Engineering Codex compliance - A coordinator agent: - Deduplicates overlapping findings - Evaluates the actual severity of issues - Produces one structured review comment - The system can actively block merges when it detects serious defects or security vulnerabilities. ## Plugin-Based Architecture - The platform uses composable plugins so it can support different: - Version-control systems - AI providers - Internal standards - Repository-specific requirements - Each plugin implements a `ReviewPlugin` interface with three lifecycle phases: - `bootstrap`: Runs concurrently and is non-fatal. - `configure`: Runs sequentially and is fatal if essential configuration fails. - `postConfigure`: Handles asynchronous work after configuration assembly. - Through `ConfigureContext`, plugins can: - Register agents and AI providers - Set environment variables - Inject prompt sections - Configure agent permissions - Plugins contribute through the context API rather than accessing the final configuration directly. - The core assembler combines these contributions into `opencode.json`. - This separation prevents unrelated components from becoming tightly coupled; for example, GitLab logic does not need to understand Cloudflare AI Gateway settings. ## Plugin Responsibilities - `@opencode-reviewer/gitlab` - Provides GitLab merge request data and a comment server. - `@opencode-reviewer/cloudflare` - Configures AI Gateway model tiers and fallback chains. - `@opencode-reviewer/codex` - Checks compliance with internal engineering RFCs. - `@opencode-reviewer/braintrust` - Adds distributed tracing and observability. - `@opencode-reviewer/agents-md` - Verifies that repository `AGENTS.md` instructions are current. - `@opencode-reviewer/reviewer-config` - Retrieves remote model overrides for individual reviewers. - `@opencode-reviewer/telemetry` - Tracks reviews asynchronously. ## Why OpenCode - Cloudflare already used OpenCode extensively and understood its behavior. - Its open-source implementation allows engineers to: - Investigate problems directly - Contribute fixes upstream - Extend the system through its SDK - Cloudflare engineers had contributed more than 45 upstream pull requests at the time of writing. - Its server-first design was especially important: - Review sessions can be created programmatically. - Prompts can be sent through an SDK. - Multiple concurrent sessions can be managed without scraping or wrapping a CLI interface. ## Coordinator Process - The coordinator runs OpenCode as a child process using `Bun.spawn`. - Its prompt is passed through `stdin` rather than a command-line argument. - This avoids Linux’s `ARG_MAX` limit, which previously caused `E2BIG` failures for unusually large merge requests containing extensive descriptions or logs. - OpenCode runs with `--format json`, emitting JSONL events through standard output. - This event-based interface allows the orchestration layer to collect and process results from concurrent reviewer sessions. A practical takeaway is to treat AI review as an orchestrated CI system rather than a single LLM prompt. Specialized agents, strict plugin boundaries, structured outputs, and observability are essential for making automated review reliable enough to influence merge decisions at organizational scale.

Read original(opens in new tab)
gitlab3 min readCurated summary

Prepare your pipeline for AI-discovered zero-days

AI is accelerating both vulnerability discovery and insecure code production, shrinking the time defenders have to respond from months to hours. The post argues that security teams cannot remain the final defense layer; security controls, automated triage, and remediation must operate directly within development pipelines. AI-generated fixes can help close the gap, but they must follow the same policies, approvals, testing, and audit requirements as human-authored code. ## The Remediation Backlog Is Already Too Large - Most exploited vulnerabilities are already known and have patches available, but organizations cannot remediate them quickly enough. - Sixty percent of breaches in the 2025 Verizon DBIR involved known vulnerabilities. - Developers spend roughly 11 hours per month fixing vulnerabilities after release. - The median time to close half of internet-facing vulnerabilities is 361 days, while exploitation can begin within hours. - AI-assisted development is increasing the volume of insecure code: - Fortune 50 repositories reportedly gained more than 10,000 security findings per month by mid-2025. - AI coding tools may introduce outdated patterns, hallucinated packages, insecure examples, and excessive dependencies. - Security AI should therefore operate within existing development policies and audit trails rather than as a disconnected tool. ## Security Enforcement Must Move Into the Pipeline - Every change should pass security controls at the merge request, which becomes the central enforcement point. - Policies should be defined once and applied consistently across teams and projects. - Exceptions should be explicitly approved and logged. - IDE checks can catch straightforward problems—such as hardcoded secrets, vulnerable imports, and deprecated APIs—before code reaches review. - This allows human reviewers to focus on complex issues such as reachability, exploitability, and architectural risk. ## Automated Triage and Governed Remediation - AI should reduce the volume of findings developers must investigate by assessing: - False positives - Reachability - Exploitability - Severity - AI-generated fixes should not bypass normal governance. - Remediation proposals should be submitted as merge requests, with: - Required scans - Policy enforcement - Human approvals - Confidence scores - Complete audit records - Human and AI-authored changes should follow the same review and compliance process. ## Example: Responding to an Emerging Vulnerability - A proof-of-concept exploit may appear before a CVE, NVD entry, or scanner signature exists. - A security agent can inspect dependency graphs across projects, identify affected versions and call paths, and rank production exposure. - Teams can then launch a coordinated remediation campaign: - Upgrade dependencies where patches exist. - Apply targeted code changes where they do not. - Block merge requests that retain the vulnerable dependency. - Require security approval for fixes. - Pipeline tests can reject faulty AI-generated patches, allowing the agent to revise them before developers approve the corrected version. - Automatically collected scan results, policies, approvals, and merge timestamps provide audit evidence without manual reconstruction. ## Strengthen the Pipeline Before Attackers Catch Up - Organizations should verify that security scans run on every merge request, not only in selected projects. - Pipelines should detect compromised or vulnerable dependencies before build time. - Critical findings should move quickly from detection to the responsible developer without unnecessary tool boundaries. - The central recommendation is to make pipeline enforcement, AI-assisted triage, and governed remediation standard parts of the software supply chain before comparable offensive AI capabilities become widely available.

Read original(opens in new tab)
cloudflare3 min readCurated summary

Introducing the Agent Readiness score. Check to see if your site is agent-ready

Cloudflare argues that websites must evolve beyond browser and search-engine compatibility to become usable by AI agents. Its new Agent Readiness score evaluates whether sites support standards for discovery, content access, bot control, and agent capabilities. Early data shows adoption is extremely low, creating both a challenge and an opportunity for sites that adopt these standards early. ## Agent readiness across the web - Cloudflare analyzed the 200,000 most visited domains, excluding categories unlikely to need agent interaction. - The resulting Cloudflare Radar dataset tracks adoption of AI-agent standards and will be updated weekly. - robots.txt exists on 78% of sites, but most files target traditional search crawlers rather than AI agents. - Only 4% of sites declare AI usage preferences through Content Signals. - Just 3.9% support Markdown content negotiation via `Accept: text/markdown`. - MCP Server Cards and API Catalogs based on RFC 9727 appear on fewer than 15 sites, showing how early these standards remain. ## The Agent Readiness score Site owners can test their websites at **isitagentready.com**. Cloudflare scans the site and scores it across four dimensions: - **Discoverability:** robots.txt, sitemap.xml, and Link Headers under RFC 8288. - **Content:** Markdown for Agents. - **Bot Access Control:** Content Signals, AI-specific robots.txt rules, and Web Bot Auth. - **Capabilities:** Agent Skills, API Catalogs, OAuth discovery standards, MCP Server Cards, and WebMCP. - The tool also checks commerce standards such as x402, Universal Commerce Protocol, and Agentic Commerce Protocol, though these do not yet affect the score. - Each failed check includes a prompt that can be handed to a coding agent for implementation. The service itself supports agents through a stateless MCP server with a `scan_site` tool and publishes Agent Skills documents explaining how to implement each supported standard. ## Discoverability for AI agents - robots.txt helps agents understand crawl permissions and locate sitemaps. - Sitemaps provide a structured list of site paths, reducing the need to discover content by following every HTML link. - HTTP Link headers, defined by RFC 8288, expose important resources directly in responses without requiring agents to parse page markup. - Sites can use headers such as `rel="api-catalog"` to point agents toward machine-readable capabilities. ## Making content easier to read - `llms.txt` provides an LLM-oriented reading list at the site root, describing the site and linking to important content in a format designed for model context windows. - Markdown content negotiation lets agents request a clean Markdown version of a page with `Accept: text/markdown`. - Cloudflare measured token reductions of up to 80% compared with HTML, improving speed, cost, and the likelihood that agents can consume an entire document within their context limits. Cloudflare’s recommendation is to evaluate sites with the Agent Readiness tool and adopt the relevant standards incrementally. With current adoption so low, early support can make a site significantly easier for AI agents to discover, understand, authenticate with, and use.

Read original(opens in new tab)
cloudflare3 min readCurated summary

Introducing Flagship: feature flags built for the age of AI

AI-generated code is moving toward autonomous production deployment, making safety and controlled rollout essential. The post argues that feature flags provide the guardrails: agents can deploy disabled code, test it with limited cohorts, monitor results, and roll back automatically. Cloudflare’s new Flagship service is designed for this workflow, evaluating flags at the edge through Workers, KV, and Durable Objects. ## Feature Flags for Autonomous Deployment - Agents can ship code behind an off flag without affecting users. - They can enable features for themselves or small test cohorts, observe metrics, and expand or disable rollouts. - Humans define boundaries while flags limit the blast radius. - This separates not only deployment from release, but also routine shipping decisions from constant human attention. ## Problems with Feature Flags on Workers - Hardcoded flags are initially convenient because Workers deploy quickly. - Over time, flags become fragmented across teams, with no central visibility or audit trail. - Troubleshooting may require searching version history with tools such as `git blame`. - Calling an external flag service adds a network request to every user request, potentially introducing significant latency. - This undermines the advantage of running applications close to users at the edge. ## Why Local Evaluation Is Difficult on Workers - Traditional local-evaluation SDKs download rules into a long-lived process. - Worker isolates may be created and evicted between requests, requiring repeated initialization. - Serverless environments therefore need a distribution system with edge-local reads and managed synchronization. - Flagship uses Cloudflare KV to provide this distribution without persistent connections or per-request external calls. ## How Flagship Works - Flagship is built on Workers, Durable Objects, and KV, without external databases or centralized evaluation servers. - Durable Objects provide a globally unique, SQLite-backed source of truth for flag configuration and changelogs. - Changes are synchronized to KV within seconds and replicated throughout Cloudflare’s network. - Evaluations read configuration from KV at the edge and execute targeting and rollout logic inside the Worker isolate. - Both flag data and evaluation logic remain close to the request. ## Worker Binding and Typed Evaluation - Workers connect Flagship through a `wrangler.jsonc` binding containing a binding name and `app_id`. - The binding supports typed methods including: - `getBooleanValue()` - `getStringValue()` - `getNumberValue()` - `getObjectValue()` - `*Details()` methods return the value, matched variant, and selection reason. - Evaluation errors return the supplied default value. - Type mismatches throw exceptions because they indicate application bugs rather than temporary service failures. ## OpenFeature Integration - Flagship is built on OpenFeature, the CNCF standard for feature-flag evaluation. - It supports Workers as well as Node.js, Bun, Deno, and browser environments. - The service is currently available in closed beta. Flagship is positioned as an edge-native feature-flag system for safely automating deployment and rollout. For Cloudflare Workers, its direct binding avoids network round-trips while providing centralized configuration, targeting, auditability, and controlled release mechanisms.

Read original(opens in new tab)
cloudflare3 min readCurated summary

Unweight: how we compressed an LLM 22% without sacrificing quality

Unweight is Cloudflare’s lossless compression system for LLM weights, reducing model size by 15–22% while preserving bit-exact outputs. It targets the memory-bandwidth bottleneck in GPU inference by compressing weights in HBM and decompressing them directly into fast on-chip memory before tensor-core computation. On Llama-3.1-8B, the approach saves roughly 3 GB of VRAM and enables more models to run per GPU. ## The GPU Memory Bottleneck - LLM inference is often limited by memory bandwidth rather than computation. - Each generated token requires reading the model’s weights from GPU high-bandwidth memory (HBM). - NVIDIA H100 tensor cores can process data far faster than HBM can supply it. - Smaller weights reduce the amount of data transferred across the memory bus. - Decompression must be carefully integrated: if it adds latency that cannot overlap with matrix multiplication, token generation becomes slower. ## Why Lossless Compression Matters - Quantization commonly converts 16-bit values into 8- or 4-bit integers. - Because quantization is lossy, it can change model behavior and response quality unpredictably. - Unweight instead preserves exact outputs and does not require specialized hardware. - Existing systems were unsuitable because they focused on CPU decompression, custom FPGA hardware, or consumer GPUs rather than Hopper-generation GPUs and production inference. ## Compressing BF16 Weights - BF16 values contain: - A sign bit - An 8-bit exponent - A 7-bit mantissa - Sign and mantissa values appear largely random and are difficult to compress. - Exponents are highly predictable: the 16 most common exponent values account for more than 99% of weights in a typical layer. - Unweight applies Huffman coding to exponent bytes while leaving sign and mantissa bits unchanged. - Rare exponents are handled by storing an entire row of 64 weights verbatim, avoiding per-element branching during decoding. ## Selective Compression of Model Layers - Unweight compresses the MLP gate, up, and down projection matrices. - These matrices represent roughly two-thirds of model parameters and generate substantial memory traffic during decoding. - Attention weights, embeddings, and layer norms remain uncompressed. - The exponent compression produces about 30% savings in the targeted streams and approximately 20% reduction in total MLP weight size. - Overall model-size reductions reach 15–22%. ## Direct GPU Decompression - Model weights normally reside in large but slower HBM and are staged into small, fast shared memory before computation. - Conventional approaches decompress full matrices back into HBM and then run standard matrix multiplication, creating additional memory traffic. - Unweight decompresses weights in shared memory and feeds them directly to tensor cores. - Different execution strategies are used depending on the weight matrix and batch size. - An autotuner selects the fastest strategy for each workload. ## Results and Availability - Tests on Llama-3.1-8B achieved: - Around 30% compression for MLP weights - 15–22% reduction in total model size - Approximately 3 GB of VRAM savings - The savings allow more models to fit on each GPU, potentially reducing inference cost and improving global deployment coverage. - Cloudflare is publishing a technical paper and open-sourcing the GPU kernels. Unweight demonstrates that lossless, inference-time compression can improve GPU utilization without changing model behavior. The practical recommendation is to compress the portions of a model that dominate memory traffic while integrating decoding directly into the GPU execution path.

Read original(opens in new tab)
meta3 min readCurated summary

Capacity Efficiency at Meta: How Unified AI Agents Optimize Performance at Hyperscale

Meta’s Capacity Efficiency Program uses AI agents to automate both the discovery and resolution of infrastructure performance issues. By combining standardized tools with encoded expertise from senior efficiency engineers, the platform turns investigations that once took hours into minutes and has recovered hundreds of megawatts of power. The approach aims to let Meta scale efficiency improvements across more product areas without proportionally increasing engineering headcount. ## Capacity Efficiency at Hyperscale - At Meta’s scale, even a 0.1% performance regression can significantly increase power consumption across systems serving more than 3 billion people. - The program has two complementary functions: - **Offense:** Proactively identify and implement optimizations. - **Defense:** Detect production regressions, identify their causes, and deploy mitigations. - Human investigation is often the bottleneck, requiring engineers to analyze profiling data, review documentation and prior fixes, inspect deployments, and search internal discussions. - AI automation can reduce roughly 10 hours of manual diagnosis to about 30 minutes. ## A Unified Platform for AI Efficiency Agents - Meta built one platform for both offensive and defensive workflows because they share the same basic structure: - Gather relevant technical context. - Apply domain-specific reasoning. - Produce a code change for review. - **MCP tools** provide standardized interfaces for querying profiling data, retrieving experiment results, examining configuration history, searching code, and accessing documentation. - **Skills** encode expert reasoning, including which tools to use and how to interpret their results. - The same tools support both use cases, while specialized skills handle different optimization and regression scenarios. ## Defense: Automated Regression Resolution - FBDetect monitors noisy production time series and can identify regressions as small as 0.005%. - Traditional root-cause analysis correlates the regression with recent pull requests or configuration changes. - Previously, teams often rolled back problematic changes—reducing engineering velocity—or left them unresolved, allowing resource waste to accumulate. - The AI Regression Solver: - Identifies affected functions and regression symptoms. - Locates the responsible pull request, files, and changed lines. - Applies mitigation expertise appropriate to the codebase, language, or regression type. - Generates a corrective pull request and sends it to the original author for review. - Faster resolution prevents small regressions from compounding across Meta’s infrastructure. ## Offense: Converting Opportunities into Code - Efficiency opportunities describe potential improvements to existing code, but implementing them traditionally required substantial investigation and engineering time. - Meta’s AI workflow gathers: - Opportunity metadata. - Optimization documentation. - Examples of similar fixes. - Relevant files and functions. - Validation criteria. - Skills then apply specialized knowledge, such as memoizing a function to reduce CPU usage. - The agent generates a guarded candidate fix, checks syntax and style, validates that it addresses the intended issue, and presents the change in an engineer’s editor for review or one-click application. - This expands the number of optimization opportunities engineers can pursue manually. ## Scaling Efficiency with AI - The platform has already recovered hundreds of megawatts of power—enough to supply hundreds of thousands of U.S. homes for a year. - Automated regression handling reduces ongoing waste, while automated opportunity resolution increases the volume of proactive improvements. - The long-term goal is a self-sustaining efficiency engine in which AI handles the long tail of investigations and fixes, allowing engineers to focus on new products and higher-value work. Meta’s approach recommends treating performance expertise as reusable, composable software: standardize access to engineering data, encode proven reasoning into skills, and let agents carry issues from detection through ready-to-review code changes.

Read original(opens in new tab)
cloudflare3 min readCurated summary

AI Search: the search primitive for your agents

AI Search is presented as a general-purpose search primitive for AI agents, handling retrieval across code, support documentation, customer history, and agent memory. It combines semantic and keyword search while providing built-in storage, indexing, and dynamically creatable search instances. The result is less infrastructure to build and the ability to maintain separate searchable contexts for agents, customers, languages, or other tenants. ## Why Agents Need Search - Agents often need access to information too large or dynamic to fit in a context window. - Common examples include: - Coding agents searching millions of repository files. - Support agents searching product documentation and ticket history. - Memory systems retrieving relevant past interactions. - Building this independently requires: - A vector index. - Document parsing and chunking. - An indexing pipeline that stays synchronized with changing data. - A separate keyword index and result-fusion layer if lexical search is also needed. ## Hybrid Search - AI Search runs vector search and BM25 keyword search in parallel. - Results are fused into a single ranking. - This supports both: - Semantic matches based on meaning. - Exact-term matches for names, identifiers, and technical terminology. - The blog’s own search is powered by AI Search. ## Built-In Storage and Dynamic Namespaces - New AI Search instances include managed storage and a vector index. - Files can be uploaded directly through an API and indexed automatically. - Developers do not need to configure R2 buckets or external data sources for every instance. - The `ai_search_namespaces` binding allows Workers to create and delete instances at runtime. - Instances can be created per: - Agent. - Customer. - Language. - Other isolated contexts. - Documents can include metadata used to boost rankings at query time. - A single query can search across multiple instances. ## Customer Support Agent Example - The example uses the Cloudflare Agents SDK and Workers AI. - A shared `product-knowledge` instance contains product documentation backed by an R2 bucket. - Each customer receives a separate instance such as `customer-abc123`. - After an issue is resolved, the agent stores a summary of the problem and its fix. - Over time, each customer’s instance becomes a searchable history of previous resolutions. ## Agent Tools and Retrieval Flow - The support agent extends `AIChatAgent` and uses Kimi K2.5 through Workers AI. - It defines tools for: - Searching shared product documentation and the current customer’s history in one call. - Saving a resolution after an issue is resolved. - The model decides when to invoke these tools based on the conversation. - Search results can prioritize recent documents using metadata, such as a descending `timestamp` boost. - The Agents SDK persists the conversation history across reconnects, while AI Search provides retrieval over larger knowledge collections. AI Search is recommended for teams that want agent-ready retrieval without separately assembling vector databases, keyword indexes, storage, and synchronization pipelines. Its dynamically isolated instances are particularly useful for multi-tenant agents and applications that need both shared knowledge and private, continuously growing context.

Read original(opens in new tab)