techlist.io
KREN

API Design

21 posts

github3 min readCurated summary

Turn one giant AI-generated pull request to a reviewable stack

Coding agents can rapidly produce complete features, but they often deliver them as enormous, shallow pull requests that are difficult to review and slow to merge. GitHub’s stacked pull requests address this by decomposing a feature into small, dependency-ordered layers. The result is a reviewable chain of changes that preserves context while reducing maintenance and merge conflicts. ## The Problem with Giant AI-Generated Pull Requests - A seemingly simple product-search feature may include: - A data model and seed data - An API route and validation - Client integration and UI states - Coding agents commonly generate all of this in a single 1,000-plus-line pull request. - Large pull requests: - Become difficult to review thoroughly - Cause reviewers to lose context - Receive lower-quality feedback - Take longer to merge - Are more likely to land under-reviewed Traditional alternatives are also imperfect: one large pull request harms reviewability, while a manually maintained chain of smaller pull requests creates synchronization work and conflict-management overhead. ## Stacked Pull Requests - Stacked pull requests break a feature into logical, dependent layers. - Each pull request focuses on one concern and remains small enough for reviewers to understand. - Later layers build naturally on earlier, already-reviewed work. - Different layers can be assigned to specialized reviewers, such as data or UI owners. For the product-search example, the proposed stack is: - **L1 – `feat/catalog-data`**: Typed catalog, seed data, validation, and data access; based on `main` - **L2 – `feat/search-api`**: Validated `/api/products/search` endpoint; based on L1 - **L3 – `feat/chat-grounding`**: Connects chat to the API and real product data; based on L2 - **L4 – `feat/grounded-ui`**: Adds product citation cards and UI states; based on L3 ## Setting Up the Stack - Choose the stack base first, because CI checks and merge rules are evaluated against it. - Place foundational work closest to the base and dependent work above it. - Install GitHub’s CLI extension: ```bash gh extension install github/gh-stack ``` - Teach coding agents how to create and manage stacks: ```bash gh skill install github/gh-stack ``` Alternatively: ```bash npx skills add github/gh-stack ``` - Ensure CI is configured, since every pull request layer is checked against the stack base. ## Assigning Agents to Layers The example uses separate agents with strict scope boundaries: - **L1:** Data modeler agent - **L2:** Backend agent - **L3:** Frontend agent - **L4:** Frontend agent This division encourages each agent to produce a focused pull request rather than reconstructing the entire feature in one pass. ## Recommended Workflow The development process starts with the foundational catalog layer and proceeds upward through the dependency chain. Agents work autonomously within their assigned scope, while each completed layer can be reviewed independently before subsequent layers are evaluated. Stacked pull requests are a practical way to preserve the productivity benefits of coding agents without sacrificing review quality. Teams should define clear layer boundaries, establish the stack base, assign appropriate reviewers or agents, and run CI for every layer.

Read original(opens in new tab)
API DesignCI/CDGithubCoding Agents+2
toss4 min readCurated summary

How DS and MLE Work Together

The post explains how Toss Bank improved collaboration between Data Scientists (DS) and ML Engineers (MLE) by progressively formalizing their responsibilities. What began as manually transferring notebooks evolved into standardized Python files and finally into installable model packages built around explicit interfaces. The result was faster deployments, consistent observability, and clearer ownership, while AI-generated code introduced a new need to standardize coding style as well. ## Problems with Notebook-Based Handoffs Initially, DS built models and inference code in Jupyter notebooks, then handed them to MLE. - MLE had to recreate the serving code from scratch. - Dependencies, configuration files, and source code were often missing or difficult to reproduce. - Preprocessing logic could be interpreted differently by DS and MLE. - “It works in the notebook” did not guarantee that it would work in production. - As the number of models increased, communication and rework grew rapidly. This approach separated people, not code, so the division of responsibility remained unclear. ## Phase 1: Separating Logic into `.py` Files The team next moved the collaboration boundary from people to files. - DS kept notebooks for experimentation and training. - Core inference logic was extracted into `.py` files. - MLE reviewed these files and validated them through CI. - DS’s intended model behavior was preserved more reliably. - Communication costs decreased. However, the files lacked a standardized structure. - Models used inconsistent function names such as `predict()`, `run()`, and `inference()`. - Code still required modifications when moved into the serving environment. - Global configuration changes in one model could affect other models sharing the same process. - Logging, metrics, and error handling could not be applied consistently across models. ## Phase 2: Defining an Interface Contract The team ultimately standardized the boundary through the `commons-ml-model` package. - A base abstraction defines a common model structure. - DS implements three methods: - `pre_process` - `inference` - `post_process` - The base class handles shared concerns such as: - Logging - Metrics - Tracing - Timing and request tracking - DS packages the implementation as a reusable library. - MLE installs the package with `pip install` and deploys it without rewriting the model. This turns the division of work into a code-level contract. DS focuses on model behavior, while MLE owns serving infrastructure and operational concerns. Updating the base abstraction can also add observability features to every model at once. ## Monorepo Collaboration The team manages the abstraction package and individual model packages in a single monorepo using `uv` workspaces. - Changes to the abstraction and affected models can be reviewed in one pull request. - DS and MLE review the same code in the same repository. - CI, release, and versioning policies are centralized. - Switching from Poetry to `uv` improved build speed by three to five times. The tradeoff is that changes to shared packages can affect many models, and the repository becomes heavier as more packages accumulate. ## Standardizing AI-Generated Code AI-assisted development created a separate collaboration problem: consistent structure did not guarantee consistent coding style. The team introduced `pfmls-stylepack` to encode team conventions for AI tools. - Naming conventions are standardized. - Exception-handling patterns are prescribed. - Rules determine when to use enums instead of hard-coded strings. - Hooks apply conventions while code is being generated. - AI-generated code can explain when a particular rule influenced its implementation. The team therefore distinguishes between: - **Structural consistency:** interfaces define what each role implements. - **Style consistency:** shared rules define how code should be written. Both are necessary for smooth reviews. ## Lessons from the Evolution - The hardest decision is choosing the right collaboration boundary: excessive structure limits flexibility, while insufficient structure recreates inconsistency. - Documentation and early DS–MLE pairing reduce the learning curve for the package-based workflow. - Shared libraries are a double-edged sword: one change can cause broad impact, but one fix can also benefit every model. - In the age of AI-generated code, teams must standardize not only responsibilities and interfaces but also implementation style. The practical recommendation is to make collaboration contracts executable: define stable interfaces, package model code for reuse, centralize shared serving behavior, and enforce coding conventions automatically.

Read original(opens in new tab)
API DesignMachine LearningCI/CDPython+4
discord2 min readCurated summary

Cost Attribution in Discord’s API

Discord’s API runs from a shared Python codebase with more than 1,700 endpoints and 700 background tasks across hundreds of Kubernetes deployments. While existing observability tracks performance and reliability, Discord lacked a way to understand hosting costs by product feature or endpoint. Because deployments share code and workers handle multiple features concurrently, the solution was to extend application profiling to allocate deployment costs according to the time spent serving each feature. ## A Large, Continuously Deployed API - Discord operates a unified Python codebase containing: - Over 1,700 API endpoints - Around 700 background tasks - Engineers deploy changes daily to several hundred Kubernetes deployments. - Phased rollouts and instrumentation help monitor: - Latency - Throughput - Error rates - These metrics make it possible to detect regressions affecting users or infrastructure. ## The Missing Cost Dimension - Discord wanted to determine how hosting costs were distributed across product features. - Example questions included: - How much does it cost to send and receive messages? - What does it cost to start a stream or send a Nitro gift? - How do feature costs change over time? - Did a recent code change materially affect a team’s hosting spend? - The goal was to measure costs at both: - Individual endpoint level - Broader feature level, such as chat ## Why Kubernetes Deployment Costs Were Insufficient - Cloud providers can generally report costs by Kubernetes deployment. - However, Discord’s deployments do not map cleanly to product features: - The same codebase runs across all deployments. - Each deployment handles a particular subset of HTTP traffic or background tasks. - Splitting deployments further would make the system impractical to operate. - Discord therefore needed cost attribution without changing its deployment topology. ## Allocating Costs Through Profiling - API worker processes handle multiple tasks concurrently. - A single worker may simultaneously perform work for many different features. - Existing traffic isolation was not detailed enough for feature-level cost analysis. - Discord’s approach was to allocate a deployment’s cost based on the amount of time spent executing code associated with each feature. - By extending its application profiling tools, Discord could track this execution time and use it to estimate feature and endpoint hosting costs. In practice, the profiling-based approach provides a way to analyze infrastructure spending within shared deployments, without requiring separate services or Kubernetes environments for every product feature.

Read original(opens in new tab)
API DesignKubernetesObservabilityPython+2
cloudflare3 min readCurated summary

Agents that remember: introducing Agent Memory

Cloudflare’s Agent Memory is a managed, retrieval-based service designed to give AI agents persistent memory without continuously expanding their context windows. It addresses context rot by extracting useful information from conversations, retaining it across sessions, and retrieving synthesized answers when needed. The service is intended for production agents that run for weeks or months, where fast ingestion, affordable retrieval, and durable knowledge matter more than benchmark performance alone. ## The Challenge of Agent Memory - Larger context windows—even beyond 1 million tokens—do not eliminate context rot; excessive context can reduce model quality. - Aggressive pruning creates the opposite risk: removing information the agent may need later. - Existing memory systems vary widely: - Managed services versus self-hosted frameworks - Raw database or filesystem access versus purpose-built APIs - Full-context approaches versus retrieval-based systems - Benchmarks such as LongMemEval, LoCoMo, and BEAM help compare systems but may encourage overfitting to clean datasets that do not reflect long-running production workloads. ## Cloudflare’s Retrieval-Based Approach - Agent Memory is a managed service with an opinionated API. - It extracts and retrieves relevant information instead of exposing agents directly to a filesystem or database. - This approach is intended to: - Reduce token usage and cost - Improve retrieval performance - Support temporal reasoning, supersession, and instruction following - Keep memory operations out of the agent’s main reasoning context - Cloudflare expects programmatic querying to be useful for specialized edge cases, but not as the default interaction model. ## Memory Profiles and Core Operations Memory is organized into named profiles that can be shared across sessions, agents, and users. - `ingest`: Processes a conversation and extracts memories, typically during context compaction. - `remember`: Stores one important fact explicitly, often through direct model tool use. - `recall`: Runs the full retrieval pipeline and returns a synthesized response. - `list`: Lists stored memories. - `forget`: Removes a specific memory. For example, an agent can ingest a conversation containing a user’s preference for pnpm and dark mode, explicitly remember an operational fact such as an increased API rate limit, and later recall that the user prefers pnpm over npm. ## Integration and Supported Architectures - Agent Memory is available through a binding in Cloudflare Workers. - Agents running outside Workers can use the REST API. - The Cloudflare Agents SDK integrates it with session compaction, memory creation, and retrieval. - It can support: - Individual coding or personal agents - Self-hosted frameworks and managed agent services - Autonomous background agents that must survive restarts - Custom agent harnesses - Shared knowledge between engineers, agents, and tools - Shared profiles can preserve coding conventions, architectural decisions, and other organizational knowledge that might otherwise be lost during context pruning. ## Practical Recommendation Agent Memory is positioned as a default persistent-memory layer for production agents: use ingestion during compaction, explicit remembering for critical facts, and retrieval when the agent needs historical context. Its private beta is particularly aimed at long-running, multi-session workloads where maintaining useful memory is more important than simply fitting more text into the context window.

Read original(opens in new tab)
API DesignReactTypeScriptCloudflare Workers+3
cloudflare3 min readCurated summary

Scaling MCP adoption: Our reference architecture for simpler, safer and cheaper enterprise deployments of MCP

Cloudflare argues that enterprise MCP adoption requires centralized governance rather than individually managed, locally hosted servers. Its reference architecture combines remote MCP servers, Cloudflare Access, MCP server portals, and AI security controls to improve visibility, authentication, policy enforcement, and performance. The company also introduces Code Mode with MCP server portals to reduce the token and context-window costs of exposing large APIs. ## Centralized Remote MCP Servers - MCP separates the AI application from corporate credentials and APIs: - The MCP client connects to the LLM or agent. - The MCP server mediates access to internal resources. - Cloudflare moved away from locally hosted MCP servers because they: - May use unvetted software and versions. - Increase supply-chain and tool-injection risks. - Are difficult for IT and security teams to administer. - A centralized team manages MCP infrastructure through a shared monorepo platform. - Approved teams can create governed MCP servers from templates, inheriting: - Default-deny write controls. - Audit logging. - Automated CI/CD pipelines. - Secrets management. - Servers are deployed remotely on Cloudflare’s developer platform and custom domains, providing centralized usage visibility and global low-latency access. ## Authentication with Cloudflare Access - Public MCP servers, such as documentation and Radar services, can remain openly accessible. - MCP servers connected to private corporate resources require employee authentication. - Cloudflare Access acts as the OAuth provider and identity layer. - It verifies: - Single sign-on. - Multifactor authentication. - IP address, location, and device-certificate context. - Access issues tokens that authorize users to reach protected resources. ## MCP Server Portals for Discovery and Governance - As the number of MCP servers grew, employees needed a central way to discover authorized services. - Users connect their MCP client to a portal, which exposes the internal and third-party MCP servers they are permitted to use. - Portals provide: - Centralized logging. - Consistent policy enforcement. - Data loss prevention controls. - Access policies for users and tools. - Administrators can restrict both portal access and the specific tools exposed by each server. - Finance users might receive only read-only repository tools. - Engineering users on corporate devices might receive read/write capabilities. - Portals support MCP servers hosted on Cloudflare as well as third-party servers. - Cloudflare emphasizes that the relevant security and networking components can run on the same physical machine in its global network, reducing latency and avoiding unnecessary traffic transit. ## Code Mode Reduces MCP Token Costs - The standard MCP design exposes every API operation as a separate tool. - For large platforms with thousands of endpoints, this exhaustive tool list consumes an agent’s context window and increases token costs. - Cloudflare presents Code Mode with MCP server portals as a way to address this scaling problem. - The provided article excerpt ends while introducing Cloudflare’s earlier use of server-side Code Mode for exposing large numbers of API endpoints. Cloudflare’s approach recommends treating MCP as enterprise infrastructure: centrally deployed, authenticated, discoverable, policy-controlled, and monitored. Organizations adopting MCP at scale should avoid unmanaged local servers and provide reusable platforms that make secure deployment the default.

Read original(opens in new tab)
API DesignCI/CDModel Context ProtocolCloudflare+4
cloudflare3 min readCurated summary

Sandboxing AI agents, 100x faster

Cloudflare argues that AI-generated code needs secure execution, but traditional containers are too slow, memory-intensive, and difficult to scale for consumer-scale agents. Its Dynamic Worker Loader uses lightweight V8 isolates to create disposable, isolated sandboxes in milliseconds, with controlled access to APIs and no internet connectivity. The result is a sandbox roughly 100 times faster and substantially more memory-efficient than containers, provided agents can write JavaScript. ## Why Containers Fall Short - AI-generated code cannot safely run directly through `eval()`, since prompts could cause the model to introduce vulnerabilities. - Containers provide isolation but typically: - Take hundreds of milliseconds to start - Consume hundreds of megabytes of memory - Require warm instances to reduce latency - May encourage unsafe container reuse - These limitations make containers poorly suited to running a fresh sandbox for every request or user agent. ## Dynamic Worker Loader - Cloudflare’s Dynamic Worker Loader lets a Worker instantiate another Worker dynamically from runtime-provided code. - The host can: - Supply generated JavaScript modules - Expose selected APIs through RPC stubs - Disable or intercept outbound internet access - Invoke methods exported by the dynamically loaded Worker - The feature is in open beta for paid Workers users. ## Faster, Smaller Isolates - Dynamic Workers use V8 isolates, the same sandboxing technology underlying Cloudflare Workers. - Isolates: - Start in a few milliseconds - Use only a few megabytes of memory - Are approximately 100 times faster and 10–100 times more memory-efficient than typical containers - A new isolate can be created for one request and discarded afterward without maintaining a pool of warm sandboxes. ## Scalability and Latency - Dynamic Worker Loader has no container-style global concurrency or sandbox-creation limits. - It relies on the infrastructure that already scales Cloudflare Workers to millions of requests per second. - Each request could theoretically load its own isolated sandbox, even at very high concurrency. - Dynamic Workers commonly run on the same machine or thread as their parent Worker, avoiding network round trips and warm-sandbox lookup delays. - They are available across Cloudflare’s global network. ## JavaScript as the Agent Runtime - The main limitation is that agent-generated code should generally be JavaScript. - Workers also support Python and WebAssembly, but JavaScript is faster to load for short-lived snippets. - Cloudflare argues this is acceptable because: - LLMs can generate major programming languages - JavaScript has extensive training data - JavaScript was designed for web-based sandboxed execution ## TypeScript APIs for Agent Tools - Agents still need access to external capabilities such as chat systems and APIs. - TypeScript interfaces provide a concise way to describe these programming APIs. - Compared with MCP’s flat tool schemas or verbose OpenAPI specifications, TypeScript can express: - Methods and parameters - Return types and promises - Objects such as messages - Subscription and disposal behavior - This gives agents precise API knowledge with fewer tokens and lets them write direct code rather than issuing numerous tool calls. Dynamic Worker Loader is presented as a practical foundation for secure, disposable AI-agent execution: use V8 isolates for low-latency sandboxing, expose only narrowly defined TypeScript/RPC capabilities, and block network access unless explicitly required.

Read original(opens in new tab)
API DesignLarge Language ModelsTypeScriptJavaScript+4
line4 min readCurated summary

Internalizing without specifications: Proving equivalence through validation logic

The article describes how LINE Plus safely internalized black-box e-commerce systems without specifications or source code. The team built an automated equivalence-testing loop using Kafka, CDC, OpenSearch, and ksqlDB to compare legacy and new behavior at massive scale. By repeatedly identifying differences, fixing logic, and rechecking results, they could reduce discrepancies toward zero while also measuring performance and protecting production stability. ## Domain: Products, Catalogs, and Data Ingestion - **Products** are individual seller-listed items, potentially with different prices and shipping conditions. - **Catalogs** group products representing the same model or product type and provide derived value such as: - Real-time lowest prices - Unit-price metrics such as price per 100 ml - **Ingestion** receives large product files from sellers, validates and transforms them into internal formats, and updates product and catalog data. - Because the platform contains tens of millions of catalogs and hundreds of millions of products, small logic differences can affect the entire service. ## The Verification Loop - The goal was not merely to find errors, but to help developers understand and correct them quickly. - Inputs had to be identical for both systems, such as: - The same IDs - The same time-based snapshot - The same product files - Outputs were compared according to system type: - API response objects - Database update values - Final registered product data - The general loop consisted of: - **Trigger:** Database changes, developer requests, or file arrivals - **Execution:** Send identical inputs to legacy and new systems - **Comparison:** Apply logic suited to reads, updates, or end-to-end flows - **Processing:** Store detailed differences and produce real-time statistics - **Action:** Developers inspect dashboards or Slack alerts, fix the implementation, and repeat ## Query Logic Verification - The catalog API was difficult to reproduce because it had over 100 response fields, complex filters, undocumented defaults, and unknown sorting behavior. - CDC streamed database binary-log changes into Kafka, allowing verification to begin from many real catalog states. - The verifier made dual API calls and compared legacy and new responses field by field. - Responses were converted into `Map<String, Object>` structures and compared recursively, avoiding the need to model every response class. - If values differed only because list ordering varied, the verifier sorted serialized values and performed a second comparison. - This helped distinguish real implementation defects from harmless ordering differences. - Kafka isolated verification traffic from production services while handling large event volumes. - Difference events were written to Kafka topics and indexed in OpenSearch for detailed investigation. - ksqlDB aggregated streaming discrepancies and sent Slack notifications when abnormal patterns appeared. - Rate limiting restricted repeated errors, such as those from the same field, to a manageable sample per minute. - Because both APIs were called in parallel, the same pipeline also measured and compared their response times. ## Update Logic Verification - The second case involved recalculating catalog statistics whenever product or catalog data changed. - Unlike read verification, this process tested state transitions and asynchronous updates. - When CDC detected a relevant change: - The new statistics logic calculated an expected result. - The verifier compared it with the result actually written by the legacy logic. - Recursive Map-based comparison checked deeply nested statistics fields. - To avoid wasting resources, verification was triggered only for updates related to the catalog-statistics module. ## Handling Asynchronous Lag - Kafka-based processing caused timing gaps: the verifier could read the database before the legacy update had completed. - The team introduced an **N-attempt retry queue**: - Temporarily inconsistent events were requeued. - Only differences that remained after several retries were treated as genuine defects. - The verifier remained a separate process rather than being embedded in the production statistics stream. - This avoided adding load or latency to the existing processing pipeline while preserving independent verification. ## ETL Batch Verification for Missing Triggers - Real-time comparison could detect incorrect results, but not cases where an update should have happened and never occurred. - During refactoring, a complex combination of product and catalog field changes contained a missing trigger condition. - As a result, some statistics remained stale without generating any comparison event. - To detect these silent omissions, the team designed a separate batch-verification process using ETL data alongside the real-time stream checks. The practical recommendation is to treat system internalization as an evidence-building process: define identical inputs and observable outputs, compare legacy and replacement systems continuously, isolate verification through event streams, and supplement real-time checks with batch validation for silent or missing updates.

Read original(opens in new tab)
API DesignDatabase DesignPostgreSQLApache Kafka+4
line3 min readCurated summary

Integration of LINE App’s Multi-party Chat Features

LINE is consolidating its two multi-person chat types—temporary “Rooms” and long-term “Groups”—into a single Group Chat model. The change aims to simplify the user experience, make all chat features available everywhere, and reduce duplicated server and client resources. A gradual migration strategy is being used to avoid disruption. ## Two Original Chat Models - **Rooms** were designed for temporary conversations: - No room name was required. - Invited friends joined immediately without approval. - Features such as albums and notes were unavailable. - **Groups** were designed for long-term communities: - They had names and supported features such as group albums and notes. - Invitees had to accept or reject invitations before joining. - Users often created Rooms without realizing their limitations, then created a new Group later when they needed additional features. ## Reasons for Unification - Users found the distinction between Rooms and Groups difficult to understand. - Existing conversations could not be converted from Rooms into Groups, forcing users to abandon their conversation history. - Users frequently created multiple chats with the same members, causing: - Cluttered conversation lists. - Unnecessary data accumulation on servers. - Increased client and server resource usage. - The unified model standardizes behavior and features while retaining flexibility in how invitations work. ## Migrating Groups to Group Chats - LINE introduced new Group Chat APIs and used **dual reads** to maintain compatibility with existing Group APIs and storage. - The migration proceeded gradually: 1. The new API initially read Group data through a routing layer. 2. The number of Group Chats was progressively increased. 3. Eventually, only Group Chats were created. - Batch processing migrated all existing Group data. - After migration, LINE stopped dual reads and relied exclusively on the Group Chat model. ## Differences Between Rooms and Groups ### Invitation Mechanisms - Groups required invitees to explicitly accept or reject an invitation. - Rooms added people immediately when they were invited. - The unified creation flow lets users choose whether invitees should join immediately or confirm participation first. ### Feature Availability - Rooms lacked many Group features because they were intended to be temporary. - The new model is based on the Group architecture, so all newly created conversations support the full feature set, including future features. ## Improving Conversation Discovery - Users often created a new chat with the same participants instead of finding an older, inactive conversation in a long chat list. - The new creation workflow displays a hint when an equivalent existing conversation is found. - Users can then return to the existing conversation, reducing duplicate rooms and improving navigation. ## Migration Plans for Existing Rooms - Conversations created in current LINE versions are already Group Chats. - Groups created with older app versions are being converted server-side. - The remaining objective is to migrate existing Rooms so their participants can use the complete set of Group Chat features. The project is a long-term effort designed to minimize disruption while improving consistency and efficiency. Duplicate conversations with identical participants fell from 15% for Rooms to 0.78% for invitation-free Group Chats, demonstrating the practical impact of the consolidation.

Read original(opens in new tab)
API DesignUser ExperienceReal Time CommunicationData Migration+2
meta2 min readCurated summary

Patch Me If You Can: AI Codemods for Secure-by-Default Android Apps

Updating security-sensitive APIs across a massive mobile codebase is difficult because vulnerable patterns may appear across hundreds of call sites and millions of lines of code. Meta’s Product Security team addresses this through secure-by-default Android frameworks and generative AI that automates migrations to those frameworks. The approach enables security patches to be proposed, validated, and submitted with minimal effort from code owners. ## Secure-by-Default Mobile Frameworks - Meta wraps potentially unsafe Android OS APIs in frameworks designed to make secure implementations the easiest option. - Developers are guided toward safer behavior by default rather than being expected to recognize and avoid every security risk manually. - This strategy helps prevent a single vulnerability class from recurring across Meta’s many mobile applications. ## AI-Assisted Code Migration - Generative AI is used to migrate existing code from unsafe APIs to the new secure frameworks. - The system operates across millions of lines of code and numerous call sites. - It can propose security changes, validate them, and submit patches for review. - This reduces the manual work required from the engineers responsible for each application or codebase. ## Security at Massive Scale - Meta’s scale—thousands of engineers, multiple apps, and billions of users—makes conventional security updates difficult to coordinate. - The initiative combines framework design, automation, and engineering ownership to reduce friction while maintaining validation. - The accompanying Meta Tech Podcast episode features Product Security engineers Alex and Tanu discussing the challenges and lessons from this effort. Meta’s approach demonstrates that large-scale mobile security improvements are most practical when safer APIs and automated migration tools work together, allowing secure changes to spread broadly without requiring every engineer to perform the migration manually.

Read original(opens in new tab)
API DesignGenerative AIAutomationMobile Security+3
datadog3 min readCurated summary

Designing MCP tools for agents: Lessons from building Datadog's MCP server

Datadog’s initial MCP server simply exposed existing APIs, but real-world agent use revealed major problems with context limits, inaccurate trend analysis, and tool overload. The team redesigned its tools around token efficiency, query-based analysis, and a smaller, more deliberate tool surface. These changes improved both answer quality and cost, though emerging agent features may eventually reduce the need for some optimizations. ## Context Efficiency Matters - Observability results can be extremely large: a log record may range from roughly 100 characters to 1 MB. - CSV or TSV is more token-efficient than JSON for tabular data, often using about half as many tokens per record. - YAML can reduce token usage for nested data by around 20% compared with JSON. - Removing rarely used fields from default responses, while allowing agents to request them when needed, further reduces output size. - Combined formatting and field-trimming improvements allowed some tools to return approximately five times more records within the same token budget. - Pagination by record count is unreliable when records vary greatly in size. Datadog instead paginates by token budget and returns a cursor when the limit is reached. - Tools such as Cursor and Claude Code increasingly write long results to disk, which could make response-format efficiency less important in the future. ## Let Agents Query Data - Retrieval-only tools forced agents to infer trends from incomplete samples, such as guessing which services generated the most errors. - Agents sometimes repeatedly fetched logs to compensate, wasting tokens and producing unreliable answers. - SQL lets agents aggregate and filter data directly: ```sql SELECT service, COUNT(*) AS error_count FROM logs WHERE status = 'error' GROUP BY service ORDER BY error_count DESC LIMIT 10 ``` - Agents can select only necessary fields, limit row counts, and calculate aggregates without loading raw data. - SQL improved correctness and reduced costs; some evaluation scenarios became about 40% cheaper. - Supporting SQL at Datadog’s scale required significant infrastructure work because traditional relational databases were insufficient. ## Tools Are Not Free - Exposing every API endpoint as a separate tool increases tool-selection errors and consumes context through tool descriptions. - Flexible tools can support multiple related workflows through carefully designed schemas, reducing the total tool count. - Toolsets provide a core collection by default while allowing users to opt into specialized capabilities, though users must anticipate their needs. - Layered tools can first explain how to accomplish a task and then execute it, keeping specialized functionality out of the initial context. - Layering introduces additional tool calls and therefore increases latency. - Improving agent context management, including tool search and dynamically loaded skills, may reduce the need for aggressive tool minimization over time. The practical recommendation is to design MCP tools for how agents actually reason: minimize and control output size, provide query and aggregation capabilities instead of raw retrieval alone, and expose a focused set of flexible tools rather than mirroring every API endpoint.

Read original(opens in new tab)
API DesignDatabase DesignDatadogObservability+4
cloudflare3 min readCurated summary

We deserve a better streams API for JavaScript

Web Streams established a cross-runtime standard for handling streaming data, but their design reflects constraints from 2014–2016 rather than modern JavaScript practices. James M. Snell argues that the API’s reader, lock, and controller machinery creates unnecessary complexity and performance costs. He presents an alternative based on JavaScript language primitives that reportedly runs 2× to 120× faster across browsers and major runtimes. ## Historical Design Constraints - The WHATWG Streams Standard aimed to provide portable APIs for creating, composing, and consuming streams. - It was adopted by browsers, Cloudflare Workers, Node.js, Deno, Bun, and APIs such as `fetch()`. - The design predates JavaScript async iteration, which was standardized in ES2018. - Because `for await...of` did not yet exist, Web Streams introduced a separate reader/writer acquisition model. ## Excessive Ceremony for Basic Reads - Reading a stream to completion traditionally requires: - Calling `stream.getReader()`. - Repeatedly awaiting `reader.read()`. - Checking `{ value, done }` on every iteration. - Releasing the reader lock in a `finally` block. - These steps are API choices rather than inherent requirements of streaming. - Modern async iteration reduces the same operation to: ```js for await (const chunk of stream) { chunks.push(chunk); } ``` - However, async iteration was added after the original design, so it does not eliminate the underlying reader, lock, and controller complexity. - Advanced features such as BYOB reads still require developers to use the lower-level APIs. ## Problems with Manual Locking - Calling `getReader()` places an exclusive lock on the stream. - While locked, other code cannot read, pipe, or cancel the stream directly. - Forgetting `reader.releaseLock()` can permanently prevent later consumers from using the stream. - The `locked` property indicates that a lock exists, but not who owns it, why it exists, or whether the reader remains usable. - Internal operations such as piping also acquire locks, which can make stream behavior surprising. - Lock-release behavior with pending reads was historically unclear and varied between implementations before being clarified by the specification. - Async iterables improve the user experience by handling reader and lock management automatically, but the underlying model remains complex. ## Proposed Direction - The post argues that Web Streams’ limitations are fundamental design consequences, not isolated bugs easily fixed through incremental changes. - A better API should be built around modern JavaScript primitives, especially async iteration. - The author’s alternative reportedly achieves between 2× and 120× the performance of Web Streams across Cloudflare Workers, Node.js, Deno, Bun, and major browsers. - The claimed gains come from different architectural choices rather than narrowly optimized implementations. A more modern streams API should make common operations natural, avoid exposing fragile manual lock management, and use JavaScript’s native asynchronous iteration model from the start.

Read original(opens in new tab)
API DesignJavaScriptPerformance OptimizationNode.Js+3
toss3 min readCurated summary

Easy-to-use Toss Front SDK

The post argues that an SDK’s stability depends not only on its internal implementation but also on how safely users can interact with it. Low-level APIs may expose every operation clearly, yet still allow human errors such as missing event handlers or cleanup. The recommended solution is an intent-driven Facade interface that simplifies common workflows, prevents misuse, and still provides low-level escape hatches for advanced cases. ## Designing an SDK That Is Easy to Use - Toss Place develops an external SDK for Toss Front payment terminals. - The SDK allows third-party developers to build plugin apps that integrate with Toss services and run on the terminal. - A simple-looking server API might require users to: - Open a server. - Register connection, message, and error handlers. - Remove handlers. - Close the server. - This approach exposes implicit responsibilities to SDK users: - A message callback might never be registered after a connection. - Handlers might not be removed before shutdown. - Improper cleanup can cause memory leaks and operational issues. - Therefore, third-party implementation mistakes can directly affect platform reliability. - A safer interface hides unnecessary internal steps: ```ts const server = await sdk.start({ onConnection, onMessage }); await server.stop(); ``` ## Facade as an Intent-Driven Interface - The Facade pattern is commonly described as wrapping a complex subsystem with a simpler interface. - In SDK design, its deeper purpose is to reorganize complexity around user intent rather than merely hide functionality. - Users should express goals such as: - “Start a server” - “Upload a file” - “Request a payment” - Internal concerns—including authentication, retries, state management, listener registration, and cleanup—should be handled by the SDK. - AWS CDK illustrates this distinction: - **L1 constructs** closely represent raw CloudFormation resources and provide fine-grained control. - **L2 constructs** provide intent-based APIs, such as creating a versioned S3 bucket with `versioned: true`, while handling the underlying configuration automatically. - The goal of a Facade is to reduce cognitive load and coupling, not simply to conceal every lower-level capability. ## Combining High-Level and Low-Level APIs - A well-designed SDK should provide both abstraction levels: - **High-level Facade:** Handles the roughly 80% of common use cases through complete workflows. - **Low-level APIs:** Serve as escape hatches for the roughly 20% of specialized cases requiring precise control. - In the example: - The Facade’s `start()` method opens the server, registers listeners, coordinates connections, and returns a unified server handle. - Low-level APIs separately expose operations such as `open`, `close`, `send`, `disconnect`, and event listeners. - This layered design improves immediate developer experience while preserving long-term compatibility and extensibility. ## Trade-offs and Escape Hatches - Higher-level abstractions inevitably reduce some flexibility. - Specialized requirements—such as keeping one connection while closing others—may not fit the Facade workflow. - As orchestration becomes more sophisticated, the SDK maintainers inherit additional implementation and maintenance costs. - Low-level escape hatches are therefore essential: users should be able to bypass the Facade when they need detailed control. ## Practical Recommendation Design SDK APIs around user intent and automate error-prone lifecycle management wherever possible. Offer a concise Facade for common workflows, but retain well-defined low-level interfaces so advanced users are not blocked by the abstraction.

Read original(opens in new tab)
API DesignSoftware ArchitectureInterface DesignFrontend+4
cloudflare3 min readCurated summary

Cloudflare outage on February 20, 2026

Cloudflare suffered a 6-hour, 7-minute outage on February 20, 2026, after a software change unintentionally withdrew Internet routes for some Bring Your Own IP (BYOIP) customers. The incident was not related to a cyberattack; a buggy automated cleanup task altered customer prefix and service configurations. Cloudflare reverted the change, restored affected prefixes, and is revising its Addressing API workflows to reduce production risk. ## Customer Impact - Approximately 1,100 of Cloudflare’s 6,500 advertised prefixes were withdrawn between 17:56 and 18:46 UTC. - This affected about 25% of the 4,306 BYOIP prefixes advertised globally. - Impacted applications became unreachable from the Internet and experienced connection failures and timeouts. - Customers initially encountered BGP Path Hunting, where networks repeatedly searched for a route until connections timed out. - The `one.one.one.one` website returned HTTP 403 errors and an “Edge IP Restricted” message. - DNS resolution through the 1.1.1.1 resolver, including DNS over HTTPS, was not affected. - The incident did not affect every BYOIP customer because the configuration change was applied incrementally and was reverted before reaching everyone. ## Recovery Efforts - Engineers detected the issue through failures involving `one.one.one.one` and reverted the change. - Cloudflare published dashboard guidance at 19:19 UTC, allowing many customers to re-advertise their prefixes themselves. - Around 800 prefixes were restored by approximately 20:20 UTC. - About 300 prefixes could not be restored through the dashboard because their service configurations had been removed from edge servers. - Engineers manually restored those remaining prefixes at 23:03 UTC. - Some customers continued to experience latency and failures while addressing configuration state propagated back to the edge. ## The Addressing API - Cloudflare’s Addressing API is the authoritative dataset for IP addresses present on its network. - Changes to the API drive workflows that propagate address and routing updates across Cloudflare’s edge. - The normal process is: - Customers request advertisement or withdrawal through the Addressing API or BGP Control. - The API instructs machines to change prefix advertisements. - Routers update BGP after enough machines receive the change. - Customers bind Cloudflare products to their BYOIP ranges. - Because the API is closely connected to production systems, manual changes are risky. - Cloudflare’s “Code Orange: Fail Small” initiative aims to replace manual Addressing API operations with safer, automated, health-checked workflows. ## Root Cause: Faulty BYOIP Cleanup Automation - The failed change automated the removal of prefixes from BYOIP, a task that had previously been performed manually. - A recurring cleanup sub-task searched for BYOIP prefixes marked for deletion and removed them. - The cleanup task issued the API request: ```go /v1/prefixes?pending_delete ``` - The request contained a bug in how the API query was interpreted. - As a result, the cleanup process unintentionally withdrew customer prefixes and removed related service configurations from some edge servers. - The incident lasted much longer than the initial withdrawal because restoring both advertisements and edge configuration state required extensive automated and manual recovery. Cloudflare’s main corrective direction is to make Addressing API changes safer through incremental, health-mediated deployment, stronger safeguards around automated deletion, and elimination of risky manual production workflows.

Read original(opens in new tab)
API DesignGoDnsBgp+3
spotify3 min readCurated summary

Our Multi-Agent Architecture for Smarter Advertising | Spotify Engineering

The post argues that fragmented advertising workflows, not backend infrastructure, are the core problem. Although buying channels share services and data, their planning and optimization logic is repeatedly reimplemented across channels and surfaces, causing drift and technical debt. The proposed solution is a shared agentic decision layer that interprets advertiser goals, orchestrates existing Ads APIs, and applies consistent reasoning across products. ## Fragmented Workflows Across a Shared Backend - Direct, Self-Serve, and Programmatic buying use largely consolidated infrastructure but retain different workflows and decision logic. - Spotify Ads Manager, Salesforce, Slack, and internal tools contain overlapping automation. - Budget allocation, inventory selection, reach, efficiency, and STR decisions are repeatedly implemented in different places. - Incremental workflow changes therefore create duplicated maintenance work and inconsistent behavior. ## Why Conventional Workflow Services Fall Short - Hard-coded state machines and REST services are poorly suited to combinatorial planning tasks. - Campaign planning depends on: - User and advertiser characteristics - Available inventory and audiences - Business priorities - Forecasts, performance, and optimization goals - A workflow optimized for one channel or “happy path” will not adapt well as requirements change. - Improvements to decision logic must be replicated across every product surface, increasing the risk of divergence. ## The Missing Intent Layer - Existing systems can perform individual actions such as creating line items, running forecasts, and retrieving insights. - They do not consistently translate high-level objectives into: - A sequence of tool calls - Explicit tradeoffs - Validation and safety checks - An objective such as maximizing reach in Brazil while protecting video inventory and meeting STR requires coordinated reasoning across multiple capabilities. ## A Modular Agentic Architecture - Campaign planning and management are modeled as cooperating specialized agents. - Agents use shared signals, including: - Inventory - Audiences - STR - Quality and risk - Historical performance - They jointly optimize advertiser goals and Spotify’s business constraints. - Existing Ads services become tools that agents orchestrate, rather than capabilities being rebuilt in each workflow. - A long-running orchestration layer delegates tasks while agents share context and evaluation logic. - The same decision engine can support every buying channel and surface. ## Engineering Implications - APIs need to be designed as agent tools, rather than only as CRUD interfaces. - Testing must include behavioral evaluation in addition to unit and integration tests. - Observability should explain what an agent decided and why, not merely track latency and errors. - Safety requires guardrails for semi-autonomous decisions, beyond ordinary input validation. - The approach avoids both duplicated deterministic workflows and a brittle, centralized rules engine for probabilistic, ML-heavy advertising logic. The overall recommendation is to centralize campaign decision-making in a reusable agentic platform while keeping existing services as specialized tools. This should reduce duplicated workflow logic, make improvements consistent across products, and allow advertising workflows to evolve without repeatedly rebuilding them.

Read original(opens in new tab)
API DesignAI AgentsMachine LearningMulti Agent Systems+3
github3 min readCurated summary

How to maximize GitHub Copilot&#8217;s agentic capabilities

GitHub’s guide presents Copilot’s agent mode as a partner for architecture, refactoring, and coordinated multi-file changes—not a replacement for engineering judgment. It argues that Copilot is most useful when developers first define system boundaries, assess cross-cutting effects, and then use the agent to implement and document changes. The examples build toward extending a modular Notes Service with tagging, validation refactoring, migrations, and test modernization. ## Preparing for Agentic Work - The guide assumes: - Copilot agent mode is enabled. - Familiarity with service-layer architectures. - Access to a GitHub Skills exercise template. - Willingness to review and challenge Copilot’s proposals. - Earlier-career engineers can use the exercises to learn how senior engineers evaluate architecture and risk. ## Using Copilot for System Design - Developers should begin by identifying boundaries between: - Domain logic - Data access - Interfaces - Module interactions - Copilot can analyze a service for: - Poor module boundaries and tight coupling - Async and transaction risks - Duplicated responsibilities - Testability and observability problems - It can also compare architectural approaches, such as hexagonal and layered architecture, and explain tradeoffs based on the codebase’s constraints. ## Building Modular Services - Once the architecture is understood, Copilot can coordinate implementation across: - Domain modules - Controllers - Repository abstractions - Suggested practices include dependency inversion and documenting module contracts and assumptions. - Copilot may generate interfaces, repository abstractions, controller logic, and Markdown documentation, reducing boilerplate while exposing developers to established design patterns. ## Adding a Tagging Subsystem - A seemingly simple tagging feature requires decisions about: - Embedded tags versus normalized or many-to-many data models - Search indexing, filtering, and relevance - Whether tags are API resources or internal details - Validation and invariant boundaries - Additive migrations, compatibility, and rollback - Copilot can first map the feature’s architectural impact, including migration requirements, caching, indexing, regressions, tests, and external consumers. - Implementation may span the domain model, database schema, repositories, controllers, tests, and documentation. - The example uses a `tags` column with a default empty array and adds `Tag[]` to the note model, illustrating how agent mode maintains consistency across files. ## Safe Schema Changes - The guide emphasizes that migration design involves more than writing SQL. - A production-ready change should be: - Backward compatible - Reversible - Safe under load - Transparent to dependent systems - Copilot can assist with reasoning about rollout strategies, but engineers must inspect its recommendations and validate them against operational constraints. The practical recommendation is to use Copilot agent mode as an architecture-aware collaborator: ask it to analyze and compare options first, then implement changes across the system while requiring explicit assumptions, diffs, tests, and documentation.

Read original(opens in new tab)
API DesignTypeScriptGithub CopilotSoftware Architecture+3