kakao·Aug 6, 2026·4 min readCurated summary

Experience Building and Operating a Personalized Airflow Testing Environment

Kakao’s data engineering team built AirZone to make Airflow DAG testing faster, easier, and safer across an ecosystem containing thousands of DAGs and multiple Hadoop clusters. Existing approaches required local setup, repeated Git synchronization, file copying, VPN access, or risky testing on production Airflow. AirZone instead creates an isolated, production-like Airflow environment for each pull request, managed through GitHub comments and Kubernetes automation. ## Limitations of Existing Testing Methods - **Local Airflow** - Requires configuring Airflow, Hadoop authentication, connections, and Docker locally. - Has a high initial setup cost and may differ from production. - **Development Airflow** - Requires committing and pushing every code change. - Git submodule updates and DAG parsing introduce long feedback delays. - **Test Airflow with SSH** - Allows files to be copied directly into a container. - Still requires copying files after every edit. - Access to production Hadoop requires connecting to a production VPN. - **Testing on production Airflow** - Heavy test DAGs consume shared scheduler, worker, and node resources. - A resource-intensive test can delay or interrupt unrelated projects. - Per-user isolation is therefore essential. ## AirZone Requirements - Provide an Airflow environment without requiring users to understand Kubernetes or Helm. - Allow code editing through a browser using Jupyter Notebook. - Execute DAGs against Hadoop and authentication mechanisms similar to production. - Create an independent environment for each pull request. - Prevent one user’s tests from affecting other workflows. ## PR-Based, Isolated Architecture - GitHub pull request comments serve as the user interface. - Users can create or delete an environment directly from a PR. - The resulting environment link is posted back to the PR. - Each PR receives a dedicated Kubernetes namespace based on the repository and PR number. - Airflow web server, scheduler, PostgreSQL, Jupyter, DAG volumes, and logs are isolated. - Multiple PRs can be tested simultaneously. - Cleanup is straightforward because the namespace defines the environment boundary. - A dedicated AirZone Helm chart packages the complete test environment. - Production-only components such as PGBouncer and external database connections are omitted where unnecessary. - Airflow, PostgreSQL, DAG storage, Jupyter, authentication, TLS, and logging are deployed together. ## Separating Requests from Deployment - `airzone-api` only validates requests: - Confirms that the PR exists and is open. - Checks branch information. - Prevents duplicate namespaces. - Kubernetes Jobs perform the long-running work: - Install the Helm release. - Run health checks. - Handle creation and deletion independently from the API process. - Job names include the operation and namespace, such as: - `create-airzone-{namespace}` - `delete-airzone-{namespace}` - Failed Jobs can be removed and recreated for retries. - Independent Job logs and status make deployment failures easier to diagnose. - A daily CronJob removes environments that remain after their PRs are closed. ## Building the Airflow Environment Each Helm deployment includes the components needed for a realistic test environment: - **Git integration:** Synchronizes the PR’s head repository and branch. - **DAG PVC:** Lets the scheduler and Jupyter use the same working directory. - **Airflow configuration:** Uses KubernetesExecutor and test-specific DAG scanning, logging, and Hadoop settings. - **Authentication:** Injects user and shared principals, keytabs, Jupyter tokens, and TLS certificates. - **Infrastructure placement:** Selects suitable node groups and a storage class in the same region. - **Centralized logging:** Connects Airflow logs to Elasticsearch and Kibana. - **Hadoop execution:** Existing infrastructure runs Hadoop tasks in dedicated pods using custom Hadoop images, Kerberos initialization, Spark, and Hive. ## Notifications and Security - KakaoWork sends: - An initial notification when a request is received. - A completion notification after deployment. - Operational error alerts. - Sensitive information, including Jupyter and Kubernetes namespace tokens, is not posted in public PR comments. - Tokens are delivered through KakaoWork instead, keeping authentication data separate from the broader PR audience. AirZone’s main recommendation is to make testing a disposable, reproducible environment tied to the pull request itself. By combining per-PR Kubernetes namespaces, Helm-based deployment, asynchronous Jobs, production-like Hadoop access, and automatic cleanup, teams can test DAGs quickly without burdening shared Airflow or production resources.

Read original(opens in new tab)

PostgreSQLKubernetesApache SparkHelm+3

aws·Jul 13, 2026·4 min readCurated summary

AWS Weekly Roundup: AWS Builder Center at 1 year, Network Scanning in Security Hub, Loom for AWS, and more (July 13, 2026) | Amazon Web Services

AWS’s weekly roundup highlights the first anniversary of AWS Builder Center and its expansion into a broader learning and experimentation ecosystem. It also introduces major security, AI, container, database, and developer-tool updates, including internet reachability scanning, Azure support in Security Hub, Hugging Face integration with SageMaker, lower GPU management fees, and Aurora DSQL change data capture. The overall direction is toward more integrated, governed, and accessible cloud development workflows. ## AWS Builder Center Turns One - Launched on July 9, 2025, Builder Center has expanded from a community hub into an ecosystem with: - Sandbox environments - Workshops and the Builders’ Library - Community Spaces, profiles, badges, and streaks - AWS regional capability listings - Saved items, article series, availability notifications, and GitHub/Amazon sign-in - In its first year: - 5,548 authors published 6,448 articles. - Articles received more than 10.4 million page views. - Builders earned 99,226 badges. - The community submitted 565 feature wishes, with 10 already delivered. - The most popular articles covered MCP and Strands Agents, Linux migration with Kiro, and multimodal AI for neurological screening. ## Free AWS Sandbox Environments - Builder Center now offers free, pre-provisioned AWS accounts for workshop exercises. - Each sandbox: - Remains active for eight hours. - Automatically removes the account and resources afterward. - Requires no personal AWS account, credit card, or manual cleanup. - Users can have one active sandbox and request one sandbox per week. ## Security Hub Adds Network and Azure Coverage - Security Hub Network Scanning probes AWS and Azure environments from the public internet. - It identifies: - Public IP addresses, virtual machines, and load balancers. - Reachable ports. - Services operating behind those ports. - Each reachable port produces a finding, which Security Hub Exposures correlates with other configuration and security findings. - Existing customers can enable the feature by account, Region, or organization policy; it is enabled by default for new customers. - Network Scanning is included with Security Hub Essentials at no extra cost. - Security Hub also now evaluates Azure VMs, container images, Function Apps, and identities for vulnerabilities, misconfigurations, and internet exposure. ## SageMaker Studio and Hugging Face Integration - Users can deploy or customize supported Hugging Face models in SageMaker Studio with one click. - New customers receive automatically configured Studio environments for: - Fine-tuning and reinforcement learning with custom reward functions. - Model evaluation. - Deployment to SageMaker or Bedrock endpoints. - Verified customers receive default GPU access to G5, G6, and G4dn instances, while quota usage is visible in Studio. ## Lower GPU Management Fees - Starting July 1, 2026, EKS Auto Mode and ECS Managed Instances reduced accelerated-instance management fees: - G-series: 35% reduction. - P-series and Trainium: 60% reduction. - Existing customers receive the reductions automatically. - EKS adds accelerator-aware node repair and parallel image pulling. - ECS provides GPU metrics through CloudWatch Container Insights and monitors GPU hardware health. ## Aurora DSQL Change Data Capture - Aurora DSQL CDC is generally available and streams insert, update, and delete events to Kinesis Data Streams. - The events can synchronize microservices, invoke Lambda functions, or feed S3, Redshift, and OpenSearch through Firehose. - CDC is designed to avoid database performance impact and requires no infrastructure management. ## Tools for Governed AI Agents - Loom for AWS is an open-source platform for deploying and managing Strands Agents on Bedrock AgentCore Runtime. - It includes: - Identity-provider integration and RBAC/ABAC. - Lifecycle management for agents, memory, MCP servers, and agent-to-agent connections. - Automated tagging for cost attribution. - Agent Registry integration and human approval for sensitive actions. - The Claude apps gateway provides centralized access, policy, and spending controls for Claude Code and Claude Desktop. - It supports OIDC identity providers, per-user and per-group spending caps, managed request settings, and routing through Amazon Bedrock or Claude Platform on AWS. - AWS MCP Server now supports browser-based OAuth using AWS Console or CLI credentials, including IAM federation and IAM Identity Center, with short-lived tokens and automatic refresh. AWS’s updates favor managed services that reduce operational work while strengthening security and governance. Teams should evaluate Builder Center sandboxes for training, enable Security Hub’s broader scanning where appropriate, and consider the new AI-agent controls and managed GPU options for production workloads.

Read original(opens in new tab)

PostgreSQLKubernetesModel Context ProtocolAWS+4

line·Jun 25, 2026·4 min readCurated summary

In the AI Era, Development Ability Is Determined by Verification Skills: Strategies for Rapid Validation and Local Environment Setup Learned While Developing the Flava API Gateway

AI coding agents iterate quickly, but their output can be inconsistent, make incorrect design decisions, or generate code that does not compile. Because CI runs, environment provisioning, and human review are slower, the article argues that reliable agent-assisted development requires three practices: spec-driven development, automated verification, and fast, self-contained local environments. ## Flava API Gateway and the Development Challenge - Flava API Gateway is part of LY Corporation’s private Flava cloud. - It provides a multi-tenant RESTful control-plane API for creating, deploying, and monitoring web APIs. - Kong serves as the data plane. - The team adopted agent-based coding while building the product and focused on preserving software reliability without sacrificing AI-driven speed. ## Spec-Driven Development The team found that agents became more unpredictable when implementation began before the design was settled. They use explicit specifications to reduce ambiguity and constrain implementation decisions. - OpenAPI is written before code to define the control-plane API. - Features are divided into smaller units and implemented with OpenSpec. - Specifications serve both as implementation guidance and as a standard for detecting deviations. ### Managing OpenAPI with Nickel - Raw OpenAPI YAML is repetitive and difficult to maintain manually. - Nickel is used to describe API resources declaratively and generate complete CRUD specifications. - A resource definition can specify: - Description and parent resource - Whether updates are allowed - Automatic timestamps - Property schemas - Required fields - Sorting and filtering behavior - The generator produces consistent endpoints such as `listPaths`, `createPath`, `getPath`, and `deletePath`. - Generated endpoints include pagination, sorting, filtering, ETags for optimistic locking, and standardized error responses. ### OpenSpec Workflow OpenSpec structures each change into four artifacts: - **Proposal:** Why the change is needed and what will change - **Design:** Technical decisions and trade-offs - **Delta specifications:** Behavioral requirements written as Given-When-Then scenarios - **Task list:** A step-by-step implementation checklist The developer and agent review the feature together, the agent creates these artifacts, and then implements the checklist incrementally. Once complete, the delta specification is archived into the main specification library, creating a versioned, evolving record of the system’s behavior. ## Automated Verification The team initially tried adding lists of pitfalls to prompts, but found this ineffective and potentially harmful. Instead, they made tests and tools reveal errors progressively so the agent could diagnose and correct them. - Automated tests, linters, and formatters provide precise feedback. - Failed tests identify what went wrong, allowing the agent to fix one issue before moving to the next. - Project-specific skills bundle these checks together. - `AGENTS.md` tells the agent when to load the relevant skills, avoiding unnecessary instructions on every turn. - Testing and linting are treated as essential infrastructure rather than optional activities, since agents frequently make errors during implementation. ## Fast, Independent Local Environments Relying on CI and shared test environments is too slow for agent-driven iteration. Long waits can disrupt the agent’s context and make repeated experimentation impractical. - A complete local environment provides immediate feedback. - Local dependencies make logs and state easier to inspect. - Developers avoid sending every failed attempt through a remote pipeline. - The local test suite contains 2,754 tests across three layers: - **Unit tests:** Isolated business logic - **Integration tests:** Real PostgreSQL, database constraints, triggers, soft-delete cascades, transactions, in-process HTTP, and OpenAPI compliance - **End-to-end tests:** Athenz authentication, Kong, API keys, and multi-tenant isolation - The full suite completes in roughly 15 seconds on a developer machine. - Parallel execution and strong test isolation are critical to achieving this speed. ## Practical Recommendation Agent-assisted development works best when agents are given clear behavioral contracts, immediate automated feedback, and a fast local loop. Teams should invest in specifications, comprehensive tests and linting, and realistic local dependencies so agents can correct mistakes continuously without waiting for CI.

Read original(opens in new tab)

PostgreSQLCoding AgentsAutomated TestingOpenapi+4

datadog·Jun 23, 2026·3 min readCurated summary

How we migrated a live routing system using AI-assisted refactoring

Stream Router evolved from a small configuration file into a critical control-plane service routing Datadog’s massive metrics workload. Its original FoundationDB key-value model eventually hit transaction-size and performance limits because relational relationships were reconstructed in application code. Datadog redesigned the system around PostgreSQL and DuckDB, using AI-assisted, test-driven refactoring to accelerate the migration without disrupting production traffic. ## Stream Router’s Role in Datadog’s Metrics Pipeline - Datadog processes more than a hundred trillion events per day. - Stream Router determines which Kafka cluster, topic, partitions, and sharding strategy should handle each datapoint. - It serves both producers and queriers but does not process Kafka messages itself. - Routing decisions change frequently as infrastructure evolves, making correctness and historical tracking essential. ## From Configuration File to Control Plane - In 2016, routing was managed through a small configuration file distributed to services. - As the platform grew, the file expanded to thousands of lines and required manual edits and rollouts. - Stream Router replaced this workflow with: - A centralized gRPC service - API-managed routes - Automated, gradual rollouts - The write path used FoundationDB, while the read path served static RocksDB snapshots restored into memory. - This eventually became a bottleneck as routing tables and operational changes grew larger. ## Why the Key-Value Model Stopped Scaling - Routes reference streams and sharding strategies, while rules reference routes. - These relationships are inherently relational and require cross-entity validation. - The KV implementation loaded tens of thousands of records into application processes and reconstructed database-like relationships in code. - Some operations exceeded FoundationDB transaction-size limits. - Moving to PostgreSQL without changing the access patterns would not solve the issue; certain operations were estimated to require 45 minutes because of thousands of sequential database round trips. - The fundamental problem was the data model and application logic, not simply the choice of database. ## Designing the New Storage Architecture - The team redesigned the schema manually before using AI tools. - The relational model introduced explicit foreign keys between: - Streams - Sharding strategies - Routes - Rules - PostgreSQL was selected for the write path because it provided the required relational semantics and transaction model. - DuckDB was selected for the read path because: - It is embeddable and suitable for snapshot-based serving - It supports array columns - Its SQL dialect is closely compatible with PostgreSQL - Shared query logic could therefore work across both storage engines. ## AI-Assisted Refactoring - Claude and Cursor were used to accelerate a systematic, test-driven migration. - For each method, developers supplied: - The old implementation - The new schema - A failing test - AI generated an initial implementation, while tests determined whether it was correct. - The models assisted with method-level refactoring rather than autonomously designing the architecture. - Human expertise remained central to schema design, migration strategy, and evaluating system-level risks. ## Foundations for a Safe Migration - The migration benefited from infrastructure already present at Datadog. - Stream Router’s storage layer was isolated behind an internal `Controller` interface. - This modularity helped contain storage changes and enabled incremental refactoring. - Existing tests and clear boundaries provided confidence in generated implementations while production traffic continued. The central lesson is that AI was most effective as an accelerator inside a disciplined engineering process. A well-designed relational schema, modular storage abstraction, and failing tests provided the safety mechanisms; AI helped implement the resulting changes faster, but did not replace human architectural judgment.

Read original(opens in new tab)

PostgreSQLDatabase DesignApache KafkaGrpc+4

aws·Jun 22, 2026·4 min readCurated summary

AWS Weekly Roundup: NY Summit recap, Local Zone in Hanoi, Grok 4.3 in Bedrock, price reductions, and more (June 22, 2026) | Amazon Web Services

AWS’s June 22, 2026 roundup centers on the New York Summit’s focus on AI agents that continuously create value across work, security, software development, and customer applications. It also highlights new regional infrastructure, developer tools, Bedrock models, data capabilities, performance improvements, and several price reductions. Overall, AWS is emphasizing agent-driven automation while lowering barriers and costs for building and operating cloud workloads. ## New York Summit: Agents Across the AWS Stack - **Agents for working:** Amazon Quick supports autonomous, multi-step agents and provides a prioritized activity feed combining email, Slack, calendars, and tasks. - **Agents for securing:** AWS Continuum is an AI-native security service designed to reason about, validate, and remediate vulnerabilities across the development lifecycle. - AWS Security Agent adds threat modeling, pull-request scanning and remediation, and IDE integrations through Kiro, Claude Code, and MCP. - **Agents for building:** Kiro, AWS DevOps Agent, and AWS Transform support continuous coding, deployment, release assessment, and autonomous modernization. - Kiro now includes a native iOS app. - AWS DevOps Agent can evaluate code changes before production release. - **Agents customers create:** Amazon Bedrock AgentCore adds a generally available infrastructure and orchestration harness, Web Search, Managed Knowledge Base, Guardrails integrations, and AWS Context for mapping organizational data relationships. ## New Infrastructure and Developer Services - **AWS Local Zone in Hanoi:** The new `ap-southeast-1-han-1a` zone supports Amazon S3 and Amazon EBS Local Snapshots, helping customers satisfy local data residency and backup requirements. - **AWS Blocks:** This preview open-source TypeScript framework provides a local environment with Postgres, authentication, and real-time messaging without requiring an AWS account. Applications can later deploy to AWS without code changes, with optional CDK integration. - **AWS Management Console Private Access:** Enterprises can access the AWS Console from isolated VPCs without internet connectivity, supporting air-gapped security models. - **AWS Marketplace Storefront:** Partners can publish branded catalogs of AWS Marketplace solutions on their own websites or applications. ## AI, Data, and Agent Capabilities - **Grok 4.3 in Amazon Bedrock:** xAI’s model is available for reasoning, agentic, and enterprise workflows, with tool calling, structured output, and response streaming. - **Amazon S3 annotations:** Objects can now carry up to 1 GB of mutable, queryable context, reducing the need for separate metadata systems in AI-agent and autonomous workflows. - **Strands Agents:** The open-source toolkit adds improved Harness SDK context management, isolated execution through Strands Shell, and chaos testing and red-team capabilities in Strands Evals. - **NVIDIA-powered EC2 G7:** G7 instances use NVIDIA RTX PRO 4500 Blackwell Server Edition GPUs and sixth-generation Intel Xeon processors, delivering up to 4.6 times the AI inference performance and 2.1 times the graphics performance of G6 instances. ## Performance and Security Improvements - **Faster Amazon ECS auto scaling:** Support for 20-second metrics reduces scale-out trigger time from 363 to 86 seconds and total scaling and task provisioning time from 386 to 109 seconds in AWS benchmarks. - **Palo Alto Networks DNS Security:** Route 53 Resolver DNS Firewall can apply PANW Advanced DNS Security protections directly, without separate firewalls or VPC changes. ## Price Reductions - **Amazon S3 Vectors:** Query charges for large vector indexes fall by up to 80%, with no application changes required. - **Amazon GameLift Servers:** Generation 6 and newer instances now include free inbound and outbound network bandwidth for both On-Demand and Spot usage. - **AWS Marketplace professional services:** Listing fees drop from 2.5% to 0.5%, reducing transaction costs for consulting, managed services, and software partners. AWS’s latest direction is to combine increasingly autonomous agents with faster infrastructure, broader model choice, stronger security, and lower operating costs. Developers and organizations should evaluate Bedrock AgentCore, AWS Blocks, S3 annotations, and the new regional and private-access options where they can simplify agent development or satisfy data and security requirements.

Read original(opens in new tab)

PostgreSQLAI AgentsGenerative AITypeScript+4

cloudflare·Jun 12, 2026·3 min readCurated summary

Scaling Security Insights: how we achieved a 10x increase in global scanning capacity

Security Insights needed a 10x throughput increase to scan all customers more frequently and detect risks sooner. The existing system was overwhelmed by Kafka backlogs, slow processing, database inefficiencies, and API timeouts. Cloudflare improved capacity by introducing parallel and lane-based processing, optimizing bulk database writes, and addressing regional latency between its API and database. ## Scaling Kafka Processing - Scans are scheduled and published to Apache Kafka. - Go-based checker services consume these messages, inspect accounts, zones, and DNS records, and send findings to an internal API. - Kafka’s partition ordering limits each consumer group to one active consumer per partition. - Slow messages could block all subsequent messages in the same partition. - Adding partitions was avoided because it would increase resource usage for shared Kafka brokers. ## Introducing Parallel Processing - Checkers were changed to consume messages in batches. - Each message in a batch is processed concurrently in its own goroutine. - This increased throughput without requiring additional Kafka partitions. - The trade-offs were higher memory usage and potentially more work to repeat after a process crash. ## Separating Slow and Fast Work - Some scans took seconds or milliseconds, while unusually large accounts or zones could take minutes or hours. - These slow messages caused head-of-line blocking for faster work. - Consumer groups and checkers were split into: - A fast lane for predictable, short-running scans - A slow lane for messages expected to require substantially more time - Fast-lane consumers skipped slow messages, allowing normal scans to continue without delay. ## Optimizing Postgres Writes - The API originally executed one insert/upsert transaction per insight. - A request containing up to 500,000 insights could therefore generate hundreds of thousands of database round trips. - Bulk insertion with `COPY` into a temporary table was tested but caused bloat in Postgres system tables. - The final hybrid approach used: - `UNNEST` for smaller batches - `COPY` for batches above a configured threshold - This delivered millisecond-level performance for small writes and completion within seconds for very large writes. ## Diagnosing API Timeouts - Client-side timeouts increased as scan volume grew. - Checkers sometimes spent 20–90% of their processing time waiting on a single API call. - Throughput initially rose but then deteriorated under heavy load. - The root cause was network latency: - Postgres was hosted in Portland, Oregon. - The API ran active-active in Portland and Amsterdam. - Requests routed to Amsterdam incurred roughly 50 milliseconds of network round-trip latency. - Amsterdam database queries held client connection-pool connections much longer—nearly three seconds on average versus about 10 milliseconds in Portland. - The connection pool became exhausted, causing requests to wait for available connections and creating uneven Kafka lag across partitions. Cloudflare’s results came from improving the full processing pipeline rather than relying on a single infrastructure change. Parallelize message handling, isolate slow workloads, batch database writes, and place latency-sensitive services close to their databases to achieve large throughput gains and more frequent security scanning.

Read original(opens in new tab)

PostgreSQLApache KafkaDistributed SystemsAPI+4

datadog·Jun 4, 2026·3 min readCurated summary

When failover isn’t safe: Building high-availability PostgreSQL on Kubernetes

Datadog’s gameday testing exposed a PostgreSQL failure mode in which network latency caused replication lag to grow until no standby could be safely promoted. Although the clusters remained writable, they could not fail over without risking data loss, forcing operators to wait for connectivity and replicas to recover. Datadog’s solution was to redesign failover candidates around synchronous replication coordinated by Patroni, balancing stronger durability with acceptable write latency. ## The Zonal Failure That Exposed the Weakness - A simulated availability-zone failure introduced network latency in a staging environment. - Several Kubernetes-based PostgreSQL clusters had primary nodes in the affected zone. - Communication between primaries and replicas degraded, causing: - Rapidly increasing replication lag - Stalled writes - Applications serving stale data - No replica being current enough for safe promotion - The clusters prioritized continued writes over durability, leaving them writable but unable to fail over safely. ## Baseline PostgreSQL Architecture - Each cluster uses a single-writer design: - One active leader handles writes. - Two standby nodes are reserved for failover and do not serve application traffic. - A separate read-replica pool handles read-only traffic and scales independently. - Read replicas are intentionally excluded from failover candidates. - Patroni manages replication, leader elections, and failover. - ZooKeeper acts as Patroni’s distributed configuration store, tracking: - The current leader lock - Cluster configuration - Member replication state and latest LSN - ZooKeeper’s ephemeral leader key ensures that only one node can become primary. - During partitions, Patroni favors safety by pausing or demoting nodes that cannot verify cluster state. ## Why Failover Was Not Safe - Patroni checks replication lag before promoting a standby using `maximum_lag_on_failover`. - During the gameday, all eligible standbys exceeded that threshold. - Patroni correctly rejected promotion because each candidate could have been missing committed transactions. - The cluster therefore had no safe writable primary, even though the original leader was impaired. - The failure was a consequence of asynchronous replication and network latency, not a failure in Patroni’s safety mechanisms. ## Asynchronous Versus Synchronous Replication - **Asynchronous replication**, used originally: - Lets the leader commit and respond without waiting for replicas. - Provides low write latency and high throughput. - Can lose transactions committed on the leader but not yet copied to a standby. - **Synchronous replication**: - Requires the leader to receive acknowledgment from at least one replica before confirming a transaction. - Reduces the chance that a failover candidate is significantly behind. - Provides stronger durability, but may increase write latency when replicas experience network or availability problems. ## The Redesigned Approach - Datadog reworked its PostgreSQL deployment so failover candidates use synchronous replication. - Patroni coordinates these replicas and continues to enforce safe leader election. - The design aims to make failover both automatic and safe while limiting performance impact. - Benchmarking and failure testing were used to evaluate the trade-off between durability and latency. Datadog’s experience demonstrates that asynchronous replication can leave a system operational but unable to fail over during network disruption. For clusters where data durability and automatic recovery are critical, synchronous replication for designated failover candidates offers a safer architecture, provided its latency and availability costs are measured carefully.

Read original(opens in new tab)

PostgreSQLKubernetesHigh AvailabilityDatabase Replication+4

discord·May 8, 2026·3 min readCurated summary

How Discord Automates ScyllaDB Clusters at Scale

Discord’s Persistence Infrastructure team replaced fragile, manually sequenced scripts with the Scylla Control Plane (SCP), a framework for safely automating large-scale database operations. The effort was driven by the difficulty of creating shadow clusters and managing hundreds of ScyllaDB nodes with a seven-person team. SCP emphasizes resumability, safety checks, configurable parallelism, and incremental development. ## The Scale of Discord’s Database Operations - Discord operates Elasticsearch, Postgres, and ScyllaDB infrastructure across dozens of clusters and hundreds of nodes. - ScyllaDB stores critical data, including messages, channels, servers, and much of Discord’s user data. - Routine work includes: - Rolling restarts after configuration changes - Cluster expansion as traffic grows - Operating-system upgrades without downtime - Creating test clusters for validating ScyllaDB releases - These operations require careful sequencing and continuous validation rather than simple, fire-and-forget automation. ## From Scripts to the Scylla Control Plane - Discord initially accumulated Python, Bash, and other scripts incrementally. - The scripts were useful but fragile and dependent on institutional knowledge. - As operational demands grew, Discord created the Scylla Control Plane, or SCP, to provide a more structured automation system. ## Shadow Clusters for Safer Upgrades - Shadow clusters are temporary, full replicas of production that receive the same reads and writes as live traffic. - They allow Discord to detect upgrade problems under realistic load before changing production. - Building one manually requires: - Provisioning and configuring nodes - Joining nodes to the cluster - Validating replication - Establishing dual-write pipelines - Eventually tearing the environment down - Repeating this process across every ScyllaDB cluster made automation essential, especially for testing operating-system, hardware, and ScyllaDB version changes. ## Lessons from the Previous Automation Discord identified three major weaknesses in its old scripts: - **Unsafe:** Scripts could be run against the wrong nodes or in the wrong order, often without precondition checks. - **Unrecoverable:** A failure late in a multi-step process required restarting from the beginning. - **Difficult to extend:** New operations often required copying and modifying existing scripts instead of composing reusable components. SCP was designed around four goals: - Provide an extensible task framework that hides orchestration complexity. - Support configurable parallelism, including constraints such as avoiding simultaneous work in different availability zones. - Make safety the default through preconditions, retries, and persisted state. - Deliver functionality incrementally and refine it through real-world use. ## SCP’s Task-Based Architecture - SCP is organized around **tasks, workflows, and jobs**. - A task represents one unit of work, such as draining a node, checking repair status, or running cleanup. - **Node tasks** operate on individual nodes. - **Cluster tasks** coordinate operations across an entire cluster and may run node tasks across many nodes. - SCP also uses **conditions**, which pause execution until a required state is reached. - Conditions poll ScyllaDB APIs or Prometheus metrics. - They either succeed when the criterion is met or fail after a timeout. - For example, after restarting a node, SCP can wait for compactions to settle before continuing. - This avoids unreliable fixed-duration sleeps and reduces the risk of creating cascading pressure during rolling operations. ## Practical Recommendation For large-scale database operations, automation should be built as a reusable, stateful orchestration framework rather than a collection of scripts. Explicit preconditions, observable conditions, retries, controlled parallelism, and resumable state make complex infrastructure changes safer and more repeatable.

Read original(opens in new tab)

PostgreSQLRustElasticsearchPrometheus+3

figma·May 4, 2026·3 min readCurated summary

PGKeeper: Building the Bouncer We Needed for Postgres | Figma Blog

Figma built PGKeeper to replace PgBouncer as its PostgreSQL connection and load-management layer. Growing traffic, sharding, and stricter reliability requirements exposed PgBouncer’s limits in scalability, prioritization, backpressure, connection protection, and extensibility. PGKeeper is a custom Go service positioned between Figma’s DBProxy routing layer and PostgreSQL, designed to protect databases from overload and connection churn. ## Figma’s Database Architecture - PostgreSQL powers Figma’s OLTP workloads. - Figma scales through horizontal and vertical sharding across multiple database instances. - DBProxy hides sharding complexity from application code by: - Parsing and analyzing queries. - Selecting the appropriate PostgreSQL instances. - Rewriting requests into queries for the selected targets. - A dedicated set of connection-pooler replicas serves each PostgreSQL machine, creating an n-to-one relationship between poolers and databases. ## Why PgBouncer Was No Longer Enough - **Limited scalability** - PgBouncer’s single-threaded architecture created a vertical scaling ceiling. - Adding replicas helped, but uneven load distribution caused performance degradation. - **Insufficient load management** - PgBouncer could not prioritize critical traffic over lower-priority or misbehaving requests. - It lacked effective backpressure and advanced load-shedding algorithms such as Controlled Delay (CoDel). - CoDel sheds work based on how long requests wait, rather than simply counting queued requests. - **Unsafe connection behavior** - PostgreSQL connections are expensive resources. - Rapid connection creation and churn could destabilize database nodes. - Recovery after overload could trigger another surge of connections, creating cascading failures and prolonged overload. - **Limited extensibility and control** - Figma needed deep observability, feature-flagged rollouts, admission control, and fair resource sharing. - Even maintaining small PgBouncer patches proved costly. - Extending PgBouncer substantially would create an ongoing maintenance burden. ## Why Connection Pooling Could Not Live in DBProxy - Figma generally limits each PostgreSQL instance to roughly 100 pooled connections. - Hundreds of stateless DBProxy replicas sit in front of those databases. - Giving every DBProxy replica its own pool would either exceed database connection limits or require complex coordination. - Centralizing pooling in a separate service provided a better fit for the mismatch between many routers and a small fixed connection budget. ## Why Figma Built PGKeeper - PGCat addressed PgBouncer’s single-threaded scalability problem, but customizing it would require deep changes to its core execution paths. - Those changes would likely require Figma to maintain a long-term fork. - Figma therefore created PGKeeper as a Go-based service tailored to its infrastructure and operational requirements. - Its role is to act like a goalkeeper: protecting PostgreSQL from harmful traffic and protecting connections from uncontrolled churn. PGKeeper was chosen because Figma needed more than a basic connection pooler: it needed a scalable, observable, controllable layer capable of prioritizing traffic and preventing database overload.

Read original(opens in new tab)

PostgreSQLGoDatabase ShardingLoad Balancing+3

netflix·May 1, 2026·3 min readCurated summary

State of Routing in Model Serving

Netflix’s centralized ML serving platform provides a single, domain-independent API for model inference across personalized experiences and other use cases. Rather than exposing individual scoring functions, Netflix packages feature computation, preprocessing, inference, and postprocessing into self-contained model workflows. The core routing challenge is directing each request to the correct model version and serving cluster while keeping client services independent from model changes and infrastructure topology. ## Models as End-to-End Workflows - Netflix distinguishes **model serving** from traditional model inference: - Inference typically means `infer(features) -> score`. - Serving includes preprocessing, feature computation, optional trained components, and postprocessing. - Example workflows include: - Ranking titles for a personalized Continue Watching row using user, country, and device context. - Predicting payment fraud using user, country, and transaction details. - Models declare the facts they need, while the serving platform retrieves those facts from other microservices. - During offline training, Netflix’s ML fact store provides snapshots for bulk feature computation. - Calling services provide standard request context and domain-specific inputs, while the platform handles feature generation, model selection, and execution. ## Platform Design Principles - **Model innovation without client changes** - Client applications integrate with the platform once. - Model versions, A/B tests, additional experimental data, logging, and model selection remain hidden behind the platform API. - **Clients decoupled from model sharding** - Models run across multiple serving cluster shards, each with its own Virtual IP address. - Shard assignments can change based on traffic, SLAs, model architecture, and resource availability. - Clients should not need to track these VIP changes. - **Flexible traffic routing** - Routing must support A/B allocations, gradual traffic shifts, new model versions, new VIPs, and client-specific overrides. - Safe lifecycle management requires support for shadow deployments, canaries, rollbacks, and migrations. ## Switchboard: Context-Aware Routing - Generic API gateways and service-mesh proxies did not satisfy Netflix’s requirements. - Netflix needed: - Native integration with its experimentation platform. - gRPC support. - Routing based on rich, domain-specific request context. - Model-specific rollout and migration controls. - Netflix built **Switchboard**, a custom proxy layer handling more than one million requests per second. - Switchboard is the mandatory entry point for clients and: - Routes requests to the appropriate model based on request context. - Applies configured context enrichment before invoking the model. - Hides model locations and infrastructure changes from client services. ## Objective Abstraction - Every request must provide an **Objective**, an enumeration defined by the serving platform. - The excerpt introduces Objectives as a central abstraction for identifying the business purpose of a serving request, but the supplied text ends before describing its full roles. Netflix’s approach is to centralize routing, experimentation, and model execution behind one stable API. This allows client applications to evolve independently while researchers can iterate on models and safely manage large-scale production rollouts.

Read original(opens in new tab)

PostgreSQLDatabase DesignMachine LearningMicroservices+4

aws·Apr 27, 2026·3 min readCurated summary

AWS Weekly Roundup: Anthropic & Meta partnership, AWS Lambda S3 Files, Amazon Bedrock AgentCore CLI, and more (April 27, 2026) | Amazon Web Services

This week’s AWS news centers on deeper AI infrastructure partnerships and tools for building production-ready agents. AWS and Anthropic are expanding Claude’s integration with AWS hardware and Amazon Bedrock, while Meta is adopting Graviton for large-scale agentic AI workloads. New services for Lambda, EKS, Aurora, and Bedrock also emphasize simpler data access, hybrid networking, serverless scaling, and faster agent development. ## Anthropic and Meta Expand AWS AI Partnerships - Anthropic is training advanced foundation models on AWS Trainium and Graviton processors. - Anthropic and AWS’s Annapurna Labs are co-engineering at the silicon level to improve efficiency across the stack. - Claude Cowork is now available through Amazon Bedrock, allowing enterprise teams to collaborate with Claude while keeping data within AWS. - A unified Claude Platform on AWS is planned, offering a single experience for building, deploying, and scaling Claude applications. - Meta signed an agreement to deploy tens of millions of AWS Graviton cores for CPU-intensive agentic AI tasks, including reasoning, code generation, search, and orchestration. ## New Lambda and Kubernetes Infrastructure - AWS Lambda can mount Amazon S3 buckets as file systems using S3 Files. - Functions can perform standard file operations without downloading data first. - Built on Amazon EFS, S3 Files combines file-system access with S3’s scalability, durability, and cost model. - Multiple Lambda functions can share the same workspace, supporting AI agents that need persistent memory or shared state. - The Amazon EKS Hybrid Nodes gateway simplifies networking between cloud-based EKS resources and on-premises Kubernetes Pods. - It enables pod-to-pod traffic, control-plane webhook communication, and access to AWS services without making on-premises pod networks routable. - The gateway is available at no additional charge. ## Aurora Serverless and Bedrock Agent Development - Aurora Serverless now offers up to 30% better performance on platform version 4. - Its scaling algorithm better handles competing workloads, including busy APIs and bursty agentic AI applications. - The service continues to scale to zero during idle periods, with no additional charge for the improvements. - Amazon Bedrock AgentCore adds a managed harness in preview, allowing developers to define a model, system prompt, and tools without writing orchestration code. - Harnesses can later be exported as Strands-based code for greater control. - The AgentCore CLI supports governed, auditable deployments through AWS CDK, with Terraform support planned. - The CLI is available in 14 AWS Regions at no additional charge, and AgentCore skills support coding assistants. ## Cost Management, Operations, and Machine Learning - Granular cost attribution for Amazon Bedrock enables teams to track usage by project or organization and support detailed chargeback. - AWS DevOps Agent can work with the Salesforce MCP Server to investigate incidents, diagnose causes, and notify customers through Salesforce Service Cloud. - AWS microcredentials are now free through AWS Skill Builder in supported countries. - These hands-on assessments use simulated business scenarios and live AWS environments rather than traditional multiple-choice testing. - Amazon SageMaker AI can recommend optimized generative AI inference configurations, including instance types, containers, and inference parameters, helping reduce latency and deployment costs. ## Upcoming AWS Events - “What’s Next with AWS” is scheduled as a virtual event on April 28. - AWS Summits continue in May across cities including Singapore, Tel Aviv, Warsaw, Stockholm, Sydney, Hamburg, Seoul, Amsterdam, Bangkok, and Milan. AWS’s latest releases point toward a more integrated AI platform: specialized hardware for model execution, managed agent tooling, shared state through serverless storage, and stronger cost and operational controls. Builders should evaluate S3 Files and AgentCore for AI workflows, while teams running production inference can benefit from SageMaker recommendations and Bedrock’s improved cost attribution.

Read original(opens in new tab)

PostgreSQLKubernetesGenerative AIAWS+4

gitlab·Apr 22, 2026·3 min readCurated summary

GitLab AI Hackathon 2026: Meet the winners

Nearly 7,000 developers participated in GitLab’s 2026 AI Hackathon, creating more than 600 agents and workflows for the GitLab Duo Agent Platform. The winning projects focused on practical software delivery challenges—including organizational knowledge loss, security, compliance, migrations, and sustainability—rather than simple chatbot interactions. The results suggest that agentic AI is becoming most valuable when integrated directly into development workflows and given richer project context. ## Hackathon Scope and Evaluation - The hackathon ran from February 9 to March 25, 2026, on Devpost. - Google Cloud and Anthropic co-sponsored the event, contributing judges, prizes, and cloud resources. - Nineteen judges evaluated projects on: - Technical execution - Design - Potential impact - Quality of the idea - Total prizes reached $65,000. ## Grand Prize: LORE - LORE, or Living Organizational Record Engine, addresses the loss of institutional knowledge when engineers leave. - It combines: - Eight specialized agents - A router that directs questions to the appropriate agent - Protections against circular loops in its knowledge graph - A visual dashboard - Carbon tracking - Its command-line tool includes 43 tests, leading judges to describe it as a polished product rather than a typical hackathon prototype. ## Google Cloud and Anthropic Winners - **Gitdefender**, the Google Cloud Grand Prize winner, detects security issues during code review, writes fixes, and opens the review automatically. - **Aegis**, the Google Cloud Runner Up, explains the reasoning behind its AI decisions and is deployed on Google Cloud. - **GraphDev**, the Anthropic Grand Prize winner, maps code relationships and shows how systems evolve, helping developers understand the impact of changes. - **DocSync**, the Anthropic Runner Up, uses Detector, Writer, and Reviewer agents to update documentation. It opens a review when confident and creates an issue for human review when uncertain. ## Category Winners - **Time-Traveler**, winner for technical achievement, creates a safe copy of a production environment and runs database migrations against it using five connected agents, PostgreSQL, real data, and Google Cloud deployment. - **RedAgent**, the most impactful project, verifies AI-generated security findings before developers act on them, addressing distrust in automated reports. - **Launch Control**, recognized for ease of use, combined polished user experience, strong infrastructure, and sustainability considerations. ## Sustainability-Focused Projects - Five projects received sustainability prizes or bonuses as the organizers highlighted the growing energy cost of CI/CD systems and large language models. - **GreenPipe** analyzes CI/CD pipelines and generates carbon-footprint reports. - Sustainable Design bonuses recognized projects including: - **BugFlow**, which generated 10 fixes from one bug report in 20 minutes - **DELTA Cyber Reasoning**, an automated fuzz-testing tool - **CarbonLint**, which applies code analysis to energy consumption - **TFGuardian**, which includes carbon-footprint analysis - One project reduced monthly costs from $556 to $18, representing a reported 96% carbon reduction. ## Honorable Mentions - **SecurityMonkey** tests security scanners by injecting known vulnerabilities. - **stregent** enables CI/CD investigation and fixes through WhatsApp. - **Compliance Sentinel** evaluates merge requests for compliance risk and blocks critical violations. - **Carbon Tracker** measures the carbon footprint of individual pipeline jobs and suggests improvements. - **RepoWarden** captures the rationale behind code, not only its behavior. - **MR Compliance Auditor** maps merge-request evidence to SOC 2 controls and displays compliance scores in real time. ## What Comes Next The projects operated within a single GitLab project, but many teams supplemented their agents with local knowledge graphs to understand code relationships and dependencies. GitLab plans to build on this approach in future hackathons by providing agents with richer context. GitLab’s hackathon demonstrates that the strongest AI agents are workflow-integrated tools that can investigate, make decisions, execute changes, and involve humans when needed. Developers can explore the 600-plus projects in the gallery or build their own agents on the GitLab Duo Agent Platform.

Read original(opens in new tab)

PostgreSQLDatabase DesignMachine LearningCI/CD+4

google·Apr 21, 2026·2 min readCurated summary

ReasoningBank: Enabling agents to learn from experience

ReasoningBank is an agent-memory framework designed to help deployed agents learn continuously from both successful and failed task attempts. Rather than storing exhaustive action histories or only successful workflows, it distills reusable reasoning strategies, decision rationales, and preventative lessons. Evaluations on WebArena and SWE-Bench-Verified show higher success rates and fewer execution steps, especially when combined with memory-aware test-time scaling. ## Distilling Generalizable Reasoning - Each memory contains: - A concise title - A brief description - Detailed reasoning steps, rationales, or operational insights - The agent retrieves relevant memories before acting. - After completing a task, an LLM judge evaluates the trajectory and identifies useful success or failure signals. - The agent converts those signals into new memories and appends them to the ReasoningBank. - Failure analysis is central: mistakes become counterfactual guidance and strategic guardrails, such as verifying the current page before repeatedly clicking “Load More.” ## Memory-Aware Test-Time Scaling - Memory-aware test-time scaling (MaTTS) connects inference-time exploration with long-term memory. - **Parallel scaling:** Multiple trajectories are generated and compared, allowing the agent to distinguish robust strategies from flawed reasoning. - **Sequential scaling:** The agent progressively refines a single trajectory, preserving useful intermediate insights from trial and error. - This creates a feedback loop: better memories guide exploration, while richer exploration produces better memories. ## Benchmark Results and Strategic Maturity - Against memory-free ReAct agents using Gemini-2.5-Flash: - Success rates improved by 8.3% on WebArena. - Success rates improved by 4.6% on SWE-Bench-Verified. - SWE-Bench-Verified tasks required nearly three fewer execution steps on average. - Adding MaTTS with parallel scaling factor **k=5** produced further gains: - A 3% success-rate increase over ReasoningBank alone on WebArena. - 0.4 fewer steps per task. - Over repeated tasks, simple procedural checklists evolved into more sophisticated memories containing compositional and preventative logic. ReasoningBank suggests that effective agent scaling requires more than additional inference compute or stored trajectories. Agents should systematically learn from both outcomes and mistakes, using structured reasoning memories to become more capable and efficient after deployment.

Read original(opens in new tab)

PostgreSQLDatabase DesignLarge Language ModelsContrastive Learning+4

aws·Apr 20, 2026·3 min readCurated summary

AWS Weekly Roundup: Claude Opus 4.7 in Amazon Bedrock, AWS Interconnect GA, and more (April 20, 2026) | Amazon Web Services

The roundup highlights major AWS advances in AI, networking, developer tooling, and security. Claude Opus 4.7 is now available through Amazon Bedrock with stronger agentic coding and research capabilities, while AWS Interconnect simplifies private connectivity across clouds and remote locations. Additional launches improve container supply-chain security, application modernization, database access, cost attribution, and quantum-resistant encryption. ## Anthropic Claude Opus 4.7 in Amazon Bedrock - Anthropic’s latest Opus model improves: - Agentic coding and long-running tasks - Complex code reasoning - Document creation, financial analysis, and multi-step research - It scores: - 64.3% on SWE-bench Pro - 87.6% on SWE-bench Verified - Bedrock features include: - Dynamic capacity allocation - Adaptive thinking and request-specific token budgets - A 1-million-token context window - High-resolution image support for charts, documents, and screen interfaces - The model launched in US East, Tokyo, Ireland, and Stockholm, supporting up to 10,000 requests per minute per account and Region. ## AWS Interconnect Reaches General Availability - **AWS Interconnect – Multicloud** provides Layer 3 private connectivity between AWS VPCs and other clouds. - Google Cloud is supported initially; Azure and OCI are planned. - Traffic uses private networks and the AWS global backbone rather than the public internet. - Includes MACsec encryption, multi-facility resilience, and CloudWatch monitoring. - The underlying specification is open source under Apache 2.0. - **AWS Interconnect – Last Mile** connects branches, data centers, and remote sites to AWS through network providers. - Automatically provisions four redundant connections across two physical locations. - Configures BGP, MACsec, and Jumbo Frames. - Supports adjustable bandwidth from 1 to 100 Gbps. - Launches in US East with Lumen. ## Developer, Database, and Modernization Updates - Amazon ECR pull-through cache now discovers and synchronizes OCI referrers such as signatures, SBOMs, and attestations. - AWS Transform is available directly in Kiro and VS Code for migrations such as language-version upgrades and AWS SDK updates. - Aurora DSQL’s PHP connector supports IAM authentication, SSL, connection pooling, and optional optimistic-concurrency retries. - AWS Transform Custom can modernize VB6 applications into C# ASP.NET Core applications, including COM, ADO, and UI migration challenges. ## Security, Access Control, and Cost Management - Amazon Q for Google Drive now enforces document-level permissions using indexed ACLs and real-time access checks. - AWS Secrets Manager supports hybrid post-quantum TLS using ML-KEM through updated agents, Lambda extensions, and CSI drivers. - Amazon Bedrock can attribute inference costs to individual IAM principals, with reporting through CUR 2.0 and aggregation by teams, projects, or cost centers. ## Compute, Kubernetes, and Storage - EC2 C8in and C8ib instances use sixth-generation Intel Xeon processors and AWS Nitro cards. - C8in offers up to 600 Gbps networking. - C8ib provides up to 300 Gbps EBS bandwidth. - Both scale to 384 vCPUs. - EKS Auto Mode automates networking components such as VPC CNI, load balancers, and DNS while retaining enterprise security controls. - EBS Volume Clones provide immediately usable point-in-time copies for development, disaster recovery testing, and CI/CD workflows. ## Additional AWS Guidance - CloudFront Functions and CloudFront KeyValueStore can support zero-downtime API decomposition using user-aware routing and the Strangler Fig pattern. - The roundup also points readers to AWS events, weekly Power Hour training, and Community.aws meetups. The most significant developments are Bedrock’s expanded AI capabilities and Interconnect’s managed private networking. Teams should evaluate Claude Opus 4.7 for complex AI workflows, use Interconnect where multicloud or resilient connectivity is required, and consider the new security and cost-attribution features for stronger governance.

Read original(opens in new tab)

PostgreSQLKubernetesAmazon BedrockDevOps+4

cloudflare·Apr 20, 2026·4 min readCurated summary

The AI engineering stack we built internally — on the platform we ship

Cloudflare built an internal AI engineering stack that now supports AI coding tools for 93% of its R&D organization. The system combines centralized authentication and model routing with internal knowledge, automated review, and sandboxed agent execution. Cloudflare argues that building these capabilities on its own platform improved security, visibility, cost control, and developer velocity, while also serving as a proving ground for products it ships publicly. ## Adoption and Impact - Over the previous 30 days: - 3,683 employees used AI coding tools, representing 60% of the company and 93% of R&D. - AI tools generated 47.95 million requests. - 295 teams used agentic AI tools or coding assistants. - AI Gateway handled 20.18 million requests and routed 241.37 billion tokens. - Workers AI processed 51.83 billion tokens. - The four-week rolling average of merge requests increased from roughly 5,600 per week to more than 8,700. - The week of March 23 reached 10,952 merge requests, nearly twice the Q4 baseline. - Cloudflare’s initial focus was MCP servers, but the effort expanded to standards, code review, onboarding, and propagating changes across thousands of repositories. ## Architecture at a Glance Cloudflare organized the stack into three layers: - **Platform layer:** Authentication, model routing, inference, MCP access, and code execution. - **Knowledge layer:** System context and repository guidance through Backstage and `AGENTS.md`. - **Enforcement layer:** Automated quality controls using AI Code Reviewer and the Engineering Codex. The stack uses Cloudflare products including: - **Cloudflare Access** for zero-trust authentication. - **AI Gateway** for centralized LLM routing, cost tracking, bring-your-own-key support, and zero-data-retention controls. - **Workers AI** for open-weight model inference. - **Workers and Access** for an MCP Server Portal with single OAuth. - **Dynamic Workers** for sandboxed agent-generated code execution. - **Agents SDK and Durable Objects** for stateful, long-running agent sessions. - **Sandbox SDK** for isolated cloning, building, and testing environments. - **Workflows** for durable, multi-step processes. - **Backstage** for a 16,000-plus-entity knowledge graph. ## Centralized Authentication and AI Routing - Cloudflare Access authenticates users and enforces zero-trust policies. - Every LLM request passes through AI Gateway, providing a single control point for: - Provider credentials - Usage and cost attribution - Model selection - Data-retention policies - Provider permissions - In the past month, frontier providers handled 91.16% of requests, while Workers AI handled 8.84%. - Cloudflare routes requests through a proxy Worker rather than connecting clients directly to AI Gateway. - The proxy enables later additions such as per-user attribution, model catalogs, permission enforcement, and support for new coding tools without changing client configurations. ## Workers AI and Open-Weight Models - Workers AI runs open-source models on GPUs distributed across Cloudflare’s global network. - Keeping inference on the same network as Workers, Durable Objects, and storage reduces latency, network failures, and cross-cloud configuration. - Kimi K2.5, with a 256,000-token context window, tool calling, and structured outputs, processes more than 7 billion tokens per day for a Cloudflare security agent. - Cloudflare estimates that running this workload on Workers AI is 77% cheaper than using a mid-tier proprietary model. - Workers AI is also used for: - Documentation review in CI - Generating `AGENTS.md` files - Lightweight inference where latency matters more than maximum model capability - Cloudflare expects open-source models to handle an increasing proportion of its internal workloads. ## One-Command Client Configuration - Engineers begin setup with: ```bash opencode auth login https://opencode.internal.domain ``` - The command uses an OpenCode discovery endpoint at: ```text https://opencode.internal.domain/.well-known/opencode ``` - The Worker-hosted endpoint provides authentication and configuration information. - This mechanism is designed to configure providers, models, MCP servers, agents, commands, and permissions without requiring engineers to edit configuration files manually. ## Overall Recommendation Cloudflare’s experience suggests that organizations adopting AI coding tools should build a centralized control plane early: authenticate users consistently, route model traffic through one managed gateway, maintain shared system knowledge, and enforce quality through automated review and isolated execution. Using the same production platform for internal tooling can also expose product gaps and accelerate improvements to the platform itself.

Read original(opens in new tab)

PostgreSQLDatabase DesignCI/CDOpen Source+4