Load Balancing | Techlist.io

cloudflareAug 7, 20263 min readCurated summary

Unifying Workers AI and AI Gateway into a single AI control plane

AI Gateway and Workers AI are converging into a unified control plane for accessing models across Cloudflare and external providers. A single Workers binding or REST API can now provide inference, observability, logging, security, and billing without requiring users to choose a product upfront. Cloudflare’s longer-term goal is model-first routing, where applications request capabilities or models while the gateway handles provider selection, failover, and load balancing. ## Unified Bindings and API - The Workers AI binding and AI Gateway now share the same entrypoint. - Requests can use the built-in `default` gateway or a named gateway for separate applications and customized policies. - The unified REST API routes requests through `/ai/` endpoints, using the `cf-aig-gateway-id` header. - This removes the need to decide between Workers AI and AI Gateway before building an application. ## Automatic Observability for Workers AI - Passing `default` as the gateway ID automatically creates an AI Gateway on the first authenticated request. - Requests receive built-in: - Full request and response logging - Token tracking by model - Cost attribution - Latency and error metrics - Developers can begin with the default gateway and later switch to a named gateway for features such as custom caching or application-specific traffic separation. - The AI Gateway dashboard provides detailed visibility into prompts, responses, latency, token usage, and failures. ## Unified Billing with AI Gateway Credits - AI Gateway credits can now pay for Workers AI usage in addition to providers such as OpenAI and Anthropic. - Users can maintain one prepaid credit balance across supported providers. - Workers AI users who use unified billing receive elevated rate limits, subject to current Cloudflare policies and documentation. ## Model-First Routing - Cloudflare plans to route requests based on the desired model rather than requiring users to select a specific provider. - The gateway could handle: - Provider selection - Failover - Load balancing - Capacity management - For example, a request for a model such as Kimi K2.7 Code could be served by Workers AI, the model’s original provider, or another vetted provider hosting the same weights. - Applications could remain available if one provider is overloaded or unavailable. - Users will still be able to restrict traffic to a single provider when necessary. - Routing is intended to preserve requirements such as Zero Data Retention and maintain model quality. Cloudflare recommends using the unified binding or REST API with the default gateway to gain observability and centralized billing immediately. As model-first routing develops, applications can rely less on provider-specific infrastructure and gain greater resilience through automatic provider management.

Read original

Load Balancing Observability Rest Api Ai Gateway+3

figmaMay 4, 20263 min readCurated summary

PGKeeper: Building the Bouncer We Needed for Postgres | Figma Blog

Figma built PGKeeper to replace PgBouncer as its PostgreSQL connection and load-management layer. Growing traffic, sharding, and stricter reliability requirements exposed PgBouncer’s limits in scalability, prioritization, backpressure, connection protection, and extensibility. PGKeeper is a custom Go service positioned between Figma’s DBProxy routing layer and PostgreSQL, designed to protect databases from overload and connection churn. ## Figma’s Database Architecture - PostgreSQL powers Figma’s OLTP workloads. - Figma scales through horizontal and vertical sharding across multiple database instances. - DBProxy hides sharding complexity from application code by: - Parsing and analyzing queries. - Selecting the appropriate PostgreSQL instances. - Rewriting requests into queries for the selected targets. - A dedicated set of connection-pooler replicas serves each PostgreSQL machine, creating an n-to-one relationship between poolers and databases. ## Why PgBouncer Was No Longer Enough - **Limited scalability** - PgBouncer’s single-threaded architecture created a vertical scaling ceiling. - Adding replicas helped, but uneven load distribution caused performance degradation. - **Insufficient load management** - PgBouncer could not prioritize critical traffic over lower-priority or misbehaving requests. - It lacked effective backpressure and advanced load-shedding algorithms such as Controlled Delay (CoDel). - CoDel sheds work based on how long requests wait, rather than simply counting queued requests. - **Unsafe connection behavior** - PostgreSQL connections are expensive resources. - Rapid connection creation and churn could destabilize database nodes. - Recovery after overload could trigger another surge of connections, creating cascading failures and prolonged overload. - **Limited extensibility and control** - Figma needed deep observability, feature-flagged rollouts, admission control, and fair resource sharing. - Even maintaining small PgBouncer patches proved costly. - Extending PgBouncer substantially would create an ongoing maintenance burden. ## Why Connection Pooling Could Not Live in DBProxy - Figma generally limits each PostgreSQL instance to roughly 100 pooled connections. - Hundreds of stateless DBProxy replicas sit in front of those databases. - Giving every DBProxy replica its own pool would either exceed database connection limits or require complex coordination. - Centralizing pooling in a separate service provided a better fit for the mismatch between many routers and a small fixed connection budget. ## Why Figma Built PGKeeper - PGCat addressed PgBouncer’s single-threaded scalability problem, but customizing it would require deep changes to its core execution paths. - Those changes would likely require Figma to maintain a long-term fork. - Figma therefore created PGKeeper as a Go-based service tailored to its infrastructure and operational requirements. - Its role is to act like a goalkeeper: protecting PostgreSQL from harmful traffic and protecting connections from uncontrolled churn. PGKeeper was chosen because Figma needed more than a basic connection pooler: it needed a scalable, observable, controllable layer capable of prioritizing traffic and preventing database overload.

Read original

Load Balancing PostgreSQL Go Database Sharding+3

googleMar 19, 2025Original article

Load balancing with random job arrivals (opens in new tab)

Research from Google explores the competitive ratio of online load balancing when tasks arrive in a uniformly random order rather than an adversarial one. By analyzing a "tree balancing game" where edges must be oriented to minimize node indegree, the authors demonstrate that random arrival sequences still impose significant mathematical limitations on deterministic algorithms. The study ultimately concludes that no online algorithm can achieve a competitive ratio significantly better than $\sqrt{\log n}$, establishing new theoretical boundaries for efficient cluster management. ### The Online Load Balancing Challenge * Modern cluster management systems, such as Google’s Borg, must distribute hundreds of thousands of jobs across machines to maximize utilization and minimize the maximum load (makespan). * In the online version of this problem, jobs arrive one-by-one, and the system must assign them immediately without knowing what future jobs will look like. * Traditionally, these algorithms are evaluated using "competitive analysis," comparing the performance of an online algorithm against an optimal offline version that has full knowledge of the job sequence. ### The Tree Balancing Game * The problem is modeled as a game where an adversary presents edges of a tree (representing jobs and machines) one at a time. * For every undirected edge $(u, v)$ presented, the algorithm must choose an orientation ($u \to v$ or $v \to u$), with the goal of minimizing the maximum number of edges pointing at any single node. * In a worst-case adversarial arrival order, it has been mathematically proven since the 1990s that no deterministic algorithm can guarantee a maximum indegree of less than $\log n$, where $n$ is the number of nodes. ### Performance Under Random Arrival Orders * The research specifically investigates "random order arrivals," where every possible permutation of the job sequence is equally likely, simulating a more natural distribution than a malicious adversary. * While previous assumptions suggested that a simple "greedy algorithm" (assigning the job to the machine with the currently lower load) performed better in this model, this research proves a new, stricter lower bound. * The authors demonstrate that even with random arrivals, any online algorithm will still incur a maximum load proportional to at least $\sqrt{\log n}$. * For more general load balancing scenarios beyond simple trees, the researchers established a lower bound of $\sqrt{\log \log n}$. ### Practical Implications These findings suggest that while random job arrival provides a slight performance advantage over adversarial scenarios, system designers cannot rely on randomness alone to eliminate load imbalances. Because the maximum load grows predictably according to the $\sqrt{\log n}$ limit, large-scale systems must be architected to handle this inherent logarithmic growth in resource pressure to maintain high utilization and stability.

Load Balancing Scheduling Algorithms Graph Theory Online Algorithms+3

datadogFeb 19, 20253 min readCurated summary

Achieving relentless Kafka reliability at scale with the Streaming Platform

Datadog built a Streaming Platform to make Kafka resilient and dynamically manageable across hundreds of clusters, thousands of topics, and millions of partitions. By decoupling applications from physical Kafka resources, the platform can reroute traffic, rebalance workloads, and fail over in seconds without redeployments. Its design trades strict processing order for parallelism and durability, while adding mechanisms to prevent poisoned events and backlogs from blocking entire partitions. ## A Fluid Control Plane for Multi-Cluster Orchestration - Kafka infrastructure is treated as interchangeable commodity hardware rather than a fixed application dependency. - Workloads can shift between clusters, availability zones, and Kubernetes environments automatically. - The control plane continuously replaces unhealthy components and maintains uninterrupted data flow. ### Resilient Pipelines with Streams - Streams abstract away Kafka topics and clusters behind stable identifiers. - A single Stream can span multiple Kafka clusters, availability zones, and Kubernetes clusters. - Producers and consumers do not need to know the underlying Kafka topology. - Infrastructure can be reconfigured in real time without application changes or redeployments. ### Live Traffic Failovers - When a cluster becomes unhealthy, the platform creates a replacement topic and redirects new traffic to it. - Consumers drain the backlog from the original topic before transitioning to the replacement. - The same mechanism supports proactive traffic redistribution, cluster decommissioning, capacity changes, and partition adjustments. - Operations that traditionally take hours can be completed in seconds. ### Consumer Semantics for Scale and Durability - Datadog uses at-least-once delivery rather than strict processing order. - Events can be processed independently and in parallel. - Ordering is restored later in the event store. - This trade-off enables efficient processing of petabytes of data across distributed infrastructure. ### The Assigner Coordinator - Kafka’s default group coordinator depends on session timeouts, making failover detection take tens of seconds or minutes. - Datadog’s Assigner monitors cluster health, resource usage, and workload distribution continuously. - It reacts in seconds to failures, traffic spikes, and capacity changes. - Workloads are balanced using real metrics such as CPU load and available resources, allowing heterogeneous environments to be used efficiently. ## Preventing Head-of-Line Blocking Kafka’s strict per-partition ordering means a single unprocessable event can block all later events. Datadog addresses this reliability problem through independent Stream lanes and a more flexible commit log. ### Stream Lanes and Quality of Service - Streams contain separate lanes for different priority levels and traffic requirements. - High-priority real-time traffic can be isolated from slower bursts or late-arriving data. - A dedicated dead-letter queue lane receives poison pills that cannot be processed. - Consumers can commit progress after moving failed events to the DLQ, preventing one bad event from blocking the partition without losing data. ### Advanced Commit Logging - Kafka traditionally maintains one committed pointer per partition. - That model prevents consumers from advancing past older events that are delayed or still being processed. - Datadog uses Kafka’s commit metadata to record multiple offsets or offset ranges simultaneously. - This allows consumers to continue processing live traffic while older events are handled separately, reducing backlog-related blocking. ## Overall Design Philosophy - The Streaming Platform combines stable logical Streams, real-time orchestration, flexible consumer semantics, QoS isolation, and enhanced offset tracking. - Together, these components make Kafka more self-healing and suitable for Datadog’s extreme scale. - The approach prioritizes uninterrupted processing, rapid recovery, and operational flexibility over Kafka’s default assumptions of fixed topology and strict ordering. Datadog’s design demonstrates that Kafka at very large scale requires a control layer around the broker infrastructure. Abstracting resources, automating failovers, and allowing independent progress through partitions can provide substantially better reliability and throughput than relying solely on Kafka’s native coordination model.

Read original

Load Balancing Database Design Kubernetes Apache Kafka+4

datadogFeb 22, 20233 min readCurated summary

Husky: Exactly-once ingestion and multi-tenancy at scale

Husky, Datadog’s distributed, time-series-oriented event store, is optimized for large scans and aggregations rather than high-volume, low-latency point lookups. This makes exactly-once ingestion challenging, especially at Datadog’s multi-tenant scale. Datadog addresses the problem with deterministic, locality-aware routing that limits deduplication scope, improves storage efficiency, and supports autoscaling ingestion pipelines. ## Husky’s Ingestion Challenge - Husky separates storage and compute, allowing each to scale independently. - Its storage engine is designed primarily for large analytical scans and aggregations. - It is not optimized for massive numbers of low-latency point lookups, complicating duplicate detection during ingestion. - The ingestion system must guarantee that every event is stored exactly once while maintaining: - Multi-tenant scalability - Reasonable ingestion latency - Controlled infrastructure and storage costs ## Routing Events to Storage Shards - Datadog uses an upstream **Shard Router** to introduce locality into Kafka pipelines. - Events are deterministically assigned to shards based on their tenant, timestamp, and event ID. - Each tenant receives a list of shards rather than being permanently assigned to one shard. - A deterministic choice from that list distributes the tenant’s events while keeping the number of active shards as small as practical. - Downstream workers consume one or more shards and perform exactly-once ingestion into Husky. ## Benefits of Data Locality - **Simpler deduplication** - An event with the same timestamp and ID always reaches the same shard. - Deduplication only needs to occur within that shard. - Workers handle smaller sets of event IDs, making in-memory deduplication more efficient. - **Lower storage costs and better performance** - Each shard processes a relatively small set of tenants. - Husky stores each tenant in a separate table and does not mix tenants within files. - More tenants per writer produce more output files, increasing blob-storage costs and compaction work. - Restricting tenant cardinality reduces file creation and improves writer and compactor efficiency. ## Challenges in Deterministic Routing - **Changing shard assignments** - Tenant traffic can increase by one or two orders of magnitude. - Assignments may change when scaling a tenant across more shards or rebalancing traffic among existing shards. - **Distributed router consensus** - Every Shard Router node must make the same routing decision for a given event. - Inconsistent decisions could send duplicates to different shards and undermine exactly-once ingestion. - **Load balancing** - Shards must receive roughly equal traffic so downstream ingestion workers remain balanced. ## Time-Bounded Shard Placements - A simple deterministic mapping can select a shard using a hash of the event ID: ```text shard = shards[hash(event_id) % num_shards] ``` - This approach is cheap and stateless when all routers know the same shard list. - However, changing the shard list can cause the same event ID to map to a different shard, so assignment changes require additional coordination. - The article introduces **time-bounded Shard Placements** to preserve consistent routing while allowing tenant assignments to evolve, though the supplied excerpt ends before explaining the mechanism in detail. Datadog’s core recommendation is to combine deterministic, tenant-aware routing with carefully coordinated assignment changes. This narrows the scope of deduplication while reducing storage overhead and enabling balanced, scalable exactly-once ingestion.

Read original

Load Balancing Database Design Apache Kafka Distributed Systems+4

datadogApr 13, 20221 min readCurated summary

It's always DNS . . . except when it's not: A deep dive through gRPC, Kubernetes, and AWS networking | Datadog

The supplied content does not include the blog post itself; it mainly contains Datadog navigation links and a promotional banner stating that Datadog was named a Leader in Gartner’s Magic Quadrant for Observability Platforms. The URL suggests the missing article concerns a gRPC, DNS, and load-balancing incident, but no incident details are provided. ## Available Content - Datadog promotes its recognition as a Gartner Magic Quadrant Leader. - The navigation lists products across: - Infrastructure and application monitoring - Logs, databases, and data observability - Security - Digital experience monitoring - Software delivery and service management - AI and platform capabilities - The linked page path references an engineering post titled “gRPC, DNS, and Load Balancing Incident.” ## Missing Technical Details - No description of the incident or its impact - No explanation of the DNS or load-balancing failure - No timeline, root-cause analysis, or remediation steps - No lessons learned or recommendations Please provide the full article text for a substantive technical summary.

Read original

Load Balancing Observability Grpc Dns+1