spotifyJul 20, 2026

Content Ingestion & Podcast Video Incident Report | Spotify Engineering (opens in new tab)

Spotify’s June 24 publishing incident delayed video podcast episodes for several hours after transcoding capacity was overwhelmed. The disruption resulted from insufficient capacity headroom, a concurrent batch job, increased processing costs, and a scheduling bug that reduced throughput by about 10%. Spotify cleared the backlog by the following morning and has since expanded capacity, improved monitoring, and launched a broader publishing reliability initiative. ## What Caused the Delay - A spike in new video podcast submissions pushed transcoding infrastructure close to its maximum capacity. - A scheduled batch job was reprocessing existing episodes, consuming capacity needed for new publications. - Recent video-quality improvements increased the processing time and compute required per episode. - A resource-scheduling bug following an infrastructure migration underused available hardware, reducing throughput by approximately 10%. - Creators re-uploaded missing episodes, unintentionally adding more load because Spotify did not clearly confirm that uploads had been received and queued. ## Incident Timeline and Response - Early monitoring alerts fired at 13:30 UTC, but they were not initially recognized as a broader capacity problem. - At 15:00, a delivery spike brought transcoding close to capacity. - The batch job was stopped at 16:35 to free resources. - Formal incident response began at 17:34 after queue thresholds were exceeded. - A scheduling fix was deployed at 20:49, and an additional processing cluster came online at 00:14 on June 25. - All queues were cleared by 01:02, with normal pipeline operation confirmed at 07:30. - Approximately four hours passed between the first alerts and formal incident response, highlighting a major monitoring and escalation gap. ## Remediation Measures - Spotify increased transcoding capacity by approximately 67% to provide more room for traffic spikes and batch processing. - The resource-scheduling bug was fixed, restoring use of previously underutilized compute capacity. - Monitoring was improved to warn earlier when capacity is approaching operational limits. ## Broader Reliability Program - Capacity planning is being expanded to account for burst traffic, background workloads, and recovery needs—not only typical steady-state demand. - Publishing systems will prioritize real-time creator uploads over background operations. - Spotify is extending rate limiting and backpressure mechanisms across the pipeline so unexpected load can be handled more gracefully. - A dedicated cross-team effort is coordinating these improvements across the publishing infrastructure. Spotify’s main lesson is that reliable publishing requires both additional capacity and better operational safeguards. Earlier detection, clearer upload acknowledgments, stronger workload prioritization, and deliberate burst-capacity planning should reduce the likelihood and impact of similar incidents.

rate-limitingmonitoringincident-responsevideo-transcoding+4

gitlabJul 8, 2026

How we used AI agents to migrate GitLab rate limiting (opens in new tab)

GitLab used a three-person engineering pod and AI agents to migrate 121 application-level rate-limit keys into a shared `labkit-ruby` implementation. The migration succeeded because humans retained ownership of architecture, scope, rollout decisions, and final review while agents handled mechanical coding, tests, and reviews. The main lesson was that disciplined workflows and meaningful observability mattered more than the agents themselves. ## Migration Setup - GitLab was consolidating two production rate-limiting systems: - `Gitlab::ApplicationRateLimiter`, with 121 keys - A separate Rack-level implementation - The target was a single observable, testable, and consistently operated implementation in `labkit-ruby`. - A three-person pod divided responsibilities across the monolith, the gem, architecture, and project scope. - AI agents: - Read project context - Drafted specifications - Implemented bounded changes - Wrote tests - Pre-reviewed merge requests - Humans controlled scope, architecture, rollout strategy, and final approvals. ## The Specification and Review Loop - The team followed a repeatable process: - Read the epic - Write a specification - Conduct adversarial review - Implement only after blockers were resolved - Verify with explicit evidence - Review the merge request adversarially - Escalate to human review - Merge - Adversarial review was limited to two resolution rounds before requiring human involvement. - The project produced 14 numbered specifications and more than 30 merge requests. - This structured loop made agents useful on legacy code without allowing them to make high-impact decisions independently. ## Successful Rollouts - The first cohort covered five heavily used keys, including: - `pipelines_create` - `notes_create` - `user_sign_in` - Rollout progressed from 1% to 10%, 50%, and finally 100% over two days. - Engineers compared the old and new implementations during rollout and deliberately generated traffic to test behavior above the configured limits. - The second cohort consolidated 95 call sites: - 83 in the monolith - 12 in Enterprise Edition - Agents were especially effective at this repetitive, large-scale codebase work, avoiding roughly 95 individual feature-flag changes and 190 YAML edits. ## Observability and Shadow-Mode Failure - During Cohort 2, an adapter dropped an identifier on an unauthenticated path by incorrectly packing three strings into two primitive slots. - Some users briefly received generic failures when enforcement began. - Shadow comparison had detected divergence, but the dashboards did not distinguish structural identifier collisions from ordinary disagreements. - The team disabled enforcement immediately and shipped a short-term fix two days later. - The deeper cleanup will replace array-based scopes with named characteristics when calling `ApplicationLimiter`. - The incident showed that having observability is insufficient if it cannot identify the failure modes that require action. ## Missed Rate Limits and Infrastructure Constraints - An audit revealed that the original five-cohort plan had missed 17 of the 121 keys. - The omissions included: - Enterprise-only limits - Registry entries - Webhook keys - `partner_*` sub-second limits - Orphaned adapter rows - The team had not maintained a complete inventory count, making it possible for keys to become effectively invisible. - A sixth cohort was added to cover the missed cases. - Redis capacity also became a constraint: - The rate-limiting service used a four-shard cluster. - `maxclients` was increased incrementally. - Rollout stopped at 75,000 connections rather than 100,000 because primary CPU usage approached saturation. - Redis command execution was limited by one core per primary, leaving no simple vertical scaling solution. ## How AI Changed the Work - Agents made code generation faster, shifting the bottleneck to: - Human review capacity - Rollout judgment - Operational monitoring - Reviewer and operator attention - Agent collaboration was not always efficient; engineers sometimes spent longer guiding agents than they would have spent coding directly. - Engineers also had to develop new skills for specifying, reviewing, and correcting agent-generated work. - Agents could execute a request mechanically—such as creating dozens of feature flags—but could not decide whether that design was appropriate. - Human judgment remained essential for simplifying the rollout and avoiding unnecessary per-key flags. ## Outcome - By mid-June, all six cohorts had reached 100%. - All 121 application rate-limit keys were running through the new framework. - The migration demonstrated that AI agents can safely support complex legacy-system changes when paired with bounded tasks, adversarial review, gradual rollouts, complete inventories, and failure-specific observability. A practical recommendation is to use agents for repetitive implementation and verification, but keep architecture, risk assessment, rollout control, and operational decisions firmly with experienced humans.

rate-limitingai-agentobservabilitygitlab-duo-agent-platform+4

cloudflareJun 5, 2026

Your AI bill is out of control. Cloudflare can fix it now. (opens in new tab)

Cloudflare argues that uncontrolled AI spending stems from shared API keys, poor attribution, and defaulting to the most expensive models. Its AI Gateway now adds dollar-based spend limits, while a closed beta combines Cloudflare Access with identity-based budgets and routing. The goal is to make AI costs visible, attributable, and controllable without disrupting workflows. ### The AI Spending Problem - Companies encouraged aggressive AI adoption before establishing budgets or monitoring. - Shared API keys make it difficult to identify whether costs came from employees, teams, training pipelines, or runaway CI jobs. - Without routing rules or spending limits, users often choose frontier models for tasks that could use cheaper alternatives. - Visibility and controls are necessary to measure AI ROI and prevent waste. ### What AI Gateway Provides AI Gateway sits between applications and providers such as OpenAI, Anthropic, and Google. - Unified billing across providers and models. - Centralized logging of requests, token counts, and costs. - Response caching and rate limiting. - Content guardrails that can block PII and secrets before they reach models. - Previously, however, it lacked detailed attribution and budget controls. ### Dollar-Based Spend Limits - Budgets are measured in dollars rather than tokens and track cumulative request costs in real time. - Limits can be scoped by: - Model - Provider - Custom attributes such as user, team, or application - Budgets support daily, weekly, or monthly windows, including fixed and rolling periods. - When a limit is reached, requests can be blocked by default. - Dynamic Routes can instead redirect requests to a cheaper fallback model. - Spend limits are available in open beta across all AI Gateway plans through the dashboard or API. ### Identity-Driven Attribution and Policies Cloudflare Access can attach verified identity information to AI Gateway requests. - Authenticated user identities are extracted from Access JWTs and added as request metadata. - Organizations can track usage by employee, team, identity-provider group, or service. - Per-user budgets can give different spending limits to individual contributors and senior engineers. - Team policies can control which models different groups may use. - Access service tokens provide named identities for CI/CD pipelines and autonomous agents. - Logs include details such as email address, IdP group, or service-token name for export to analytics systems. - Configuration uses Cloudflare Access applications and existing identity-provider groups, without requiring custom Workers or manual JWT parsing. ### Cloudflare’s Internal Use - Cloudflare routes millions of requests and billions of tokens through AI Gateway each month. - It uses identity metadata to attribute costs, understand team usage, and manage budgets. - The company is making this internal approach available through the closed beta for identity-driven budgets and policies. Organizations should route AI traffic through a gateway, assign verified identities, set dollar-based budgets, and use model fallback rules to balance cost control with developer productivity.

rate-limitingci-cdoauthcloudflare-access+4

cloudflareMar 11, 2026

Slashing agent token costs by 98% with RFC 9457-compliant error responses (opens in new tab)

Cloudflare argues that HTML error pages are poorly suited to AI agents, wasting tokens while offering little actionable guidance. It now provides RFC 9457-compliant Markdown and JSON responses for Cloudflare-generated errors, including instructions on whether to retry, wait, stop, or escalate. The structured formats reduce payload size and token usage by more than 98% compared with HTML. ## Problems with HTML Error Pages - Cloudflare errors often result from customer policies or edge conditions, including: - DNS or host configuration problems - WAF, geographic, ASN, or bot restrictions - Rate limits - Traditional responses contain large amounts of HTML, CSS, and human-oriented text. - Agents may struggle to identify: - What went wrong - Whether retrying is appropriate - How long to wait - Whether human intervention is required - Custom Error Rules are configuration-dependent and therefore cannot provide a consistent contract across websites. ## RFC 9457 Structured Responses - Cloudflare now supports structured responses for all 1xxx-class errors. - Clients can request: - `Accept: text/markdown` - `Accept: application/json` - `Accept: application/problem+json` - Markdown responses include: - YAML frontmatter with machine-readable fields - “What happened” and “What you should do” guidance - JSON responses expose the same information as a flat object. - Support for Cloudflare-generated 4xx and 5xx errors is planned next. ## Machine-Readable Error Guidance Responses include stable fields such as: - `error_code`, `error_name`, and `error_category` for classification - `retryable` and `retry_after` for automated backoff - `owner_action_required` to indicate when the agent should stop or escalate - `ray_id`, `timestamp`, and `zone` for debugging and support This allows agents to implement durable control flow without scraping HTML or interpreting changing prose. ## Compatibility with RFC 9457 Cloudflare’s JSON format follows the standard Problem Details structure: - `type`: Documentation URL for the error - `status`: Actual HTTP status code - `title`: Short problem description - `detail`: Explanation of the specific occurrence - `instance`: Identifier for the individual error, corresponding to the Ray ID Cloudflare-specific operational fields are included as RFC 9457 extension members. Clients that do not recognize those fields can safely ignore them. ## Example: Rate Limiting A Cloudflare 1015 response identifies the request as rate-limited and includes: - HTTP status `429` - Error code `1015` - Category `rate_limit` - A description stating that the website owner’s configuration caused the limit - A Ray ID and timestamp - Retry-related guidance for implementing backoff The change is automatic across Cloudflare’s network. Browsers continue receiving HTML, while agents receive structured instructions when they explicitly request Markdown or JSON.

rate-limitingcloudflaremarkdownjson+3

pinterestFeb 24, 2026

Piqama: Pinterest Quota Management Ecosystem (opens in new tab)

Piqama is Pinterest’s generic quota management ecosystem for controlling physical resources, service limits, and application-specific capacity. It centralizes quota definition, validation, authorization, distribution, enforcement, usage tracking, and optimization while allowing individual applications to customize implementation details. Its integrations demonstrate how the same platform can support both capacity management for Big Data and rate limiting for online services. ## Platform Architecture - Provides a centralized management portal accessible through REST and Thrift. - Supports multiple quota types and platforms. - Applications may use Piqama’s default enforcement mechanisms or supply their own. - Manages quotas throughout their lifecycle, from creation and updates to usage feedback and optimization. ## Quota Lifecycle Management - **Schema management:** Defines quota identifiers and hierarchical relationships, such as workloads within projects. - **Validation:** Supports pluggable schema and semantic validation, including remote checks to ensure quotas do not exceed cluster capacity. - **Authorization:** Requires ownership-based authorization for quota updates and deletions; owners may be individuals or groups. - **Update dispatch:** Can distribute changes through Piqama clients, Pinterest’s PinConf system, or custom dispatchers. - **Enforcement:** Default clients can make real-time decisions, such as serving or dropping requests when usage exceeds limits. - Applications can customize schema handling, validation, update delivery, and enforcement logic. ## Governance and Auto-Rightsizing - Piqama clients collect quota enforcement and usage statistics transparently. - Non-client applications can submit data through system-based or storage-based feedback loops. - Data is stored in Apache Iceberg on Amazon S3 using predefined schemas and pre-aggregation to reduce storage costs. - An independent rightsizing service consumes historical data from Presto, Iceberg, and other sources. - Rightsizing strategies account for organic growth, traffic bursts, and underutilization. - Pinterest has developed a capacity-quota strategy intended to maximize resource allocation without saturating Big Data systems. ## Quotas and Budgets - Budgets assign dollar amounts to organizations, teams, or projects, while quotas define the resources available within those financial constraints. - Chargeback systems convert resource consumption into costs. - Projects that exceed their budgets may receive reduced resource allocations based on their tier. - Teams may need additional funding or workload prioritization when resources are restricted. - Piqama is expected to integrate further with Pinterest’s Entitlement system. ## Capacity-Based Quotas in Big Data - Pinterest’s Moka platform uses Apache YuniKorn to schedule batch-processing resources such as memory, CPU, and GPUs. - Piqama manages project-level quotas including: - Guaranteed memory and vcore allocations. - Maximum memory and vcore consumption. - Maximum concurrent applications. - Quota values are generated through: - **Auto-rightsizing:** Uses historical usage within a sliding window to estimate future needs. - **Manual adjustments:** Allows development teams to make immediate quota changes. - Pinterest is also developing a budget-based method for generating quota values. Piqama provides a flexible foundation for governing resource consumption across Pinterest. Organizations adopting it can combine centralized policy and visibility with application-specific enforcement, while usage data enables more efficient and financially aligned quota allocation.

rate-limitingamazon-s3apache-icebergresource-scheduling+4

airbnbOct 9, 2025

From Static Rate Limiting to Adaptive Traffic Management in Airbnb’s Key-Value Store (opens in new tab)

Airbnb evolved Mussel’s QoS system from static, per-client QPS limits into adaptive traffic management designed to maximize goodput. The newer approach accounts for the actual cost of requests, prioritizes critical workloads under stress, and detects hot keys or attack traffic before they overwhelm storage. Together, resource-aware quotas and real-time load shedding provide stronger protection against traffic spikes, uneven workloads, and DDoS-like bursts. ## Why Static QPS Limits Fell Short - Mussel is a multi-tenant key-value store serving millions of point and range reads across Airbnb. - Its original Redis-backed limiter assigned each client a fixed requests-per-second quota. - Requests exceeding the quota received HTTP 429 responses. - This model worked when backend effort roughly matched request count. - As usage grew, it could not account for: - The difference between a cheap one-row lookup and a 100,000-row scan. - Hot keys accessed by many clients simultaneously. - Localized storage-shard overload that affected unrelated traffic. - Sudden events such as bot floods, DDoS attacks, or large uploads. ## Resource-Aware Rate Control - Mussel replaced raw request counting with request units (RU), which represent estimated backend work. - RU calculations incorporate: - Fixed per-request overhead. - Rows and payload bytes processed. - Request latency, which distinguishes cached operations from disk-heavy ones. - The system uses calibrated linear formulas for reads and writes, with weights based on compute, network, and disk-I/O measurements. - Dispatchers debit a local token bucket according to each request’s RU cost rather than charging every request equally. - Periodic RU refills preserve simple, static quotas while making them more proportional to actual resource consumption. - Requests are rejected with HTTP 419 when the RU bucket is exhausted. - Load shedding remains separate, allowing latency-based protection to react dynamically without changing the underlying quota-refill mechanism. ## Load Shedding Under Sudden Stress - RU rate limiting smooths normal traffic but may react too slowly to rapidly changing workloads. - Mussel adds a load-shedding layer based on: - Traffic criticality. - A real-time latency ratio. - A CoDel-inspired queue-management policy. - Each dispatcher compares long-term p95 latency with short-term p95 latency. - A ratio near 1.0 indicates stable performance; a drop toward 0.3 signals rapidly increasing latency. - When stress crosses the threshold: - The system raises the effective RU cost for a designated lower-priority client class. - That class’s token bucket drains faster, causing its traffic to back off. - If conditions worsen, the penalty expands to additional classes. - Critical workloads, such as customer support and trust-and-safety traffic, can remain responsive while less important traffic is reduced. - The latency estimate uses the constant-memory P² algorithm, avoiding raw sample storage and cross-node coordination. ## Hot-Key Detection and DDoS Protection - Client-level quotas cannot prevent overload when many clients request the same popular key. - Mussel therefore detects skewed access patterns in real time. - When duplicate requests target a hot key, the system can protect storage by: - Serving responses from cache. - Coalescing identical requests before they reach the backend. - This approach protects the underlying shard whether the traffic comes from legitimate popularity, automation, or a DDoS burst. Mussel’s experience suggests that mature multi-tenant services should move beyond fixed QPS limits. Combining resource-based accounting, priority-aware load shedding, and hot-key mitigation provides a more effective way to preserve reliability while maximizing useful work during unpredictable traffic conditions.

rate-limitingk8sdistributed-systemsredis+4

figmaApr 11, 2017

An alternative approach to rate limiting | Figma Blog (opens in new tab)

Figma built a Redis-backed rate limiter to protect its web application from excessive traffic and spam. The system needed to work across multiple servers, add minimal latency, remove stale data efficiently, remain accurate, and use little memory. Common algorithms each met some of these goals, but introduced trade-offs involving atomicity, burst behavior, or memory consumption. ## Requirements and Redis - Rate limits cap requests from a user or IP within a time period. - Figma needed shared state because its application ran on multiple machines. - Redis was preferred over PostgreSQL because it provides: - Faster in-memory reads and writes - Built-in expiration for stale tracking data - Efficient storage for rate-limit state ## Token Bucket - Stores each user’s last-request timestamp and remaining token count in a Redis hash. - Tokens refill over time; a request is rejected when no tokens remain. - It is memory-efficient and conceptually elegant. - Its read-then-write operations are not atomic: - Two servers can read the same remaining token count. - Both may accept a request even though only one token was available. - Redis locks could prevent this race but would slow concurrent requests and add complexity. - Lua scripting could make the operations atomic, but Figma avoided introducing that complexity. ## Fixed Window Counters - Stores a request count for each user and fixed time interval, such as one Redis key per minute. - Each request atomically increments its interval’s counter. - Keys expire after the interval, preventing stale data from accumulating. - The approach is simple, memory-efficient, and avoids the token bucket’s race condition. - Its major flaw is boundary bursts: - With a five-request-per-minute limit, a user could send five requests at the end of one minute and five more immediately afterward. - This allows up to twice the intended traffic over a short sliding period. ## Sliding Window Logs - A sliding window log records the timestamps of individual requests. - Older timestamps can be removed as the window advances, and the remaining entries provide an accurate count. - This avoids the boundary problem of fixed windows. - The trade-off is memory usage: users making many requests require many timestamps to be stored. ## The Central Trade-off - Token buckets use little memory but require careful handling of distributed atomicity. - Fixed counters are atomic and efficient but can permit bursts at window boundaries. - Sliding logs are accurate but consume more memory. - Figma’s rate limiter was designed around balancing these competing concerns rather than choosing the theoretically simplest algorithm. The practical lesson is to select a rate-limiting strategy based on the required accuracy, concurrency model, storage system, and memory budget. Redis is a strong fit for distributed rate limiting, but the algorithm must account for both race conditions and burst behavior.

rate-limitingpostgresqldistributed-systemsredis+2