spotifyJul 20, 2026

Content Ingestion & Podcast Video Incident Report | Spotify Engineering (opens in new tab)

Spotify’s June 24 publishing incident delayed video podcast episodes for several hours after transcoding capacity was overwhelmed. The disruption resulted from insufficient capacity headroom, a concurrent batch job, increased processing costs, and a scheduling bug that reduced throughput by about 10%. Spotify cleared the backlog by the following morning and has since expanded capacity, improved monitoring, and launched a broader publishing reliability initiative. ## What Caused the Delay - A spike in new video podcast submissions pushed transcoding infrastructure close to its maximum capacity. - A scheduled batch job was reprocessing existing episodes, consuming capacity needed for new publications. - Recent video-quality improvements increased the processing time and compute required per episode. - A resource-scheduling bug following an infrastructure migration underused available hardware, reducing throughput by approximately 10%. - Creators re-uploaded missing episodes, unintentionally adding more load because Spotify did not clearly confirm that uploads had been received and queued. ## Incident Timeline and Response - Early monitoring alerts fired at 13:30 UTC, but they were not initially recognized as a broader capacity problem. - At 15:00, a delivery spike brought transcoding close to capacity. - The batch job was stopped at 16:35 to free resources. - Formal incident response began at 17:34 after queue thresholds were exceeded. - A scheduling fix was deployed at 20:49, and an additional processing cluster came online at 00:14 on June 25. - All queues were cleared by 01:02, with normal pipeline operation confirmed at 07:30. - Approximately four hours passed between the first alerts and formal incident response, highlighting a major monitoring and escalation gap. ## Remediation Measures - Spotify increased transcoding capacity by approximately 67% to provide more room for traffic spikes and batch processing. - The resource-scheduling bug was fixed, restoring use of previously underutilized compute capacity. - Monitoring was improved to warn earlier when capacity is approaching operational limits. ## Broader Reliability Program - Capacity planning is being expanded to account for burst traffic, background workloads, and recovery needs—not only typical steady-state demand. - Publishing systems will prioritize real-time creator uploads over background operations. - Spotify is extending rate limiting and backpressure mechanisms across the pipeline so unexpected load can be handled more gracefully. - A dedicated cross-team effort is coordinating these improvements across the publishing infrastructure. Spotify’s main lesson is that reliable publishing requires both additional capacity and better operational safeguards. Earlier detection, clearer upload acknowledgments, stronger workload prioritization, and deliberate burst-capacity planning should reduce the likelihood and impact of similar incidents.

content-processingmonitoringincident-responserate-limiting+4

dropboxJul 20, 2026

How our universal content processing platform Riviera evolved for AI and beyond (opens in new tab)

Riviera evolved from Dropbox’s preview-generation service into a shared content-processing platform used by products including Search, Replay, Sign, and Dash. Its core insight was to compose reusable transformations rather than build separate pipelines for every file type and output. As AI increased demand for consistent document extraction and preparation, Dropbox expanded Riviera’s capabilities and began offering them through APIs and Model Context Protocol tools. ## The Preview Problem - Dropbox supports more than 300 file formats, each requiring outputs such as: - Thumbnails - Full previews - Extracted text - Streaming manifests - Metadata - Building a separate service for every format and output would duplicate logic, dependencies, and operational work. - Configurations and package versions could drift across services, making the system harder to maintain and scale. ## Reusable Transformations as the Foundation - Riviera treats previews as sequences of smaller, reusable transformations. - For example, a PowerPoint preview can be produced by: - Converting the presentation to PDF - Rendering each PDF page as an image - The same PDF-to-image transformation can support PDFs and other workflows requiring page images. - This approach enables new formats and products to reuse existing capabilities instead of starting from scratch. ## Separating Coordination from Execution - Riviera uses a central coordinator to: - Collect and validate requests - Compose transformation workflows - Cache responses - Dispatch jobs to backend workers - Each worker handles a specific transformation, creating a clear unit for maintenance and scaling. - The platform now includes more than 100 capabilities and performs hundreds of thousands of transformations per second. - New formats and transformations can generally be added as plugins without changing the core system. ## From Internal Service to Shared Platform - Other Dropbox teams quickly adopted Riviera when they discovered overlapping content-processing needs. - Machine learning teams reused preview thumbnails for image normalization, avoiding duplicate generation. - Search used Riviera to prepare documents for indexing, while Sign, DocSend, and Replay reused existing transformations. - Dropbox eventually opened the plugin model to product teams, allowing them to add capabilities while the Riviera team maintained the platform’s core architecture. - Replay particularly benefited from Riviera’s complex video transcoding and manipulation capabilities, accelerating product development from months to weeks. ## Supporting AI Workloads - Dash introduced greater demand for reliable document preparation before AI processing. - AI systems require content to be transformed into consistent, machine-readable representations, including: - Extracted text - Data from scanned pages - File metadata - Normalized versions of hundreds of file types - These are fundamentally content-transformation challenges rather than AI-model challenges. - Because Riviera already supported many formats and transformations, Dash could build on existing infrastructure instead of creating a separate document-processing system. ## Broader Availability - Dropbox is making Riviera’s capabilities available to external developers and design partners. - Access is provided through APIs and Model Context Protocol tools. - The platform is intended for applications such as content management, document automation, search indexing, and AI document processing. Riviera’s evolution demonstrates the value of a shared transformation platform: reusable workers reduce duplication, centralized coordination improves reliability, and each new capability benefits multiple products. For teams building content-heavy or AI-powered applications, using standardized transformation infrastructure can be more efficient than maintaining format-specific pipelines independently.

content-processingaimodel-context-protocoldistributed-systems+3