spotifyJul 20, 2026

Content Ingestion & Podcast Video Incident Report | Spotify Engineering (opens in new tab)

Spotify’s June 24 publishing incident delayed video podcast episodes for several hours after transcoding capacity was overwhelmed. The disruption resulted from insufficient capacity headroom, a concurrent batch job, increased processing costs, and a scheduling bug that reduced throughput by about 10%. Spotify cleared the backlog by the following morning and has since expanded capacity, improved monitoring, and launched a broader publishing reliability initiative. ## What Caused the Delay - A spike in new video podcast submissions pushed transcoding infrastructure close to its maximum capacity. - A scheduled batch job was reprocessing existing episodes, consuming capacity needed for new publications. - Recent video-quality improvements increased the processing time and compute required per episode. - A resource-scheduling bug following an infrastructure migration underused available hardware, reducing throughput by approximately 10%. - Creators re-uploaded missing episodes, unintentionally adding more load because Spotify did not clearly confirm that uploads had been received and queued. ## Incident Timeline and Response - Early monitoring alerts fired at 13:30 UTC, but they were not initially recognized as a broader capacity problem. - At 15:00, a delivery spike brought transcoding close to capacity. - The batch job was stopped at 16:35 to free resources. - Formal incident response began at 17:34 after queue thresholds were exceeded. - A scheduling fix was deployed at 20:49, and an additional processing cluster came online at 00:14 on June 25. - All queues were cleared by 01:02, with normal pipeline operation confirmed at 07:30. - Approximately four hours passed between the first alerts and formal incident response, highlighting a major monitoring and escalation gap. ## Remediation Measures - Spotify increased transcoding capacity by approximately 67% to provide more room for traffic spikes and batch processing. - The resource-scheduling bug was fixed, restoring use of previously underutilized compute capacity. - Monitoring was improved to warn earlier when capacity is approaching operational limits. ## Broader Reliability Program - Capacity planning is being expanded to account for burst traffic, background workloads, and recovery needs—not only typical steady-state demand. - Publishing systems will prioritize real-time creator uploads over background operations. - Spotify is extending rate limiting and backpressure mechanisms across the pipeline so unexpected load can be handled more gracefully. - A dedicated cross-team effort is coordinating these improvements across the publishing infrastructure. Spotify’s main lesson is that reliable publishing requires both additional capacity and better operational safeguards. Earlier detection, clearer upload acknowledgments, stronger workload prioritization, and deliberate burst-capacity planning should reduce the likelihood and impact of similar incidents.

video-transcodingmonitoringincident-responserate-limiting+4

metaMar 2, 2026

FFmpeg at Meta: Media Processing at Scale (opens in new tab)

FFmpeg is central to Meta’s media infrastructure, running tens of billions of times daily. Meta’s heavily modified internal fork became difficult to maintain as upstream FFmpeg evolved, so the company collaborated with the FFmpeg community to upstream key capabilities. Threaded multi-lane transcoding and real-time quality metrics now allow Meta to rely entirely on upstream FFmpeg for VOD and livestreaming. ## Moving Away from an Internal FFmpeg Fork - Meta’s fork had diverged substantially from upstream while supporting specialized requirements. - Maintaining both the fork and newer open-source FFmpeg versions created: - Divergent feature sets - Difficult rebases - Increased regression risk - Collaboration with FFmpeg developers, FFlabs, and VideoLAN enabled the needed functionality to be integrated upstream. ## More Efficient Multi-Lane Transcoding - Meta creates multiple DASH encodings for each uploaded video, varying resolution, codec, framerate, and quality. - Running separate FFmpeg processes wastes resources by repeatedly decoding the same source and starting multiple processes. - A single FFmpeg command can decode once and send frames to multiple encoder instances. - Earlier FFmpeg versions still processed those encoders serially for each frame. - Improved parallel encoder threading, introduced from FFmpeg 6.0 and completed in FFmpeg 8.0, allows encoder instances to run concurrently. - The change reduces compute overhead across Meta’s more than one billion daily video uploads. ## Real-Time Quality Metrics for Livestreaming - Metrics such as PSNR, SSIM, and VMAF measure compression-related visual quality loss. - Traditional FFmpeg workflows calculate these metrics after encoding, which is unsuitable for live video. - Meta needed a decoder after each encoder to reconstruct compressed frames and compare them with the original frames during processing. - FFmpeg’s “in-loop” decoding, available beginning in FFmpeg 7.0, enables real-time per-lane quality measurement within one command. ## Deciding What to Upstream - Meta aims to upstream changes that provide broad value to FFmpeg users. - Infrastructure-specific patches are kept internal when they cannot be reasonably tested or used by the wider community. - Meta’s custom MSVP transcoding ASIC is integrated through FFmpeg’s standard hardware APIs, alongside NVIDIA, AMD, and Intel technologies. - Because external developers cannot access MSVP hardware, Meta maintains and validates those internal patches itself. ## Continued Investment in FFmpeg - Upstream improvements allowed Meta to retire its internal fork for all VOD and livestreaming pipelines. - Standardized hardware interfaces make it easier to combine specialized ASICs with software-based encoding and decoding. - Meta plans to continue contributing to FFmpeg to improve efficiency, reliability, codec support, and media experiences across the industry. Meta’s experience demonstrates that upstreaming broadly useful infrastructure improvements can reduce long-term maintenance costs while strengthening the entire FFmpeg ecosystem.

video-transcodingffmpegmedia-processingdash+4