Smpte 2022 7

1 posts

netflix3 min readCurated summary

The Human Infrastructure: How Netflix Built the Operations Layer Behind Live at Scale

Netflix’s live-streaming growth required more than resilient technology—it demanded a dedicated human and physical operations layer. In three years, Netflix expanded from one live show per month to roughly 70 events in March 2026, including a World Baseball Classic game watched concurrently by more than 9.6 million accounts. The company evolved from engineers operating improvised setups to specialized teams, permanent facilities, and standardized broadcast procedures designed for continuous global scale. ## From Improvised Launches to Global Scale - Netflix’s first live events in March 2023 were operated by the engineers who built the streaming pipeline. - There was no dedicated operations team, formal command center, or live-specific incident response process. - Engineers monitored dashboards on laptops, coordinated through Slack, and troubleshot while millions of members watched. - Temporary control rooms were assembled in conference rooms, while larger events used rented broadcast facilities and equipment. - By March 2026, Netflix was operating 24/7 from facilities in Los Gatos and Los Angeles, with international coverage from Tokyo. - The company streamed approximately 70 events in that month—nearly as many as it had streamed throughout all of 2024. ## The Broadcast Operations Center - The Broadcast Operations Center (BOC) is Netflix’s physical command center for live events. - It receives the fully produced feed from a venue and hands it off to Netflix’s streaming infrastructure. - BOC responsibilities include: - Signal ingest and inspection - Audio and video conditioning - Closed-caption validation - Graphics insertion - Advertising management - A hub-and-spoke design, dual internet circuits, and SMPTE 2022-7 seamless switching reduce dependence on venue-specific infrastructure. - Centralizing these functions makes live events more repeatable and resilient. ## Protecting the Venue Signal - Netflix requires three completely independent transmission paths for every show-critical feed. - Approved contribution methods are prioritized as follows: - Dedicated video fiber - Single-feed satellite links - Dedicated enterprise-grade internet - SRT contribution systems - Production trucks must use redundant routers and transmission hardware, including separate router line cards. - Transmission equipment requires two independent power sources, UPS battery protection, and surge conditioning. - Before each event, operators conduct FACS/FAX facilities checks, including: - Audio/video synchronization tests - Latency and quality testing - Closed-caption verification - Backup switcher validation ## The Evolution of Netflix’s Operations Teams ### Phase 1: All-Hands Engineering - Core software engineers configured, launched, monitored, and dismantled every live event. - This approach worked for early broadcasts but could not scale as event volume increased. - Requiring developers to manually operate each show limited their ability to build new platform capabilities. ### Phase 2: Specialized Engineering Teams - Streaming Operations Engineers (SOEs) took responsibility for configuring and supporting events on the live-streaming pipeline. - SOEs became the first escalation point, allowing core developers to focus on platform development. - Broadcast Operations Engineers (BOEs) were later added to manage physical broadcast facilities and hardware. - BOEs oversee facility-related issues and support all shows running during a shift. ### Phase 3: The Co-Pilot Control Room - Dedicated Broadcast Control Operators (BCOs) assumed responsibility for operating the audio and video feeds. - Two BCOs worked together in a “first captain/second captain” model similar to a pilot and co-pilot. - This arrangement provided strong focus and execution quality for one or two events per day. - It became too space- and labor-intensive when Netflix began targeting up to ten simultaneous events. Netflix’s experience shows that live streaming at global scale depends on integrating broadcast discipline with software-engineering expertise. The key recommendation is to treat operations, redundancy, facilities, and specialized human roles as core parts of the product—not as temporary support added after the technology is built.

Read original(opens in new tab)