netflix·May 29, 2026·3 min readCurated summary

From Silos to Service Topology: Why Netflix Built a Real-Time Service Map

Netflix built Service Topology to give engineers a real-time, unified view of dependencies across its thousands of microservices. Traditional metrics, logs, and traces provide isolated signals but do not reveal the broader service relationships needed to diagnose failures or assess blast radius. The system combines multiple dependency sources into a living map that supports fast, context-rich troubleshooting. ## The Observability Problem - Netflix’s distributed architecture involves thousands of services and complex chains of calls for actions such as playback, authentication, recommendations, and optimization. - During incidents, engineers need to determine: - Which services depend on one another - What the potential blast radius is - Whether a failure originates locally or upstream - Existing observability tools show symptoms, logs, or individual request paths, but not the complete steady-state topology. - Manually combining information from different tools is slow and error-prone, especially during urgent incidents. ## Why Real-Time Service Mapping Matters - Frequent deployments and changing traffic patterns make static architecture diagrams quickly obsolete. - Netflix’s Live programming and advertising-supported plans increase the need for rapid diagnosis and operational awareness. - Engineers repeatedly asked about dependencies, failures, maintenance impact, unknown metrics, and recent call-path changes. - These recurring questions demonstrated the need for accurate, near-real-time dependency information. ## Lessons from Earlier Approaches - Netflix evaluated vendor platforms, graph databases, and internal prototypes before developing Service Topology. - Key lessons included: - Dependency data must update in near real time. - Storage and query systems must operate at Netflix’s scale. - The solution should integrate with existing observability workflows. - Incorrect or incomplete topology data can mislead engineers during incidents. - No single data source captures every aspect of service relationships. ## Requirements for a Living Map Service Topology was designed to provide: - Real-time updates as services deploy and dependencies change - Sub-second queries for traversing service call graphs - Both network-level and application-level views - Context such as health, availability tiers, ownership, and business domains - A visual interface for engineers and programmatic APIs for automation, resilience systems, and blast-radius analysis ## Combining Multiple Sources of Truth Netflix separates dependency information into physically distinct graphs so each layer can evolve and be queried independently. When a unified view is requested, the system traverses the layers in parallel and merges the results to maintain fast response times. ### eBPF Network Flows - eBPF captures network activity at the kernel level, recording which services communicate over the network. - This provides broad coverage, including services that lack application instrumentation. - It supports both cluster-level and application-level topology. - Its limitation is that network traffic alone does not provide application-specific context, such as the APIs or endpoints involved. Netflix’s approach is to combine complementary perspectives rather than rely on a single imperfect dependency source, producing a more complete and actionable service map.

Read original(opens in new tab)

ObservabilityDistributed SystemsMicroservicesEbpf+2

netflix·May 8, 2026·4 min readCurated summary

Scaling ArchUnit with Nebula ArchRules

Netflix’s Nebula ArchRules extends ArchUnit so architectural and API-lifecycle rules can be shared across thousands of Gradle repositories. Unlike AST-based tools, ArchUnit analyzes compiled JVM bytecode, supports multiple JVM languages, and offers a type-safe Java API for authoring and testing rules. The approach helps identify unsafe API usage, technical debt, and deviations from Netflix’s preferred development practices at fleet scale. ## The API Lifecycle Problem - Netflix operates tens of thousands of Java repositories in a polyrepo environment. - A library incident involving a backwards-incompatible change highlighted the difficulty of deciding when deprecated APIs can safely be removed. - Netflix introduced lifecycle annotations: - `@Deprecated` for APIs scheduled for removal - `@Public` for APIs intended for downstream use - `@Experimental` for APIs that may change - Unannotated APIs are treated as internal - The remaining challenge was identifying downstream projects that use internal, experimental, or deprecated APIs incorrectly. - The same tooling could support large migrations, such as major Spring Boot upgrades. ## Why ArchUnit - ArchUnit is an open-source library commonly used within JUnit suites to enforce architectural rules. - It is built on ASM and analyzes compiled JVM bytecode rather than source syntax. - Its main strengths are: - Cross-language JVM support for Java, Kotlin, Scala, and other JVM languages - A fluent builder API for readable rule definitions - A lower-level API for complex custom analysis - Access to class relationships, dependencies, and call sites through its class graph - Standard ArchUnit is primarily designed for one repository, so Netflix created Nebula ArchRules to distribute rules across many Gradle projects. ## Bytecode Analysis vs. AST Analysis - AST-based tools such as PMD inspect source-code structure and can be sensitive to language-specific syntax and syntactic sugar. - Supporting multiple JVM languages may require separate rules for each language. - ASM analyzes the bytecode that will actually execute, regardless of how the source was written. - This makes rules more consistent across Java, Kotlin, Scala, and other JVM languages. ## Rule Authoring - Tools such as PMD and SpotBugs are generally optimized for built-in rules or third-party plugins rather than custom rule development. - PMD custom rules may require difficult-to-maintain XPath expressions and separate tooling for testing. - ArchUnit rules are written as type-safe, fluent Java code. - Rules can be unit tested directly by passing them class references, without running a separate analysis process. - ArchUnit’s class graph provides contextual information about dependencies and call relationships, enabling more sophisticated checks. ## ArchRules Libraries - The Nebula ArchRules Library Plugin adds an `archRules` source set to a Gradle project. - A class implementing `ArchRulesService` exposes a `Map<String, ArchRule>`: - The map key names the rule. - The `ArchRule` defines the constraint using ArchUnit’s API. - Rule code and its dependencies are kept separate from the application’s main code. - Gradle publishes the rules in a separate JAR using the `arch-rules` classifier and an `arch-rules` usage attribute. - Downstream projects must use Gradle Module Metadata to resolve the rules variant. ## Standalone and Bundled Rule Libraries - Standalone rule libraries contain only `archRules` code. - They are useful for: - Enforcing rules around APIs the organization does not own - Checking usage of Java or open-source libraries - Applying generic rules, such as prohibiting use of deprecated APIs - Bundled rule libraries contain both normal library code and rules specific to how that library should be used. - Netflix maintains open-source standalone rule libraries as examples and reusable building blocks. Nebula ArchRules turns ArchUnit from a repository-local testing library into a reusable organization-wide policy mechanism. Teams can publish rules as Gradle artifacts and apply them consistently across JVM projects, making API governance, dependency policies, and architectural standards easier to enforce at scale.

Read original(opens in new tab)

KotlinJavaStatic AnalysisJvm+4

netflix·May 4, 2026·3 min readCurated summary

Democratizing Machine Learning at Netflix: Building the Model Lifecycle Graph

Netflix’s growing use of machine learning across personalization, Studio, payments, advertising, and other domains has created a fragmented ecosystem of tools and metadata. The Metadata Service (MDS) addresses this problem by building a Model Lifecycle Graph that connects models, features, pipelines, experiments, datasets, and ownership information. Its goal is to make ML assets discoverable, understandable, and reusable across organizational boundaries. ## A Fragmented Machine Learning Landscape - Netflix ML has expanded from personalization into areas such as: - Studio production and post-production - Fraud detection and payment optimization - Advertising and real-time targeting - Each domain uses different technologies, metrics, and organizational structures. - Valuable assets often remain isolated in specialized systems. - For example, Studio-generated content embeddings could support: - Contextual ad matching - Episodic merchandising - Recommendations based on tone, topic, or mood - Practitioners struggle to answer basic questions because relevant information is split across: - Model registries - Pipeline orchestrators - Experimentation platforms - Feature stores - Dataset systems - This fragmentation makes discovery, lineage tracking, impact analysis, and ownership difficult. ## The Challenge of Connecting ML Infrastructure - MDS must unify metadata from many independent systems, including: - Pipeline execution and transformation data - Model versions, artifacts, deployments, and staleness - A/B test configurations - Feature definitions and usage - Dataset creation and discovery - User, team, and organization information - These systems use different identifiers, formats, and conceptual models. - The core challenge is transforming heterogeneous metadata into a common entity model and connected graph—not merely creating a consolidated user interface. ## The Model Lifecycle Graph - Netflix’s Metadata Service indexes ML-related assets and materializes relationships between them. - It supports real-time metadata ingestion and cross-domain questions such as: - Which experiments use a particular model? - Which models depend on a feature? - What data sources feed a model? - Who owns each part of the workflow? - The graph is intended to make every ML asset discoverable and reusable regardless of its originating team or business domain. ## Core Concepts and Vocabulary - **Component:** Any uniquely addressable object identified by an AIP URI, such as: - `aip://model/registry/ranking-v5` - `aip://user/identity/alice` - `aip://pipeline/orchestrator/weekly-training` - **Entity:** A component enriched with properties such as name, description, creation date, and ownership. - **Entity type:** A group of entities sharing the same data shape and required properties. - **Domain:** An abstract interface for a category of ML assets, such as Models or Pipelines. - **Provider:** A concrete backend implementation of a domain, such as Netflix’s internal model registry. - Separating domains from providers allows multiple systems to implement the same interface without changing how consumers interact with MDS. - URI-based addressing gives services a consistent way to reference assets and resolve them to connected metadata. ## From Events to a Queryable Graph - MDS receives metadata events through Kafka and AWS SNS/SQS. - Source systems emit lightweight events containing an event type and resource identifier. - For example, a model registry might emit a `model_instance_created` event with the new instance’s ID. - This keeps event producers simple while allowing MDS to enrich events, construct entities, and infer relationships such as connections between models and A/B tests. The Model Lifecycle Graph provides Netflix with a common layer for connecting previously isolated ML systems. By standardizing identifiers, entities, domains, and providers, MDS can support cross-domain discovery, lineage, impact analysis, and collaboration at scale.

Read original(opens in new tab)

Machine LearningApache KafkaFeature StoreGraph Database+3

netflix·May 1, 2026·3 min readCurated summary

State of Routing in Model Serving

Netflix’s centralized ML serving platform provides a single, domain-independent API for model inference across personalized experiences and other use cases. Rather than exposing individual scoring functions, Netflix packages feature computation, preprocessing, inference, and postprocessing into self-contained model workflows. The core routing challenge is directing each request to the correct model version and serving cluster while keeping client services independent from model changes and infrastructure topology. ## Models as End-to-End Workflows - Netflix distinguishes **model serving** from traditional model inference: - Inference typically means `infer(features) -> score`. - Serving includes preprocessing, feature computation, optional trained components, and postprocessing. - Example workflows include: - Ranking titles for a personalized Continue Watching row using user, country, and device context. - Predicting payment fraud using user, country, and transaction details. - Models declare the facts they need, while the serving platform retrieves those facts from other microservices. - During offline training, Netflix’s ML fact store provides snapshots for bulk feature computation. - Calling services provide standard request context and domain-specific inputs, while the platform handles feature generation, model selection, and execution. ## Platform Design Principles - **Model innovation without client changes** - Client applications integrate with the platform once. - Model versions, A/B tests, additional experimental data, logging, and model selection remain hidden behind the platform API. - **Clients decoupled from model sharding** - Models run across multiple serving cluster shards, each with its own Virtual IP address. - Shard assignments can change based on traffic, SLAs, model architecture, and resource availability. - Clients should not need to track these VIP changes. - **Flexible traffic routing** - Routing must support A/B allocations, gradual traffic shifts, new model versions, new VIPs, and client-specific overrides. - Safe lifecycle management requires support for shadow deployments, canaries, rollbacks, and migrations. ## Switchboard: Context-Aware Routing - Generic API gateways and service-mesh proxies did not satisfy Netflix’s requirements. - Netflix needed: - Native integration with its experimentation platform. - gRPC support. - Routing based on rich, domain-specific request context. - Model-specific rollout and migration controls. - Netflix built **Switchboard**, a custom proxy layer handling more than one million requests per second. - Switchboard is the mandatory entry point for clients and: - Routes requests to the appropriate model based on request context. - Applies configured context enrichment before invoking the model. - Hides model locations and infrastructure changes from client services. ## Objective Abstraction - Every request must provide an **Objective**, an enumeration defined by the serving platform. - The excerpt introduces Objectives as a central abstraction for identifying the business purpose of a serving request, but the supplied text ends before describing its full roles. Netflix’s approach is to centralize routing, experimentation, and model execution behind one stable API. This allows client applications to evolve independently while researchers can iterate on models and safely manage large-scale production rollouts.

Read original(opens in new tab)

Database DesignMachine LearningPostgreSQLMicroservices+4

netflix·Apr 24, 2026·3 min readCurated summary

Scaling Camera File Processing at Netflix

Netflix built its Media Production Suite (MPS) to automate repetitive media workflows, improve consistency, and give filmmakers more time for creative work. Rather than develop an image-processing engine internally, Netflix partnered with FilmLight and integrated its FilmLight API (FLAPI) into Netflix’s cloud infrastructure. This combination provides reliable, camera-aware processing at global scale while supporting open standards, auditability, and rapid turnaround. ## Why Netflix Built MPS - Netflix productions use a wide range of cameras, formats, workflows, regions, and vendors. - File-based workflows created recurring problems: - Manual file wrangling reduced creative time. - Media handling varied between productions. - Human-driven processes were difficult to audit. - Teams repeatedly rebuilt similar workflows. - MPS aims to: - Standardize media management and movement from production through post-production. - Improve efficiency, consistency, and quality control. - Reduce errors and non-creative administrative work. ## Choosing FilmLight’s Processing Engine - Building a complete image-processing engine would require long-term collaboration with camera manufacturers and the broader industry. - Netflix needed a system that could: - Inspect, trim, and transcode camera-original files. - Preserve trusted color science and metadata. - Support many current and future camera formats. - Run within Netflix’s scalable, observable encoding infrastructure. - FilmLight’s Baselight and Daylight products already serve professional color grading, dailies, and transcoding workflows. - FLAPI allowed Netflix to use this proven processing technology as a backend API instead of duplicating it internally. ## Camera Metadata Inspection - Productions upload media with ASC Media Hash List (MHL) files to verify ingest completeness and integrity. - During the subsequent inspection phase, FLAPI: - Extracts metadata from original camera files. - Maps critical fields into Netflix’s normalized schema. - Makes the metadata searchable and reusable. - The metadata supports: - Matching footage by timing and reel name. - Automated retrieval. - Pipeline validation and troubleshooting. - Investigating why footage appears a certain way after processing. - Packaging FLAPI in Docker allows nearly identical deployments across Netflix’s cloud and global production compute environments. ## VFX Plates and Media Deliverables - MPS generates VFX plates and other outputs while preserving framing, color management, and camera-specific decoding behavior. - FLAPI is used to: - Debayer original camera files with format-appropriate parameters. - Crop and de-squeeze images according to ASC Framing Decision Lists. - Apply ACES Metadata Files for repeatable color workflows. - Produce deliverables in multiple formats. - The workflows are automated, repeatable, and auditable. - AMF files accompany OpenEXR outputs so recipients can identify which color transformations have already been applied. - Because the backend uses FilmLight technology, Netflix specialists can validate automated decisions in Baselight before production begins. ## Cloud-Native Media Processing - Traditional facilities often rely on powerful GPU systems and specialized high-performance storage. - Netflix instead designed its processing around the Cosmos compute and storage platform. - Cloud-compatible tools must: - Run as short-lived serverless functions in Linux Docker containers. - Operate effectively on CPU-only instances. - Support headless execution through Java, Python, or command-line interfaces. - Remain stateless so failed workers can be terminated and relaunched. - This model favors parallel processing across many workers rather than maximizing the power of one machine. - It improves cost and performance efficiency while maintaining production turnaround targets. - FLAPI’s API-driven, container-friendly, and low-state architecture made it straightforward for Netflix to integrate and operate reliably. Netflix’s approach demonstrates the value of combining established industry expertise with cloud-scale orchestration. By using FLAPI for specialized media processing and Cosmos for elastic execution, MPS can deliver consistent, traceable camera-file workflows without requiring Netflix to build and maintain every component itself.

Read original(opens in new tab)

Database DesignCloud ComputingDocker & ContainersImage Processing+4

netflix·Apr 17, 2026·3 min readCurated summary

The Human Infrastructure: How Netflix Built the Operations Layer Behind Live at Scale

Netflix’s live-streaming growth required more than resilient technology—it demanded a dedicated human and physical operations layer. In three years, Netflix expanded from one live show per month to roughly 70 events in March 2026, including a World Baseball Classic game watched concurrently by more than 9.6 million accounts. The company evolved from engineers operating improvised setups to specialized teams, permanent facilities, and standardized broadcast procedures designed for continuous global scale. ## From Improvised Launches to Global Scale - Netflix’s first live events in March 2023 were operated by the engineers who built the streaming pipeline. - There was no dedicated operations team, formal command center, or live-specific incident response process. - Engineers monitored dashboards on laptops, coordinated through Slack, and troubleshot while millions of members watched. - Temporary control rooms were assembled in conference rooms, while larger events used rented broadcast facilities and equipment. - By March 2026, Netflix was operating 24/7 from facilities in Los Gatos and Los Angeles, with international coverage from Tokyo. - The company streamed approximately 70 events in that month—nearly as many as it had streamed throughout all of 2024. ## The Broadcast Operations Center - The Broadcast Operations Center (BOC) is Netflix’s physical command center for live events. - It receives the fully produced feed from a venue and hands it off to Netflix’s streaming infrastructure. - BOC responsibilities include: - Signal ingest and inspection - Audio and video conditioning - Closed-caption validation - Graphics insertion - Advertising management - A hub-and-spoke design, dual internet circuits, and SMPTE 2022-7 seamless switching reduce dependence on venue-specific infrastructure. - Centralizing these functions makes live events more repeatable and resilient. ## Protecting the Venue Signal - Netflix requires three completely independent transmission paths for every show-critical feed. - Approved contribution methods are prioritized as follows: - Dedicated video fiber - Single-feed satellite links - Dedicated enterprise-grade internet - SRT contribution systems - Production trucks must use redundant routers and transmission hardware, including separate router line cards. - Transmission equipment requires two independent power sources, UPS battery protection, and surge conditioning. - Before each event, operators conduct FACS/FAX facilities checks, including: - Audio/video synchronization tests - Latency and quality testing - Closed-caption verification - Backup switcher validation ## The Evolution of Netflix’s Operations Teams ### Phase 1: All-Hands Engineering - Core software engineers configured, launched, monitored, and dismantled every live event. - This approach worked for early broadcasts but could not scale as event volume increased. - Requiring developers to manually operate each show limited their ability to build new platform capabilities. ### Phase 2: Specialized Engineering Teams - Streaming Operations Engineers (SOEs) took responsibility for configuring and supporting events on the live-streaming pipeline. - SOEs became the first escalation point, allowing core developers to focus on platform development. - Broadcast Operations Engineers (BOEs) were later added to manage physical broadcast facilities and hardware. - BOEs oversee facility-related issues and support all shows running during a shift. ### Phase 3: The Co-Pilot Control Room - Dedicated Broadcast Control Operators (BCOs) assumed responsibility for operating the audio and video feeds. - Two BCOs worked together in a “first captain/second captain” model similar to a pilot and co-pilot. - This arrangement provided strong focus and execution quality for one or two events per day. - It became too space- and labor-intensive when Netflix began targeting up to ten simultaneous events. Netflix’s experience shows that live streaming at global scale depends on integrating broadcast discipline with software-engineering expertise. The key recommendation is to treat operations, redundancy, facilities, and specialized human roles as core parts of the product—not as temporary support added after the technology is built.

Read original(opens in new tab)

Live StreamingBroadcast OperationsSignal IngestSmpte 2022 7+4

netflix·Apr 10, 2026·3 min readCurated summary

Evaluating Netflix Show Synopses with LLM-as-a-Judge

Netflix developed an LLM-as-a-Judge system to evaluate show synopses at the scale of its extensive catalog. The system assesses creative quality against expert-defined standards while also examining whether scores predict member behavior. With calibrated prompts, extended reasoning, and consensus scoring, the approach achieves more than 85% agreement with creative writers and can identify potentially impactful synopsis problems before a title launches. ## Defining a Good Synopsis - Synopsis quality is measured in two ways: - **Creative quality:** how well a synopsis follows Netflix’s editorial standards. - **Member feedback:** how the synopsis affects viewing decisions and early engagement. - Strong synopses help members quickly understand and choose titles. - Weak or misleading synopses can cause frustration, abandonment, and reduced viewing. ## Building Expert-Labeled Evaluation Data - Creative experts initially labeled roughly 1,000 diverse synopses. - Three writers scored each synopsis and explained their decisions. - Because the task was subjective, Netflix used eight calibration rounds to improve consistency. - Techniques that increased agreement included: - Replacing 1–4 ratings with binary scores. - Allowing writers to consult previous examples. - Maintaining a searchable taxonomy of recurring errors. - A model-in-the-loop process helped resolve disagreements: - Multiple writers supplied scores. - An LLM aggregated the judgments. - Writers reviewed cases with significant disagreement. - The resulting “golden set” contains about 600 synopses with criterion-level labels and explanations. ## Measuring Member Impact - Netflix uses two behavioral metrics: - **Take fraction:** how often members who see a synopsis start watching the title. - **Abandonment rate:** how often viewers stop shortly after beginning. - These metrics act as short-term proxies for long-term retention and have been validated through A/B testing. - Netflix evaluates whether LLM-generated quality scores can predict these engagement outcomes. ## Criterion-Specific LLM Judges - Initial prompts provide: - Relevant show metadata. - A summary of the applicable quality guidelines. - A request for an explanation followed by a binary score. - A single prompt covering every criterion performed poorly because it overloaded the model. - Separate judges for individual criteria performed better. - Binary outputs make evaluation straightforward using accuracy against the expert-labeled golden set. ## Improving Prompts and Reasoning - Netflix applies Automatic Prompt Optimization to a development set of about 300 examples. - Prompts are then manually refined with LLM assistance. - Performance varies significantly by criterion: prompts work well for areas such as precision but less well for subjective criteria such as clarity. - Inference-time scaling improves difficult judgments through: - **Longer rationales**, which give the model more room to reason. - **Consensus scoring**, which samples multiple judgments and combines their results. ## Tiered Rationales - Longer explanations generally improve accuracy, but they become harder for creative experts to read and audit. - Netflix therefore uses tiered rationales: - The model may reason at length internally. - It produces a concise explanation before the final score. - This approach preserves the benefits of extended reasoning while improving interpretability. - For example, the tone evaluator’s accuracy increased from 86.55% to 87.85% with tiered rationales. Netflix’s approach combines expert standards, calibrated evaluation data, specialized prompts, and inference-time reasoning to scale synopsis-quality review. The practical recommendation is to use LLM judges as carefully aligned evaluators—not generic critics—while validating their scores against both human judgment and real member behavior.

Read original(opens in new tab)

Database DesignLlm As A JudgeAutomatic Prompt OptimizationCreative Quality+4

netflix·Apr 6, 2026·4 min readCurated summary

Stop Answering the Same Question Twice: Interval-Aware Caching for Druid at Netflix Scale

Netflix’s Druid deployment now exceeds 10 trillion rows and can ingest 15 million events per second, but repetitive dashboard queries became a scaling problem. Its new experimental caching layer handles rolling time windows by reusing settled historical results and querying Druid only for recent, changing data. Netflix accepts up to five seconds of additional staleness in exchange for substantially lower query load. ## The Scaling Problem - A dashboard with 26 charts can issue 64 queries per load. - Viewed by 30 people and refreshed every 10 seconds, that becomes roughly 192 queries per second. - Druid’s full-result cache misses whenever a rolling time interval changes. - Druid avoids caching realtime segments to preserve result correctness and determinism. - Per-segment caching reduces historical scans but still requires brokers to gather and merge results for every request. - Adding hardware to handle this redundant workload would be prohibitively expensive. ## Caching Only the Unsettled Data - In a three-hour query, most data is already stable; only the newest minutes are likely to change. - The cache stores previously returned historical portions and sends Druid only the uncached interval. - This approach is designed for time-grouped queries such as timeseries and groupBy queries. ## Deliberate Staleness - The cache can make the newest data up to five seconds stale. - This is acceptable because dashboards typically refresh every 10–30 seconds. - Netflix’s pipeline already has up to roughly five seconds of latency at P90. - Many queries also intentionally end at `now-1m` or `now-5s` to avoid unstable, newly arriving data. ## Exponential TTLs - Cache lifetimes increase with the age of each data point because older data is less likely to change. - Data under two minutes old has a minimum TTL of five seconds. - After that, TTL doubles for each additional minute: - 10 seconds at two minutes old - 20 seconds at three minutes - 40 seconds at four minutes - TTLs are capped at one hour. - Fresh data is refreshed frequently to account for late-arriving events, while older data remains cached longer. ## Time-Based Bucketing - A single cache entry per query and interval would still miss whenever a rolling window shifted. - Netflix instead uses a map-of-maps: - The outer key is a hash of the query excluding its time interval. - Inner keys represent timestamps bucketed by query granularity or at least one minute. - Big-endian timestamp encoding preserves chronological order for efficient range scans. - A three-hour query at one-minute granularity becomes 180 independently cached buckets. - When the window moves, most buckets can be reused and only the newly exposed range must be fetched. ## Router-Integrated Cache Service - The cache currently operates as an external service behind the Druid Router. - Cacheable requests are intercepted transparently: - Fully cached requests are answered directly. - Partially cached requests are narrowed to the missing interval and sent to Druid. - Metadata queries and queries without time-based grouping bypass the cache. - The proxy can be enabled or disabled without changing clients. - Netflix views this as an interim design while exploring deeper integration with Druid. ## Query Identification and Lookup - Incoming queries are parsed to extract their interval, granularity, and structure. - A SHA-256 hash is generated from the query’s logical contents, including datasource, filters, aggregations, and relevant context properties, while excluding the time interval. - The cache looks for buckets within the requested range. - Lookup requires cached buckets to be contiguous from the beginning of the requested interval; the provided article text ends while explaining the handling of expired or missing buckets. Netflix’s approach is best suited to frequently repeated rolling-window dashboards where a small, slightly stale tail is acceptable. Segmenting results by time and assigning age-based TTLs allows the system to preserve freshness where it matters while eliminating most redundant Druid work.

Read original(opens in new tab)

Database DesignCachingQuery OptimizationData Ingestion+3

netflix·Apr 4, 2026·3 min readCurated summary

Powering Multimodal Intelligence for Video Search

Video search is difficult because it must combine many kinds of information—characters, scenes, dialogue, labels, and embeddings—across enormous volumes of footage. The post argues that solving this problem requires a distributed pipeline that separates reliable ingestion, computationally intensive data fusion, and low-latency search indexing. Temporal bucketing, hybrid ranking, and deduplication turn billions of model outputs into searchable moments for editors. ## Why Video Search Is Complex - Video contains multiple overlapping modalities, each analyzed by specialized models. - Models produce different outputs, including: - Text labels such as characters or objects - Scene classifications - High-dimensional embedding vectors - Time ranges with varying boundaries - Overlapping model timelines must be synchronized into a chronological representation. - A 2,000-hour archive may contain more than 216 million frames, expanding to billions of records after multimodal processing. - Search must avoid returning thousands of redundant clips from continuous shots. - Ranking therefore combines: - Symbolic text matching for precision and interpretability - Semantic vector similarity for contextual relevance - Clustering and deduplication to identify the best moments - Sub-second response times are essential because delays interrupt editors’ creative workflows. ## Three-Stage Ingestion and Fusion Pipeline ### Transactional Persistence - Raw model annotations are ingested through highly available pipelines. - Apache Cassandra stores the annotations with an emphasis on: - Data integrity - Distributed availability - High write throughput - An annotation can include a type, nanosecond time range, embedding vector, label, and confidence score. ### Offline Data Fusion - After persistence, Apache Kafka publishes an event that starts asynchronous processing. - The offline pipeline performs expensive temporal intersections without slowing ingestion or search. - Model outputs are normalized into fixed one-second time buckets. - The fusion process: - Maps continuous detections into discrete intervals - Intersects annotations sharing a bucket - Combines them into unified records - Writes the enriched records back to Cassandra - For example, a “Joey” character detection from seconds 2–8 can be combined with a “kitchen” scene detection from seconds 4–9 to create a fused record for the 4–5 second interval. - Each fused record retains links to the original annotations and source asset. ### Real-Time Search Indexing - Enriched buckets are later sent from Cassandra to Elasticsearch. - Upserts use a composite key consisting of the asset ID and time bucket. - If a bucket already exists, it is updated rather than duplicated. - This creates one consistent record for each second of footage while allowing new model results to be incorporated. The overall recommendation is to treat multimodal video search as a distributed data-fusion problem rather than a single-model retrieval task. Decoupling ingestion, offline processing, and indexing allows the system to handle massive archives while preserving reliable data capture and fast, context-rich search.

Read original(opens in new tab)

Database DesignApache KafkaElasticsearchApache Cassandra+4

netflix·Apr 2, 2026·3 min readCurated summary

Smarter Live Streaming at Scale: Rolling Out VBR for All Netflix Live Events

Netflix switched all Live events from Constant Bitrate (CBR) to capped Variable Bitrate (VBR), using AWS Elemental MediaLive’s QVBR setting. VBR allocates bits according to scene complexity, reducing delivery costs and improving playback quality, but its unpredictable spikes and dips invalidate traditional capacity-planning assumptions. Netflix addressed this by reserving delivery capacity according to each stream’s nominal bitrate rather than its current traffic level. ## Why Netflix Moved Live Streaming from CBR to VBR - CBR delivers streams near a fixed target, making server capacity and traffic patterns easy to predict. - However, CBR wastes bits on simple scenes and may provide insufficient bits for complex action. - VBR targets consistent visual quality instead: - Simple scenes use substantially fewer bits. - Complex scenes receive higher bitrate to prevent artifacts. - Netflix’s tests found: - Approximately 15% fewer bytes transferred on average. - Around 10% less traffic during the peak minute. - About 5% fewer rebuffers per hour. - Lower average traffic improves Open Connect scalability and can reduce startup delays and playback interruptions. ## Why VBR Creates Stability Risks - VBR bitrate can remain well below its nominal target during simple scenes, sometimes using only 2 Mbps for a 5 Mbps stream. - Delivery systems may interpret these low-traffic periods as spare server capacity and route additional sessions to the server. - When complex content appears—such as fights, confetti, rapid camera movement, or detailed crowds—bitrate can quickly rise to 6–8 Mbps or more. - If too many sessions were admitted during the low-bitrate period, aggregate traffic can exceed link or NIC capacity, causing: - Higher latency - Packet loss - Playback stalls - Quality downshifts ## Making Capacity Planning Aware of VBR - Netflix changed traffic-steering decisions so they no longer rely solely on current throughput. - Each stream reserves capacity based on its nominal bitrate, even when its current bitrate is much lower. - This treats every stream as capable of quickly returning to its expected capacity level. - The approach prevents servers from being overfilled during low-complexity scenes and keeps delivery behavior consistent across CBR and VBR. ## Matching VBR Bitrates to CBR Quality - Identical nominal bitrates do not produce identical behavior: - CBR remains clustered around its target with frequent small variations. - VBR spends far less on simple scenes and increases bitrate only when complexity demands it. - Netflix therefore needed to revisit which nominal VBR bitrates correspond to the quality previously delivered by CBR, rather than assuming the same configured bitrate would provide equivalent results. Netflix’s rollout shows that VBR is more than an encoder setting: it requires coordinated changes to bitrate ladders, capacity reservations, and traffic steering. With those safeguards, VBR can deliver comparable or better quality while using significantly less network capacity.

Read original(opens in new tab)

Live StreamingContent Delivery NetworkAws Elemental MedialiveVariable Bitrate+4

netflix·Mar 6, 2026·3 min readCurated summary

Scaling Global Storytelling: Modernizing Localization Analytics at Netflix

Netflix is modernizing its localization analytics to support more than 300 million members across 190+ countries and 50+ languages. Rapid growth created duplicated pipelines, inconsistent business logic, and siloed dashboards, making basic questions such as who produced a dub difficult to answer reliably. The company’s solution is to consolidate data foundations, improve usability, and centralize reusable business logic. ## The Challenge of Fragmented Localization Data - Localization metrics were historically built independently across different teams and workflows. - Determining who created a dub or subtitle required combining multiple sources with complex, frequently changing rules. - Duplicated logic led to: - Inconsistent reporting across tools - High maintenance costs when upstream systems changed - Siloed analytics and dashboards ## Auditing and Consolidating Analytics - Netflix audited more than 40 dashboards and tools for usage, quality, and code health. - The focus shifted from repeatedly fixing frontend visualizations to consolidating backend data pipelines. - Three legacy dashboards covering dubbing-partner operations, capacity, and finances are being unified around a shared data and backend layer. - This foundation can support multiple future frontend experiences instead of forcing each dashboard to maintain separate logic. ## Reducing User Experience Debt - Netflix defines “Not-So-Tech Debt” as stakeholder friction caused by confusing tools or weak analytical storytelling. - The Language Asset Consumption tool was redesigned to combine audio and text languages into a single consumption-language view. - This distinguishes: - Original-language viewing from localized consumption - Subtitle, dubbing, or combined preferences - Recurring member preferences for a given language - The result is more intuitive analysis aligned with real stakeholder questions. ## Centralizing Reusable Business Logic - Netflix is adopting a “write once, read many” architecture. - Shared tables, including a Language Asset Producer table, solve common questions in one centralized location. - The same trusted data can feed downstream domains such as Dub Quality and Translation Quality. - Updates to business rules propagate across the analytics ecosystem instead of requiring changes in multiple pipelines. ## Moving Toward Event-Level Analytics - Future work will analyze individual timed-text events rather than only complete language assets. - A generic model will capture details such as individual subtitle lines and reading speed. - Netflix plans to connect subtitle characteristics with member engagement. - These findings can improve style guidelines for subtitle linguists and ultimately enhance the localized viewing experience. Netflix’s recommendation is to treat analytics modernization as both a technical and product-quality effort: consolidate data foundations, centralize business logic, and design tools around how stakeholders actually make decisions. This creates more trustworthy reporting while enabling deeper analysis of how localization affects member enjoyment.

Read original(opens in new tab)

Database DesignPostgreSQLData ModelingData Pipelines+4

netflix·Mar 3, 2026·3 min readCurated summary

Optimizing Recommendation Systems with JDK’s Vector API

Netflix’s Ranker service used significant CPU for video serendipity scoring, which compares candidate-title embeddings with a member’s viewing history. The team reduced CPU usage by progressively replacing scalar dot products with batched computation, improving memory layout, reusing buffers, and investigating optimized matrix-multiplication libraries. The main lesson was that mathematical optimization alone is insufficient; allocation behavior, cache locality, SIMD support, and runtime overhead all matter. ## The Serendipity Scoring Hotspot - Each candidate title and history item is represented by a vector embedding. - The service computes cosine similarity between every candidate and every history item. - It selects the maximum similarity and converts it into a novelty score: - `serendipity = 1.0 - maxSimilarity` - The original implementation performed `M × N` individual dot products, creating: - Sequential computational work - Repeated embedding lookups - Scattered memory access - Poor cache locality - This logic consumed roughly 7.5% of CPU per Ranker node. - Although 98% of requests contained one video, large batch requests represented about half of the total videos processed. ## Batching Similarity Computations - The team reorganized the calculation as matrix multiplication: - Candidate embeddings form an `M × D` matrix. - History embeddings form an `N × D` matrix. - Rows are normalized to unit length. - Similarities are computed as `C = A × Bᵀ`. - This replaces many separate dot products with one larger operation better suited to CPU-optimized kernels. - The implementation added `batchEncode()` while preserving the existing `encode()` path for single-video requests. ## Why the First Batched Version Regressed - Initial canary tests showed a 5% performance regression. - The batched implementation created `double[][]` arrays for candidates, history, and results on every request. - These allocations: - Increased garbage-collection pressure - Used non-contiguous memory - Added pointer chasing and reduced cache efficiency - The matrix multiplication itself was scalar Java code and did not exploit SIMD hardware. - Batching therefore introduced overhead without delivering corresponding compute gains. ## Flat Buffers and Thread-Local Reuse - The team replaced multidimensional arrays with flat `double[]` buffers in row-major order. - Contiguous storage improved predictability and cache locality. - A `ThreadLocal<BufferHolder>` was used to retain reusable candidate, history, and scratch buffers per thread. - Buffers grow when necessary but do not shrink, avoiding repeated allocations while preventing cross-thread contention. - This reduced GC pressure and made batch performance more stable. ## Evaluating BLAS - BLAS appeared promising in isolated microbenchmarks but did not provide the expected production improvement. - The default `netlib-java` configuration used F2J, a Java implementation rather than truly native BLAS. - Native BLAS introduced setup costs and JNI transition overhead. - Java’s row-major data layout also created an impedance mismatch with common BLAS expectations.

Read original(opens in new tab)

Database DesignJavaCosine SimilarityMatrix Multiply+4

netflix·Feb 28, 2026·3 min readCurated summary

Mount Mayhem at Netflix: Scaling Containers on Modern CPUs

Netflix’s effort to modernize its container runtime exposed a hardware-level bottleneck rather than an application problem. Under heavy startup concurrency, containers with many image layers triggered massive mount and unmount activity, causing kernel lock contention, systemd stalls, and container startup failures. The issue was especially severe on older dual-socket NUMA instances, while newer single-socket systems scaled much more reliably. ## Container Startup at Netflix - New AWS capacity is rapidly filled with pods as applications scale. - Some nodes became unresponsive, with: - Health checks timing out for more than 30 seconds - Kubelet requests to containerd timing out - systemd processing huge numbers of mount events - The mount table taking tens of seconds to read - The problem primarily affected `r5.metal` instances running images with more than 50 layers. ## Mount Lock Contention - With user namespaces, containerd performs several mount operations for every image layer: - `open_tree()` references the layer. - `mount_setattr()` applies the container’s ID mapping. - `move_mount()` creates an ID-mapped bind mount. - These bind mounts become OverlayFS lower directories and are later unmounted. - The Linux VFS uses global mount-related locks, so concurrent container creation causes CPUs to contend on the same kernel locks. - For 100 containers with 50 layers each, containerd performs the process twice: - `100 × 2 × (1 + 50 + 50) = 20,200` mount operations - This makes startup cost depend heavily on both container concurrency and image layer count. ## Why the New Runtime Exposed the Problem - The old Docker-based runtime shifted file ownership while unpacking images. - All containers shared one host user range, avoiding repeated per-container mount work. - The new containerd-based runtime assigns each container a unique host user range for stronger isolation. - Instead of rewriting file ownership during extraction, it uses Linux ID-mapped mounts to apply ownership mappings efficiently. - This improves security and avoids expensive image copying, but creates many additional mount operations during startup. ## Differences Between AWS Instance Types Netflix compared: - `r5.metal`: 5th-generation Intel, dual-socket, multiple NUMA domains - `m7i.metal-24xl`: 7th-generation Intel, single-socket, single NUMA domain - `m7a.24xlarge`: 7th-generation AMD, single-socket, single NUMA domain Results showed: - At low concurrency—around 20 containers or fewer—all systems performed similarly. - `r5.metal` began failing at roughly 100 concurrent container launches. - Newer Intel instances maintained lower startup times and better success rates. - AMD-based `m7a` instances scaled most consistently and had the fewest failures. ## Kernel and CPU-Level Diagnosis - Profiling showed that containerd spent most of its time in Linux VFS path lookup code. - Specifically, threads were spinning in `path_init()` while waiting on a sequence lock. - Intel Topdown Microarchitecture Analysis found: - 95.5% of pipeline slots stalled on contested accesses - 57% attributed to false sharing - Cache-line bouncing and global lock contention, rather than raw CPU capacity, dominated performance. ## NUMA as a Contributing Factor - NUMA systems divide memory among processor sockets. - Local memory access is faster, while remote access crosses an interconnect and introduces additional latency. - The dual-socket layout of `r5.metal` amplified contention around shared mount-related data. - The better behavior of newer single-socket instances indicated that CPU topology and memory locality were key contributors to the container startup bottleneck. ## Practical Conclusion High-concurrency container launches can overwhelm kernel mount infrastructure, especially when using per-container ID mapping and images with many layers. Netflix’s results suggest minimizing image layers, controlling startup concurrency, and favoring newer single-socket hardware can substantially improve reliability and scaling.

Read original(opens in new tab)

Database DesignKubernetesAWSDocker & Containers+4

netflix·Feb 24, 2026·3 min readCurated summary

MediaFM: The Multimodal AI Foundation for Media Understanding at Netflix

Netflix’s Media Foundational Model (MediaFM) is a tri-modal AI system that combines video, audio, and timed text to understand long-form entertainment. It represents sequences of shots while using title-level metadata and temporal context to produce richer content embeddings. Netflix concludes that these contextual embeddings improve many downstream tasks, including advertising relevance, clip selection, tone classification, and popularity prediction. ## Motivation for MediaFM - Netflix needs machine-readable understanding of its expanding catalog, including films, series, live events, and podcasts. - Long-form media requires recognizing narrative dependencies, emotional arcs, scene transitions, and subtle tones across entire episodes or films. - Combining visual, audio, and textual signals provides a more complete understanding than relying on video alone. - The resulting embeddings support applications such as: - Cold-start recommendations for new titles - Promotional art and trailer optimization - Advertising relevance - Clip tagging and internal content analysis ## Multimodal Input Representation - The model uses a shot as its fundamental unit, with titles segmented using shot-boundary detection. - Each shot receives three modality-specific embeddings: - **Video:** Frames sampled from the shot are encoded with SeqCLIP, Netflix’s video-retrieval model. - **Audio:** Sound is encoded using Meta FAIR’s wav2vec2. - **Timed text:** Captions, subtitles, or audio descriptions are encoded with OpenAI’s `text-embedding-3-large`. - The three embeddings are concatenated and unit-normalized into a 2,304-dimensional fused vector. - Training examples consist of temporally ordered shot sequences from a movie or episode, with up to 512 shots. - Title metadata, such as synopses and tags, is also embedded and supplied as global context. ## Transformer Architecture - MediaFM uses a BERT-like Transformer encoder. - Fused shot embeddings are first projected into the model’s hidden dimension. - Two special tokens are prepended: - `[CLS]`, a learnable sequence-level embedding - `[GLOBAL]`, containing projected title-level metadata - Positional embeddings and self-attention allow each shot representation to incorporate surrounding narrative context. - A final projection maps contextualized representations back into the original 2,304-dimensional embedding space. ## Masked Shot Modeling - The model masks 20% of shot embeddings in each training sequence. - Masked inputs are replaced with a learnable `[MASK]` embedding. - The Transformer must reconstruct the original fused embedding for each masked shot. - Training minimizes cosine distance between predicted and ground-truth embeddings. - Hidden parameters are optimized with Muon, while other parameters use AdamW; Netflix reports noticeable gains after adopting Muon. ## Evaluation Through Linear Probes - Netflix evaluates MediaFM by freezing its representations and training task-specific linear layers on top. - Most evaluation tasks involve short clips extracted from larger titles. - Embedding a clip within the context of its surrounding episode or film performs better than embedding the clip in isolation, demonstrating the value of long-range contextualization. ## Downstream Applications - **Ad relevancy:** Multilabel classification identifies clips suitable for relevant advertising; MediaFM helps retrieve candidate clips before ad-serving optimization. - **Clip popularity ranking:** The model predicts relative clip performance and click-through rate within a title, evaluated using Kendall’s tau. - **Clip tone:** Clips are classified into 100 categories, such as creepy, scary, or humorous. - **Clip genre:** Clips are assigned to core genres including Action, Comedy, Documentary, Drama, Horror, Romance, and Thriller. - **Clip retrieval:** The system distinguishes “clip-worthy” content from unsuitable clips based on human annotations, using Average Precision. MediaFM’s main practical lesson is that effective media understanding depends on fusing all available modalities and preserving long-form temporal context. Netflix’s approach provides a reusable embedding foundation for recommendation, promotion, advertising, and content-analysis systems rather than building a separate representation for every task.

Read original(opens in new tab)

Database DesignTransformerMultimodalVideo Embedding+4

netflix·Feb 13, 2026·4 min readCurated summary

Scaling LLM Post-Training at Netflix

Netflix argues that LLM post-training at production scale is as much an infrastructure challenge as a modeling challenge. Its internal framework abstracts distributed data processing, model sharding, GPU orchestration, checkpointing, and complex training workflows so developers can focus on experimentation. The result is a flexible system supporting SFT, DPO, reinforcement learning, and knowledge distillation across hundreds of GPUs. ## Why Post-Training Becomes an Engineering Problem - Pre-training provides general language ability, but post-training adapts models to Netflix’s catalog, member histories, recommendation tasks, personalization, and search. - Production-scale training introduces challenges involving: - Large proprietary datasets - Multi-node GPU coordination - Distributed model state - Workflows that combine training and inference - Failure recovery and experiment tracking - A simple Hugging Face fine-tuning script is insufficient for reliable, large-scale jobs. ## Preparing Data Correctly - Chat templates serialize conversations but do not determine which tokens should contribute to the loss. - Netflix applies explicit loss masking so training focuses on assistant responses rather than prompts or other non-target text. - Variable-length examples can waste GPU memory through padding and create synchronization overhead across FSDP workers. - Sequence packing combines multiple samples into fixed-length sequences. - A document mask prevents attention across separately packed samples while improving GPU utilization. ## Loading and Optimizing Large Models - Models that do not fit on one GPU require sharding strategies such as FSDP or tensor parallelism. - Partial weights should be loaded directly onto the device mesh rather than materializing the entire checkpoint on a single device. - Developers can choose full fine-tuning or LoRA and use: - Activation checkpointing - Compilation - Appropriate precision settings - Reinforcement learning requires compatible precision between rollout generation and policy training. - Large vocabularies create memory pressure because logits have dimensions `[batch, seq_len, vocab]`. - The framework reduces peak memory by removing ignored tokens before projection and computing logits and loss in sequence chunks. ## Distributed Training and Workflow Management - The framework supports standard forward/backward training for SFT as well as workflows that interleave: - Rollout generation - Reward-model and reference-model inference - Policy updates - Ray actors orchestrate distributed jobs while keeping hardware concerns separate from modeling code. - Experiment tracking covers both quality metrics, such as loss, and efficiency metrics, such as Model FLOPS Utilization (MFU). - Standardized checkpointing allows jobs to resume after failures. ## Netflix’s Post-Training Framework - The stack is built on: - Mako for AWS GPU provisioning - PyTorch, Ray, and vLLM - Netflix’s framework library for reusable utilities and training recipes - Jobs are generally defined through configuration files that select a recipe and provide task-specific components. - Unlike narrower fine-tuning systems, the framework supports: - Custom output heads - Expanded vocabularies and semantic IDs - Special tokens - Transformer models trained on non-natural-language sequences - This flexibility is important for Netflix-specific recommendation and personalization use cases. ## Four Core Abstractions ### Data - Dataset abstractions cover SFT, reward modeling, and RL. - Streaming supports datasets larger than local disk capacity. - Asynchronous sequence packing overlaps CPU preprocessing with GPU execution to reduce idle time. ### Model - The framework supports architectures such as Qwen3 and Gemma3, including Mixture-of-Experts variants. - LoRA is integrated into model definitions. - High-level sharding APIs distribute models across device meshes without requiring developers to write low-level distributed code. ### Compute - A unified job interface scales from one node to hundreds of GPUs. - MFU measurement remains accurate for custom architectures and LoRA configurations. - Checkpoints include parameters, optimizer state, dataloader state, and data-mixer state, enabling exact resumption. ### Workflow - The system supports SFT, DPO, RL, and knowledge distillation. - Online RL uses a hybrid architecture combining a single controller with Single Program, Multiple Data (SPMD) workers. - This extends conventional SPMD training to multi-stage workflows that cannot be represented as a simple training loop. Netflix’s approach is to standardize the difficult operational parts of post-training while preserving enough flexibility for unconventional models and objectives. A framework built around reusable data, model, compute, and workflow abstractions can help teams iterate faster and scale experiments without repeatedly rebuilding distributed infrastructure.

Read original(opens in new tab)

Database DesignLarge Language ModelsCloud ComputingPytorch+4