Netflix models device capabilities to determine which features can be safely supported across its diverse hardware ecosystem. By tracking hardware, software, and platform limitations in scalable analytical datasets, Netflix can measure feature reach and identify adoption bottlenecks. This enables more precise feature management for capabilities such as 4K, spatial audio, cloud gaming, and new UI experiences.
## Building a Device Capability Model
- Devices vary significantly in RAM, CPU cores, display resolution, audio support, and platform capabilities.
- Netflix maintains detailed capability data for each device model, including:
- Screen dimensions and resolution
- Supported video profiles and codecs
- Surround sound support
- RAM capacity
- Software version and platform information
- Internal feature flags are integrated into the model to connect device capabilities with feature availability.
## Cumulative Tables for Current Device State
- Netflix uses a cumulative table to track the latest known capabilities for each device.
- Capabilities are stored in a structured format, such as supported screen sizes and video profiles.
- This design supports large-scale analytics and reporting by providing an up-to-date view of device functionality.
## Histogram Tables for Feature Distribution
- A histogram table measures active devices over the previous 28 days.
- Results are broken down by device model and software version.
- The table also counts how many devices support particular capabilities.
- For example, Netflix can analyze external display support on streaming sticks:
- 100% of devices may support the HD PlayReady profile.
- Only 20% may support the UHD HEVC profile.
## Using Analytics for Feature Management
- Netflix uses these datasets to evaluate feature penetration for products such as:
- 4K Ultra HD
- Netflix Spatial Audio
- Cloud Gaming
- Updated user interfaces
- Capability data helps teams identify hardware or software bottlenecks.
- Feature decisions can therefore be made at a more granular level, improving performance, reliability, and user experience.
Netflix’s approach demonstrates that a detailed, analytics-focused capability model is essential for managing features across a global and highly varied device ecosystem.
GenRec is Netflix’s LLM-backed recommendation ranker, designed to reduce dependence on thousands of hand-engineered features. It verbalizes user history, item metadata, and context, then post-trains a Netflix-adapted foundation model for catalog-aware ranking and long-term member value. In large-scale A/B testing, it reportedly improved both short- and long-term metrics while using far fewer labeled examples and input signals than an established production ranker.
## Motivation for an LLM-Native Recommender
- Netflix’s existing recommendation stack supports many content types and product surfaces but is costly to extend.
- New use cases can require substantial feature engineering, architectural changes, infrastructure work, and experimentation.
- LLMs offer:
- Shared semantic representations for users, items, and interactions
- Natural-language steering through prompts
- Rich understanding of content and user histories
- General-purpose LLMs are not production-ready on their own because they may:
- Over-recommend globally popular titles
- Hallucinate items outside the catalog
- Ignore business constraints
- Provide limited personalization
## Ranking Problem and Long-Term Utility
- GenRec ranks the full Netflix catalog, or a provided candidate set.
- It uses the user, interaction history, current context, and time to produce a personalized ranking.
- The optimization target is expected long-term member utility—a proxy for satisfaction and retention—rather than only immediate clicks or plays.
## Two-Phase Training
### Netflix-Adapted Foundation Model
- Netflix first adapts an open-source LLM using proprietary Netflix data.
- The model learns:
- Netflix content and metadata
- Member behavior and preference patterns
- General language understanding and generation
- This relatively stable foundation can support multiple Netflix applications.
### GenRec Post-Training
- A second training phase specializes the foundation model for recommendation.
- It focuses on ranking quality, steering, reward alignment, and serving-cost constraints.
- This phase is refreshed more frequently to reflect new content and changing member preferences.
## Interaction Data as Conversations
- Hundreds of billions of events—including views, play duration, feedback, add-to-list actions, and abandons—are converted into single- or multi-turn recommender conversations.
- Each user message includes verbalized:
- Context and profile
- Interaction history
- Item metadata
- The recommendation task
- Assistant messages represent actual member behavior, such as titles played, viewing duration, or feedback.
- During inference, GenRec uses the verbalized request and a catalog-aware scoring head; it does not generate conversational responses.
- The conversational format mainly supports language-model training and comprehension of rich textual inputs.
## Context Engineering Instead of Feature Engineering
- GenRec represents histories and context as natural language rather than relying primarily on dense, manually designed features.
- The token budget becomes the new feature budget, so histories are selectively compressed:
- Retain detailed, high-signal events such as long plays and thumbs-up
- Remove low-signal events such as brief plays and hovers
- Summarize repetitive behavior such as binge-watching
- Add detail for important or cold-start items, including new releases
- Recent and high-value interactions receive priority, while older information is compressed or dropped.
- Prompt structure is also optimized for shared prefixes and prefix caching, reducing serving costs.
## Ranking, Language, and Reward Objectives
- GenRec combines several training objectives:
- **Catalog-aware ranking:** Uses high-value engagements as positive labels, applies denoising and thresholds, and trains the model with cross-entropy over the catalog or candidate set.
- **Language modeling:** Preserves understanding of natural-language histories and metadata and supports potential future features such as recommendation explanations.
- **Reward-weighted alignment:** Incorporates business requirements and longer-term satisfaction into training.
- Reward signals can help balance content categories such as movies, series, games, live content, and podcasts instead of optimizing only immediate engagement.
## Serving and Results
- GenRec runs in prefill-only mode on Netflix’s LLM serving infrastructure, avoiding the cost of decoding generated text.
- A catalog-aware scoring head converts the model’s representations into item rankings.
- Compared with a mature production ranker, GenRec achieved statistically significant improvements in short- and long-term online metrics.
- It did so with a small fraction of the labeled data and input signals used by the existing system.
GenRec suggests that recommendation systems can shift from extensive manual feature construction toward careful context engineering, LLM post-training, and reward alignment. The approach is most promising when paired with catalog constraints, efficient serving, and objectives that reflect long-term member value rather than raw engagement alone.
Netflix built an in-house LLM serving platform within its existing production ML infrastructure rather than creating a separate ML stack. The platform combines a JVM-based serving layer, NVIDIA Triton, GPU-backed Model Scoring Service, and an OpenAI-compatible HTTP frontend. Its main design choices—vLLM, model packaging, API compatibility, and deployment strategy—prioritize operational flexibility and seamless movement from hosted models to self-hosted ones, while production exposed versioning and compatibility risks.
## Architecture and Serving Model
- Netflix’s unified JVM serving system handles routing, A/B testing, feature retrieval, inference, post-processing, and logging.
- Callers access models through:
- A gRPC path integrated with the existing serving system.
- A direct HTTP path for newer LLM applications.
- Small CPU models run in-process to avoid remote-call overhead.
- Larger GPU models run through Model Scoring Service (MSS), which supports XGBoost, TensorFlow, PyTorch, and LLMs.
- NVIDIA Triton manages model loading, batching, and GPU scheduling.
- A Java control plane provides deployment, versioning, health checks, autoscaling, and multi-region rollout.
## Choosing vLLM as the Standard Engine
- Netflix originally used TensorRT-LLM, but re-evaluated its choice as open-source engines improved and workloads diversified.
- vLLM was selected based on operational fit rather than benchmark performance alone:
- Supports custom model architectures without lengthy compilation.
- Provides hooks for custom decoding and constraint logic.
- Is easier to debug than earlier compiled-engine workflows.
- Is familiar to many researchers, reducing the research-to-production transition cost.
- The workload includes embeddings, prefill-only inference, autoregressive decoding, and custom per-step decoding constraints.
## Triton Integration and Model Packaging
- Triton offers both a Python backend and a dedicated vLLM backend.
- The Python backend requires explicit input and output tensor definitions, coupling packaged artifacts to frontend changes.
- The vLLM backend uses a JSON configuration pointing to model weights and tokenizers, generating tensor specifications dynamically.
- Netflix considers the vLLM backend the preferred default because models and frontends can evolve independently.
- Production revealed two limitations:
- Triton and vLLM must be version-pinned because incompatible APIs can prevent the backend from loading entirely.
- Models requiring custom preprocessing, postprocessing, tokenization, or ensemble execution still need Triton’s Python backend.
## OpenAI-Compatible HTTP Frontend
- Netflix keeps LLMs compatible with the same internal gRPC model-serving interface used by other model types.
- It also exposes an OpenAI-compatible API because that interface is widely supported by inference engines, orchestration tools, evaluation systems, and client libraries.
- This makes replacing a hosted model with a fine-tuned self-hosted model largely transparent to callers.
- The implementation uses Triton’s OpenAI-compatible frontend, FastAPI, and a `TritonLLMEngine` that translates requests into Triton inference calls.
- KServe HTTP and gRPC frontends remain available for the Java control plane.
- Netflix found that Triton’s frontend silently discarded the `response_format` parameter, meaning JSON requests could reach vLLM without guided decoding and produce malformed output.
- The team patched the frontend to translate `response_format` into vLLM guided-decoding parameters.
## Deployment and Rollout Strategies
- GPU services require longer startup times than CPU services, and model versions may change input/output schemas.
- Netflix supports Red-Black deployment:
- Runs the new version alongside the old one.
- Performs health checks before shifting traffic.
- Gradually scales up the new version while scaling down the old one.
- Supports atomic rollback if deployment fails.
- Red-Black deployment works well when the model interface remains stable.
- Production exposed a schema-coordination problem: if a new model changes tensor dimensions or other I/O requirements, upstream callers may send old requests to the new model during the migration window, causing failures.
- The post introduces a Versioned strategy as a solution, but the provided text ends before explaining its implementation.
Netflix’s experience suggests that successful in-house LLM serving depends as much on compatibility and deployment mechanics as on raw inference speed. A practical platform should standardize on an extensible engine such as vLLM, preserve ecosystem-compatible APIs, tightly control engine versions, retain escape hatches for custom models, and explicitly coordinate model-schema changes during rollout.
The post explains how Netflix built a real-time service topology system capable of processing millions of network-flow records per second at production scale. Its core design combines streaming ingestion, reactive backpressure, physically separate data layers, and a distributed aggregation pipeline that resolves network intermediaries into meaningful service dependencies. The system favors slightly delayed but complete updates over stale batch data or incomplete results caused by dropping records.
## The Need for Real-Time Topology
- Traditional topology tools rely on hourly or daily batch processing, making their data outdated during incidents.
- Netflix combines:
- eBPF network flows
- IPC metrics delivered through Server-Sent Events
- Distributed tracing data
- These sources are stored in separate graph or columnar storage layers and can be queried independently or merged.
- The goal is near-real-time freshness, faster incident response, blast-radius analysis, and immediate change validation.
## Backpressure for Reliable Streaming
- Processing millions of flow records per second creates a risk that downstream systems will become overwhelmed.
- Common alternatives are inadequate:
- Unbounded queues eventually exhaust memory.
- Dropping records produces incomplete topology.
- Batch processing introduces unacceptable delays.
- Reactive streams propagate slowdown upstream:
- A graph database signals Stage 2.
- Stage 2 slows Stage 1.
- Stage 1 pauses Kafka consumption.
- Kafka retains the data until capacity returns.
- This allows the system to degrade gracefully during traffic spikes, garbage-collection pauses, or temporary storage slowdowns.
- Updates may be delayed by seconds or minutes, but the data remains substantially more complete than a dropped or hourly-processed stream.
## Physically Separate Topology Layers
Netflix keeps each data source in storage optimized for its characteristics:
- **Network layer:** eBPF flow logs provide broad coverage but limited application context.
- **IPC layer:** Application metrics offer detailed endpoint information but cover only instrumented services.
- **Tracing layer:** Parquet-based distributed traces show actual request paths but are sampled.
- Separate storage enables each layer to evolve and scale independently.
- Queries can run in parallel and merge results while preserving sub-second response times.
## Three-Stage Distributed Aggregation
The network layer uses a distributed pipeline to transform individual network hops into logical service dependencies.
- Cloud traffic commonly passes through load balancers, NAT gateways, API gateways, and proxies.
- Flow logs therefore show relationships such as:
- `App A → Load Balancer`
- `Load Balancer → App B`
- The useful topology must infer the logical dependency: `App A → App B`.
### Stage 1: Initial Flow Aggregation
- Consumes flow logs from Kafka across four regions.
- Filters invalid records.
- Groups data into five-minute windows.
- Creates initial aggregators for each window.
- Uses consistent hashing to distribute aggregators.
- Streams the results to Stage 2 through SSE.
### Stage 2: Intermediary Resolution
- Receives the initial aggregators from Stage 1.
- Groups flows by intermediary components.
- Resolves multi-hop network paths into application-level relationships.
- This prevents infrastructure components from dominating the resulting service graph.
## Engineering Trade-offs
- Streaming provides much fresher data than batch processing but introduces greater operational and conceptual complexity.
- Backpressure is essential for stability at Netflix’s scale, even though reactive pipelines are harder to reason about than synchronous systems.
- The architecture prioritizes reliable, complete topology updates over perfectly immediate processing.
- Production behavior differed substantially from local testing: consumers lagged, memory was exhausted, traffic became unevenly distributed, and garbage collection consumed significant resources.
Netflix’s approach demonstrates that large-scale real-time topology requires streaming ingestion, end-to-end backpressure, specialized storage, and staged aggregation. For similar distributed systems, the practical recommendation is to design explicitly for overload and partial slowdown rather than relying on unbounded buffering, dropped data, or stale batch snapshots.
GenPage is Netflix’s end-to-end generative approach to building personalized homepages. Instead of separately ranking rows and items, one transformer autoregressively generates the entire page—including rows, entities, and layout—from user and request context. In production, it outperformed Netflix’s mature multi-stage recommender on a core engagement metric while reducing serving latency by 20%.
## Reframing Homepage Recommendation
- Netflix’s homepage is a personalized two-dimensional structure, not a single ranked list.
- Traditional systems use separate candidate-generation and ranking stages for rows and entities.
- GenPage treats homepage construction as a prompt-response task:
- The prompt contains user history, profile information, and request context.
- The response is the complete homepage generated autoregressively.
- The approach aims to:
- Replace complex multi-stage pipelines with one end-to-end model.
- Optimize the whole page using reinforcement learning.
- Capture interactions such as diversity and the trade-off between high-value rows and continued browsing.
- Scale more predictably with additional data, compute, and model capacity.
- Support new content types, layouts, UI components, and personalized artwork with fewer architectural changes.
## Production Challenges and Results
- Real-time generation makes serving latency a major constraint.
- The system must address:
- Cold-start entities in a constantly changing catalog.
- Shifting user interests and cultural trends.
- Product and business rules that constrain generated pages.
- An online A/B test against Netflix’s optimized production recommender produced:
- Statistically significant improvement on Netflix’s primary launch engagement metric.
- A 20% reduction in end-to-end serving latency.
- Offline experiments found that:
- Improving the prompt helped more than increasing model capacity in the tested regime.
- Reinforcement-learning post-training improved homepage diversity, even though diversity was not an explicit objective.
## Tokenizing Context and Pages
- Each training example contains:
- **Context:** user history, profile attributes, and request information.
- **Page:** displayed rows and entities in layout order.
- **Feedback:** interactions such as plays, thumbs-up, and abandonment.
- Context and page are tokenized as model inputs and outputs.
- Feedback is used to derive reward and supervision signals rather than being directly generated.
## Domain-Specific Tokenization
- GenPage uses a custom recommender-system tokenizer instead of a general-purpose text tokenizer.
- This reduces sequence length and improves inference cost and latency.
- For example, an action such as watching *Orange Is the New Black* can be represented with four tokens:
- Entity ID
- Action type
- Time bucket
- Duration bucket
- Direct token mappings to product concepts, such as rows and entities, also make it easier to enforce generation rules and business constraints.
## Context Representation
- User-history tokens encode:
- Action type
- Entity ID
- Timestamp
- Duration
- The history includes explicit signals, such as playback, adding titles to My List, and thumbs-up, as well as implicit signals such as trailer views and detail-page visits.
- Profile tokens represent attributes including language and profile type.
- Request-context tokens include time of day, day of week, and device.
- Long data sources, such as complete impression histories, are summarized to control sequence length and cost.
- These summaries improve practicality but introduce handcrafted prompt engineering; learning to compress such information end to end remains a future direction.
- Special segment markers help the model distinguish between different context sources.
Netflix explores AI video-editing tools designed to preserve artists’ creative control rather than regenerate entire clips indiscriminately. The research addresses two major problems: unintended changes to untouched footage and physically implausible results when objects are removed. Its proposed systems, Vera and VOID, generate targeted edits while preserving scene identity, performance, and continuity.
## Challenges in Generative Video Editing
- Full-video regeneration can unintentionally change:
- Actors’ identities and performances
- Backgrounds and objects
- Important scene details
- Object removal often produces unnatural results because models erase the target without reconstructing realistic motion and physical interactions.
- Professional editors need precise control over what changes and what remains untouched.
## Vera: Layered Video Diffusion
- Vera generates:
- An edit layer containing the requested visual change
- An alpha matte defining where that change should appear
- These layers are composited with the original footage, leaving pixels outside the edited region intact.
- The approach supports tasks such as:
- Adding objects
- Changing backgrounds
- This layered design helps preserve original identities, performances, and details.
## Training Dataset
- Netflix created a custom dataset because existing public datasets lacked high-quality layered video data.
- The dataset contains 486,000 frames at 832×480 resolution.
- It includes:
- **Synthetic composites:** Foreground objects with alpha mattes placed over generated backgrounds.
- **Realistic single-object videos:** Real footage processed with segmentation, matting, background generation, and human review.
- **Realistic multi-object videos with effects:** Objects isolated along with shadows, reflections, and other scene effects.
## Vera’s Model Architecture
- Vera uses a Mixture-of-Transformers design with three specialized DiTs for:
- The edit layer
- The alpha matte
- The composite video
- Each branch has its own attention projections and feed-forward weights, allowing specialization while joint attention enables communication between layers.
- The model is initialized from a pretrained text-to-video model.
- Additional embeddings and input layers help distinguish source-video, mask, alpha, and composite information.
## Evaluation and Results
- Netflix tested Vera on:
- 72 object-addition video-prompt pairs
- 69 background-change pairs
- The benchmark included varied motion speeds, camera movements, object counts, and scene complexity.
- Evaluation measured:
- Preservation of untouched content
- Compliance with text instructions
- Temporal and per-frame video quality
- Vera-1.3B and Vera-14B substantially outperformed existing methods on content preservation while achieving comparable instruction-following and visual quality.
Netflix’s research favors localized, layered editing over unrestricted video regeneration. Vera demonstrates how separating edits from original footage can make generative tools safer and more controllable for professional workflows; the accompanying VOID research aims to apply similar principles to physically plausible object and interaction removal.
Netflix replaced much of its custom Compute Managed Batch (CMB) queuing and scheduling logic with Kubernetes-native Kueue. The migration preserved the existing user experience while enabling features such as preemption, fair sharing, all-or-nothing scheduling, and topology-aware placement. Kueue now manages millions of batch workloads across Netflix’s Titus-based infrastructure.
## CMB and Titus Architecture
- CMB manages workloads that run to completion using:
- Hierarchical tenants
- Priority-based ordering
- Per-tenant capacity management
- Workloads ultimately run on Titus, Netflix’s container platform.
- Titus provides federation across multiple Kubernetes cells and shared capacity reservations, allowing CMB to interact with a unified endpoint.
- CMB tenants are either:
- **Internal tenants**, which organize child tenants but do not accept jobs
- **Leaf tenants**, which accept jobs through associated queues
- Capacity includes:
- **Reserved capacity**, providing predictable resources within a tenant hierarchy
- **Shared capacity**, a global pool that tenants can burst into
- CMB enforced fair sharing only at admission time because it lacked preemption; admitted jobs ran to completion even when demand changed.
## Why Netflix Chose Kueue
- CMB was developed before many Kubernetes batch features became available in open source.
- Kueue provided capabilities Netflix had previously built or wanted to build, including:
- Fair sharing
- Hierarchical tenancy
- Capacity management
- Priority queues
- Preemption
- Unlike schedulers such as YuniKorn and Volcano, Kueue works with the existing Kubernetes scheduler rather than replacing it.
- This allowed Netflix to retain Titus scheduling profiles and avoid inefficient job placement.
- Kueue also supports:
- Multi-tenant quotas across heterogeneous hardware
- Native Kubernetes objects such as `Pod` and `Job`
- Higher-level workloads such as `RayJob` and `RayCluster`
- All-or-nothing admission and topology-aware scheduling
## Migrating CMB Workloads
- The migration, called **Netflix Batch**, was designed to:
- Require no changes from CMB users
- Avoid regressions in launch rates and maximum throughput
- Move queuing and scheduling responsibilities to Kueue
- Kueue runs in enabled Titus cells, while a custom router and Titus federation direct workloads to the appropriate cell.
- Tenant enrollment was exposed as a simple operator action in Netflix’s UI, making rollout and rollback straightforward.
- Internally, the migration mapped:
- CMB internal tenants to Kueue **Cohorts**
- Leaf tenants to **ClusterQueues** and **LocalQueues**
- Capacity configurations to Kueue **resource flavors** and **nominal quotas**
## Lessons from the Rollout
- Maintaining API compatibility reduced customer disruption and allowed Netflix to replace backend components incrementally.
- Migrating the largest and most complex customer early exposed problems sooner and increased confidence in the broader rollout.
- The production migration took approximately four weeks.
- Kueue required substantially higher QPS, burst, and `groupKindConcurrency` settings than its defaults.
- Netflix validated these settings early through load tests in an environment modeled on Titus.
## Kueue in Production
- Kueue is fully deployed at Netflix and manages millions of batch workloads.
- Netflix is extending its use to additional Titus batch workloads.
- Fair sharing and preemption are being expanded to improve utilization of reserved capacity.
- Netflix’s experience is also informing other internal Kubernetes-native systems, including training infrastructure.
Netflix’s migration demonstrates that a batch platform can adopt Kubernetes-native scheduling incrementally without forcing users to change APIs or abandoning existing placement infrastructure. For organizations with mature custom systems, preserving the external contract while delegating queueing and admission to Kueue offers a lower-risk path to modern features and simpler long-term operations.
Netflix built an automated “data canary” system to validate catalog metadata changes with real production traffic. The system compares a new catalog version against a known-good baseline, detects customer-impacting regressions in under 10 minutes, and blocks corrupted data before it reaches most members. The effort treats data deployments with the same rigor traditionally applied to code deployments.
## Why Catalog Data Needs Canarying
- Catalog metadata defines available titles, artwork, playback eligibility, and regional availability.
- A previous incident corrupted a feed without any code or configuration change.
- The resulting empty data for some titles prevented manifest generation and caused playback failures.
- Existing code canaries detected nothing because the failure occurred in transformed data, not application code.
- Validating individual upstream feeds was insufficient because corruption could emerge during final transformation.
## Challenges of Fast, Production-Level Validation
- Data cycles occur frequently, leaving only one cycle to detect problems and block publication.
- Traditional canary analysis requires 30–60 minutes to reach statistical confidence.
- Shadow traffic could replay catalog requests but could not reproduce the full playback lifecycle across services.
- Real production traffic was necessary to expose actual customer impact.
- The system also needed to contain regressions so that validation itself did not create a large outage.
## The Data Canary Orchestrator
- Netflix created a dedicated canary environment with:
- An orchestrator instance coordinating validation.
- A permanent baseline cluster serving the latest production catalog.
- A canary cluster receiving the new catalog version.
- Before testing, the orchestrator verifies that both clusters are healthy and version-synchronized.
- It then triggers a chaos experiment that compares customer behavior across the two versions.
- Results are returned to the transformer through a generic REST endpoint, allowing other data sources to adopt the pattern without transformer-specific changes.
## Extending the Chaos Platform
- Experiment thresholds were customized to meet the 10-minute detection requirement.
- Separate tests were run for major client types because they have different traffic patterns and dependencies.
- Playback traffic was especially effective at revealing failures.
- Sticky canaries used session affinity to keep each user on either the baseline or canary cluster, enabling a clean comparison.
- Starts Per Second (SPS) became the primary metric because it measures successful playback attempts more directly than latency or catalog-service error rates.
- Metrics are streamed in real time, and experiments abort immediately when a regression appears.
- This prioritizes rapid protection over maximum statistical confidence, which is appropriate given the strong customer-impact signal.
## Production-Hardened Reliability
- The orchestrator resumes polling experiments after restarts instead of abandoning active validation cycles.
- Leader election prevents multiple orchestrator instances from triggering duplicate experiments during deployment.
- Version tracking ensures baseline and canary clusters are aligned across tenants with different data-consumption schedules.
## Controlled Failure Injection
- Netflix validated the validator by deliberately corrupting catalog data.
- Tests included denylisting prominent titles and simulating realistic data-corruption scenarios.
- These experiments demonstrated whether the canary could identify meaningful playback regressions before corrupted metadata was broadly released.
Netflix’s approach shows that high-velocity data pipelines require deployment safeguards distinct from code canaries. Teams managing critical data should validate final transformed outputs with representative production traffic, use direct business-impact metrics, and automatically stop publication when regressions appear.
Data Projects address Netflix’s difficulty managing millions of data assets and tens of thousands of workloads as teams and employees change. They replace asset-level permissions and human-owned workload identities with project-level grants and durable, synthetic identities. This makes access easier to maintain, workflows more resilient, and newly created assets easier to organize automatically.
## The Limits of Asset-Level Permissions
- Netflix historically managed access through individual ACLs on each table.
- Organizational changes required updating hundreds or thousands of permissions manually.
- This overwhelmed support teams and encouraged overly broad access, such as granting access to the entire company.
- The model did not scale with frequent reorganizations, team changes, and ownership transfers.
## The Limits of Human-Owned Workloads
- Scheduled jobs and asynchronous workloads traditionally ran under the identity of their author.
- When that person changed roles or left Netflix, the workload’s permissions changed or disappeared.
- Reassigning the job to another employee often introduced new permission gaps.
- This created a recurring “permissions whack-a-mole” across tens of thousands of business-critical workflows.
## Data Projects as a Management Container
- A Data Project groups related tables, workflows, secrets, and other assets under one logical umbrella.
- Teams manage permissions for the project instead of maintaining ACLs across every individual asset.
- Grants can be assigned to users, groups, applications, and CI jobs.
- Roles such as Contributor and Viewer define read/write or read-only access at the project level.
## Durable Project Identities
- Each project receives a Netflix application identity and, optionally, an AWS IAM role.
- Scheduled workloads execute as the project rather than as an individual employee.
- The IAM role supports AWS use cases such as Spark jobs on Amazon EMR.
- Privileged project members can assume the project identity from laptops or notebooks for testing and troubleshooting.
- This provides a development context that matches the identity used in production.
## Gravity and Automatic Asset Organization
- Assets created by workloads running under a project identity are automatically added to that project.
- For example, tables created by a Maestro workflow become project assets without extra configuration.
- This “gravity” keeps related outputs organized and makes future access and discovery easier.
- Newly created assets inherit the project’s access model rather than requiring separate permissions.
## Securing Maestro Workflows
- Maestro runs ETL pipelines, data movement jobs, machine-learning training, and other batch workloads.
- As a Trusted Workload Manager, Maestro can mint identity tokens for scheduled executions.
- A single workflow may be checked against table ACLs, Netflix resource policies, and AWS IAM policies.
- Using a durable project identity prevents failures caused by changes to the original author’s account.
- Project-scoped secrets also remain available when ownership changes.
Data Projects provide Netflix with a scalable foundation for access control, workload execution, and asset ownership. Moving management from individual assets and employees to durable, team-owned projects makes the platform more stable, auditable, and resilient to organizational change.
Netflix developed boosted-tree models to predict when in-progress productions will deliver Locked Cuts and final IMF media. The models address gaps and inaccuracies in manually maintained schedules, improving delivery-date accuracy and providing earlier warnings of risk. Backtesting shows that predictive dates reduce error and Accumulated Error Days (AED), a measure strongly associated with launch delays.
## Launch Preparation and Schedule Risk
- After production, titles move through post-production and launch preparation.
- Final IMF assets trigger work on:
- Artwork and trailers
- Subtitles
- Maturity ratings
- Quality control
- Teams can begin earlier with a non-final Locked Cut, but later changes may require conformance work.
- Waiting for the IMF risks compressing the launch timeline if delivery is late.
## Problems with Manual Schedules
- Delivery estimates are manually supplied by content partners.
- Schedules often contain missing dates and inaccurate estimates.
- Dynamic production conditions—schedule changes, conflicts, and unforeseen obstacles—frequently cause delays.
- Predictive modeling can fill missing ETAs and improve existing ones.
## Accumulated Error Days and Launch Misses
- Accumulated Error Days (AED) measures the cumulative difference between estimated and actual delivery dates.
- Titles with launch misses have significantly higher mean AED than titles without misses.
- Inaccuracies close to delivery are more strongly associated with launch misses than errors accumulated over longer periods.
- Improving schedule accuracy near launch is therefore especially valuable.
## Predicting Time to Delivery
- Netflix uses boosted-tree regression models to predict the number of days until Locked Cut or IMF delivery.
- Models use:
- Production progress signals
- Title metadata
- Seasonal indicators
- Daily snapshots of production data
- Snapshot-based modeling keeps predictions current and supports changing features throughout all production phases.
## Evaluating Predictive Performance
- Netflix compares predicted and scheduled dates using:
- Mean and median absolute error
- Mean and median bias
- Error standard deviation
- Rates of large errors beyond specified day thresholds
- Predictive dates offer full coverage, unlike schedules that may lack estimates at some horizons.
- Backtesting showed lower errors and fewer outliers for predicted IMF and Locked Cut dates.
## Earlier Accuracy Signals
- Predictions can become reliable earlier than manual schedules.
- Six months before Locked Cut delivery, predictions were more accurate than scheduled dates for 76% of titles.
- Their 6.1-week mean absolute error matched the accuracy of scheduled dates only 11 weeks later.
- Across six months before delivery, predicted dates reduced AED for most buying organizations and content types.
## Integrating Predictions into Existing Workflows
- Because delivery dates already support stakeholder workflows, predictive estimates can be introduced without redesigning those processes.
- The remaining challenge is deciding when to trust scheduled dates versus predictions.
- Although predictions are generally more accurate, manual schedules can still outperform them in some situations, requiring a way to select the more reliable estimate.
Netflix’s modeling approach turns production data into an ongoing risk signal rather than relying solely on static partner schedules. Using predictive dates alongside existing workflows can give teams earlier, more accurate information for launch planning and help reduce avoidable launch delays.
Netflix replaced its monolithic Cassandra-to-Iceberg connector, Casspactor, with a layered data movement engine built around direct reads from Cassandra backups in Amazon S3. Casspactor handled about 1,200 jobs and 3 PB daily but suffered from fragile metadata dependencies, skewed-partition failures, excessive intermediate tables, and limited support for higher-level data models. The new architecture uses Spark DataFrames and reusable, data-model-aware connectors to improve reliability, scalability, and cost efficiency.
## Casspactor’s Role and Limitations
- Casspactor moved Cassandra data into Apache Iceberg using SSTables and metadata stored in S3 backups.
- It supported critical Netflix workloads, including Member, Billing, Recommendations, and Subscriptions.
- Its metadata view depended on several independent systems, each with different failure modes and update schedules.
- Metadata could become inconsistent with actual backups, causing stale or incorrect data to be processed.
- Cassandra maintenance or node replacement could break an entire region’s movement jobs because all nodes had to snapshot at the same clock second.
## Constraints for Higher-Level Data Abstractions
- Cassandra-backed abstractions such as Key Value and Time Series inherited Casspactor’s limitations.
- Large or skewed partitions caused executor memory failures and out-of-memory crashes.
- Casspactor had no awareness of application-level data models, forcing downstream connectors to reconstruct them through costly post-processing.
- Multiple intermediate Iceberg and snapshot tables increased storage costs and operational complexity.
- Its backup composition model prevented reliable time travel to earlier backups after topology or keyspace schema changes.
- The monolithic connector could not serve as a reusable foundation for specialized connectors.
## Direct S3 Metadata as the Source of Truth
- The new design reads backup metadata directly from the S3 storage layer.
- This removes the chain of external metadata dependencies.
- Backup existence and completeness are determined from the files that actually contain the data.
- Direct backup access also enables restoration of historical backup states.
## A Layered Connector Architecture
- The Cassandra Analytics Wrapper builds on open-source Cassandra Analytics and Netflix’s internal backup format.
- It uses an S3 client to read Cassandra backup files and convert them into standard Spark DataFrames.
- A Connector Factory, implemented through Java UDFs and transforms, lets each abstraction define its own optimized connector.
- Key Value, Time Series, and other models can transform generic DataFrames according to their own semantics.
- Improvements to the shared reading engine automatically benefit every connector.
## Performance and Operational Improvements
- Mutation compaction and processing run at Spark executor level, allowing better handling of wide and highly skewed partitions.
- Reduced data shuffling helps prevent memory failures on large datasets.
- Direct DataFrame output eliminates costly intermediary Iceberg tables.
- Automatic job sizing adjusts resource usage based on source-table characteristics, reducing manual tuning.
- Fewer dependencies improve reliability and make the system easier to maintain.
Netflix’s new engine provides a shared, backup-native foundation while keeping data-model-specific logic in separate connectors. This approach is better suited to expanding Cassandra abstractions and large-scale data movement than maintaining another monolithic connector.
Netflix’s personalized notification system separates long-term messaging strategy from real-time content selection. A “slow” policy sets each member’s personalized weekly pacing plan, while a “fast” policy chooses the best message when an opportunity arises. This hierarchy addresses the limits of short-term optimization by balancing immediate engagement with fatigue, opt-outs, and long-term member experience.
## Limitations of the Previous System
- The earlier system used a causal model to estimate the short-term incremental effect of sending a single notification.
- It optimized immediate actions, but could not capture cumulative effects such as:
- Notification fatigue
- Declining responsiveness over time
- Sustained viewing behavior
- Gradual opt-out risk
- Send frequency and message ranking were coupled:
- A relevance threshold implicitly controlled overall send volume.
- Changing the threshold affected both frequency and the quality or distribution of selected messages.
- Frequency could not be explicitly personalized according to each member’s engagement patterns.
## The Hierarchical Slow-Fast Architecture
- The **Slow policy** makes strategic decisions over a longer horizon, such as a week.
- It selects a personalized “Pacing Plan Action” that defines intended push and email frequencies.
- The action space contains roughly 100 combinations of cross-channel pacing strategies.
- The **Fast policy** operates in real time, selecting the most relevant message within the limits established by the slow policy.
## Utility-Based Strategic Planning
The Slow policy chooses the action that maximizes a personalized utility function:
`U(member, action) = Σ wₖ · Rewardₖ(member, action) — Cost(action)`
- Positive signals estimate whether a member will value and engage with notifications.
- Negative signals estimate fatigue and the likelihood of opting out of a channel.
- Explicit negative feedback is sparse, so predicted messaging costs alone are too small to prevent excessive sending.
- Netflix adds a universal cost to every message, ensuring that the utility remains well-behaved and discourages “always send” strategies.
- This cost is tuned through online experiments and offline evaluation.
## Pacing Messages Over Time
- A basic pacing strategy converts the target frequency into a per-opportunity probability.
- At each eligible opportunity, the system uses weighted randomization to decide whether to send.
- This produces a naturally varied schedule while maintaining the desired expected frequency.
- The architecture can also support structured patterns, including:
- Day-of-week preferences
- User-activity-based pacing
- Bursts aligned with product launches
## Communication Between Policies
- The Slow policy calculates a member’s plan and stores it in a low-latency feature store.
- The Fast policy retrieves that plan whenever a notification opportunity occurs.
- This asynchronous event-and-state design lets the planner focus on long-term member health while the executor focuses on immediate relevance.
The main recommendation is to decouple notification frequency and pacing from message ranking. A hierarchical system can explicitly manage long-term communication strategy while preserving the responsiveness and personalization of real-time selection.
The post presents a human-augmenting agentic workflow for observational causal inference (OCI), designed to automate repetitive analysis while preserving expert oversight. It combines an actor agent that executes analyses with a critic agent that evaluates assumptions, diagnostics, and credibility. The authors argue that transparent artifacts and process audits are essential because observational analyses rarely have definitive ground truth.
## Why Causal Inference Requires Oversight
- Data agents can quickly query data and run regressions, but may overlook confounding, selection bias, or differences between average users and specific subgroups.
- OCI requires substantial domain judgment, particularly when estimating effects from observational data under an unconfoundedness assumption.
- Automation is best used to reduce repetitive work—such as repeated balance checks, sensitivity analyses, and tracking iterations—so practitioners can focus on framing questions and scrutinizing assumptions.
## Target Trial Emulation and Design Diagnostics
Netflix’s OCI toolkit frames each analysis around the ideal randomized controlled trial that would answer the question.
- This “target trial” clarifies the treatment, outcome, population, timing, and assumptions required for a credible estimate.
- The workflow evaluates:
- **Covariate balance:** weighted standardized mean differences should generally be below 0.2.
- **Overlap:** propensity scores should remain between 0.1 and 0.9.
- **Placebo outcomes:** treatment should not appear to affect variables measured before treatment.
- **Sensitivity to hidden confounding:** estimated effects should be assessed against hypothetical omitted variables.
- These diagnostics help identify whether treated and untreated groups are sufficiently comparable.
## Human-Augmenting Agent Design
The workflow uses three personas:
- **Principal:** The human data scientist who defines the research question, context, threats to validity, tools, and data.
- **Actor:** The software agent that turns the plan into an analysis specification, executes the analysis, runs diagnostics, and produces reproducible artifacts.
- **Critic:** The software agent that reviews the plan and results, identifies omissions, assesses credibility, and recommends improvements.
The actor and critic operate in an iterative loop. Actors must use only approved tools, create human- and machine-checkable outputs, and report how they address failed diagnostics. Critics check for missing confounders, inconsistencies between the plan and execution, differences between the estimated estimand and the ATE, and gaps relative to the ideal randomized trial.
## Transparent Evaluation Through Artifacts
Because observational data generally lacks ground truth, evaluating an agent solely by comparing its answer to a known result is insufficient.
- Agents produce plans, specifications, plots, reports, and executed notebooks.
- Reports are version-controlled and notebooks are stored so principals can download and re-run them.
- Human reviewers can inspect every analytical step rather than trusting only the final estimate.
- The workflow also supports conventional evaluations using simulated datasets, while emphasizing process audits for real-world analyses.
## Empowering Practitioners
The system provides a templated notebook built on Netflix’s vetted, non-agentic OCI toolkit.
- The toolkit uses doubly robust learning for causal effect estimation.
- Humans remain responsible for writing the initial analysis plan and reviewing the executed notebook and critic’s report.
- The design is intended to extend beyond unconfoundedness-based OCI to methods with different assumptions, such as panel methods requiring parallel trends.
## Case Study: New Entertainment Types
Netflix applies the workflow to questions about whether newer entertainment offerings affect member satisfaction and subscription retention.
- The case study focuses on an entertainment category referred to as **Type X**.
- The broader goal is to estimate effects that could inform business strategy and understanding of member behavior.
- The workflow is positioned as a way to combine automated analysis with human judgment in this setting.
The recommended approach is not to let an agent make an unaudited causal claim. Instead, practitioners should use agents to execute standardized analyses and diagnostics, then inspect reproducible artifacts and critically assess the assumptions behind the result.
Netflix’s TimeSeries Abstraction uses Cassandra to ingest and query petabytes of temporal data with millisecond-scale latency, but growing partitions can cause seconds-long reads, timeouts, and resource exhaustion. Its initial time-based partitioning works well when workload estimates are accurate, yet traffic changes and outlier IDs can make partitions too large or too small. Netflix therefore developed automated time-slice repartitioning and, for isolated hot IDs, asynchronous dynamic partitioning at the individual-ID level.
## Cassandra and the Wide-Partition Problem
- Cassandra provides:
- High-throughput, low-latency reads and writes
- Cost-effective operation at scale
- Strong operational familiarity within Netflix
- TimeSeries datasets accumulate events over time, creating potentially very wide partitions.
- Wide partitions can lead to:
- Read latencies increasing from milliseconds to seconds
- Request timeouts
- Garbage-collection pauses
- High CPU utilization and thread queueing
- Scaling Cassandra clusters can help, but Netflix sought more targeted solutions.
## Initial Time-Based Partitioning
- TimeSeries divides data into discrete time slices to keep partitions manageable.
- This structure also makes it efficient to:
- Query data by time
- Drop old data without creating large tombstone problems
- At dataset creation, users provide expected workload characteristics.
- Netflix’s provisioning pipeline uses those inputs, along with Monte Carlo simulations, to select infrastructure and partition settings.
## Why Static Provisioning Falls Short
- Workloads may be unknown or inaccurately estimated during initial provisioning.
- Traffic patterns, client behavior, and product needs can change over time.
- A small number of TimeSeries IDs may generate far more events than the rest.
- Time slices provide a way to change partitioning for future data, but manually updating thousands of datasets is impractical.
## Repartitioning Entire Time Slices
- Cassandra introspection tools, such as `nodetool tablehistograms`, expose partition-size distributions.
- Netflix added a background worker that:
- Monitors partition histograms for time slices
- Publishes observations through a Cassandra virtual table
- Detects partitions that are too large or too small
- Calculates a new partitioning adjustment factor
- Target partition density is typically between 2 MiB and 10 MiB, depending on workload.
- The worker updates the strategy for future time slices. For example, it may expand a `time_bucket` interval from 60 seconds to 604,800 seconds when partitions are too small.
- This approach reduced read latency and timeouts caused by thread queueing.
- Its limitation is that it changes partitioning broadly and is ineffective when only a minority of IDs produce oversized partitions.
## Handling Isolated Problem IDs
Netflix considers several responses when only some IDs are problematic:
- **Do nothing:** Appropriate when wide partitions do not affect application-level metrics.
- **Partial returns:** Abort a request after it exceeds a latency SLO while returning data already collected; useful when latency matters more than completeness.
- **Block IDs:** Prevent exceptionally bad test, spam, or otherwise harmful IDs from destabilizing the system.
- These options are inadequate when valid, important IDs must return all their data despite generating large partitions.
## Dynamic Partitioning per ID
Dynamic partitioning addresses outliers by splitting partitions for individual TimeSeries IDs rather than modifying an entire table.
The asynchronous pipeline has three stages:
- **Detection:** The read path identifies partitions that exceed a configured size threshold.
- **Planning and splitting:** The system asynchronously plans and executes splits into appropriately sized partitions.
- **Serving reads:** Once splits are available, read requests are transparently rerouted to them.
During each read, the server tracks the bytes retrieved for a partition. If usage exceeds the threshold, it emits a detection event to Kafka containing information such as:
- The Cassandra time-slice table
- The affected TimeSeries ID
- The existing time and event bucket
- Whether the partition is immutable
- A version identifier
## Practical Recommendation
Use whole-time-slice repartitioning when an entire dataset is systematically over- or under-partitioned. For isolated but important high-volume IDs, dynamic per-ID partitioning provides a more precise way to control latency without disrupting the rest of the dataset.
Netflix’s Graph Abstraction is designed for OLTP graph workloads requiring millions of operations per second and millisecond-level latency, rather than open-ended analytical exploration. Built on existing Netflix abstractions, it supports real-time and optional historical graph views while handling nearly 10 million operations per second across 650 TB of data. Its core design emphasizes strong schemas, efficient traversal planning, low-latency caching, and controlled trade-offs such as eventual consistency and bounded query depth.
## OLTP Graph Use Cases
- Netflix distinguishes between:
- **OLAP workloads**, which prioritize large-scale exploration using RDF/SPARQL, property graphs, Gremlin, openCypher, or SQL.
- **OLTP workloads**, which require extremely high throughput, low latency, and global availability.
- OLTP queries may restrict traversal starting points, depth, or complexity to meet performance goals.
- Key applications include:
- **Real-Time Distributed Graph**, modeling dynamic relationships and interactions across Netflix.
- **Social Graph**, supporting social connections in Netflix Gaming.
- **Service Topology**, enabling real-time and historical analysis of internal services during incidents.
## Architecture and Netflix Data Abstractions
- The Graph Abstraction builds on existing platform components rather than implementing storage and caching independently.
- **Key-Value (KV) Abstraction** provides the latest state of nodes and edges and serves as the real-time index.
- **TimeSeries (TS) Abstraction** can be added for historical graph views.
- **EVCache** delivers low-millisecond latency, with additional specialized caching layers under experimentation.
- The **Data Gateway Control Plane** manages:
- Graph schemas
- Dataset provisioning and deletion
- KV and TS configuration
## Property Graph Model
- Graphs contain typed nodes and edges, each with associated properties.
- Properties are strongly typed to support:
- Efficient filtering
- Consistent data exports
- Validation during writes
- Edges may be:
- **Unidirectional**, representing one-way relationships
- **Bidirectional**, representing relationships traversable in both directions
## Namespaces and Provisioning
- Data is isolated into logical units called **namespaces**.
- Each namespace maps to a physical storage layer and may use dedicated or shared hardware.
- Provisioning automation selects an appropriate hardware configuration based on:
- Required throughput
- Latency targets
- Dataset size
- Workload criticality
## Graph Schema and Query Optimization
- Every namespace has an explicit schema defining:
- Node and edge types
- Valid properties and their types
- Allowed relationships
- Edge directions
- Schemas are represented through edge mappings, such as an `account owns profile` relationship or a bidirectional `profile linked_to device` relationship.
- Property definitions can specify types such as `TIMESTAMP` and `STRING`.
- Servers load schemas into an in-memory metadata graph, enabling:
- Rejection of invalid nodes, edges, and properties
- Faster traversal-path planning
- Deduplication of bidirectional edge traversals
- Removal of impossible paths and incompatible filters
- Servers periodically poll the Control Plane so schema changes are reflected without requiring manual updates.
- Planned improvements include:
- Using edge cardinality to reduce query fanout
- Generating type-safe data-access layers
- Making the Gremlin-like API schema-aware
## Real-Time Indexing with Key-Value Storage
- KV stores the real-time representation of all graph nodes and edges.
- Each namespace corresponds to a table, partitioned into records by unique IDs.
- Records contain multiple sorted key-value items, effectively forming a map of sorted maps.
- Writes to the same ID and key are idempotent, allowing safe retries and request hedging.
- KV uses timestamp-based tokens to enforce **Last-Write-Wins (LWW)** semantics.
- The post begins discussing the two-tier partitioning strategy for node storage, but the provided content ends before that design is explained.
Netflix’s approach demonstrates that high-throughput graph serving depends on specialized constraints and platform integration rather than unrestricted graph querying. Strong schemas, bounded traversals, KV-based indexing, automated provisioning, and low-latency caching together provide a practical foundation for production-scale OLTP graph workloads.
