Data Projects: Managing Data Assets at Netflix Scale (opens in new tab)

Data Projects address Netflix’s difficulty managing millions of data assets and tens of thousands of workloads as teams and employees change. They replace asset-level permissions and human-owned workload identities with project-level grants and durable, synthetic identities. This makes access easier to maintain, workflows more resilient, and newly created assets easier to organize automatically.

The Limits of Asset-Level Permissions

• Netflix historically managed access through individual ACLs on each table.
• Organizational changes required updating hundreds or thousands of permissions manually.
• This overwhelmed support teams and encouraged overly broad access, such as granting access to the entire company.
• The model did not scale with frequent reorganizations, team changes, and ownership transfers.

The Limits of Human-Owned Workloads

• Scheduled jobs and asynchronous workloads traditionally ran under the identity of their author.
• When that person changed roles or left Netflix, the workload’s permissions changed or disappeared.
• Reassigning the job to another employee often introduced new permission gaps.
• This created a recurring “permissions whack-a-mole” across tens of thousands of business-critical workflows.

Data Projects as a Management Container

• A Data Project groups related tables, workflows, secrets, and other assets under one logical umbrella.
• Teams manage permissions for the project instead of maintaining ACLs across every individual asset.
• Grants can be assigned to users, groups, applications, and CI jobs.
• Roles such as Contributor and Viewer define read/write or read-only access at the project level.

Durable Project Identities

• Each project receives a Netflix application identity and, optionally, an AWS IAM role.
• Scheduled workloads execute as the project rather than as an individual employee.
• The IAM role supports AWS use cases such as Spark jobs on Amazon EMR.
• Privileged project members can assume the project identity from laptops or notebooks for testing and troubleshooting.
• This provides a development context that matches the identity used in production.

Gravity and Automatic Asset Organization

• Assets created by workloads running under a project identity are automatically added to that project.
• For example, tables created by a Maestro workflow become project assets without extra configuration.
• This “gravity” keeps related outputs organized and makes future access and discovery easier.
• Newly created assets inherit the project’s access model rather than requiring separate permissions.

Securing Maestro Workflows

• Maestro runs ETL pipelines, data movement jobs, machine-learning training, and other batch workloads.
• As a Trusted Workload Manager, Maestro can mint identity tokens for scheduled executions.
• A single workflow may be checked against table ACLs, Netflix resource policies, and AWS IAM policies.
• Using a durable project identity prevents failures caused by changes to the original author’s account.
• Project-scoped secrets also remain available when ownership changes.

Data Projects provide Netflix with a scalable foundation for access control, workload execution, and asset ownership. Moving management from individual assets and employees to durable, team-owned projects makes the platform more stable, auditable, and resilient to organizational change.