The Evolution of Cassandra Data Movement at Netflix
Netflix replaced its monolithic Cassandra-to-Iceberg connector, Casspactor, with a layered data movement engine built around direct reads from Cassandra backups in Amazon S3. Casspactor handled about 1,200 jobs and 3 PB daily but suffered from fragile metadata dependencies, skewed-partition failures, excessive intermediate tables, and limited support for higher-level data models. The new architecture uses Spark DataFrames and reusable, data-model-aware connectors to improve reliability, scalability, and cost efficiency. ## Casspactor’s Role and Limitations - Casspactor moved Cassandra data into Apache Iceberg using SSTables and metadata stored in S3 backups. - It supported critical Netflix workloads, including Member, Billing, Recommendations, and Subscriptions. - Its metadata view depended on several independent systems, each with different failure modes and update schedules. - Metadata could become inconsistent with actual backups, causing stale or incorrect data to be processed. - Cassandra maintenance or node replacement could break an entire region’s movement jobs because all nodes had to snapshot at the same clock second. ## Constraints for Higher-Level Data Abstractions - Cassandra-backed abstractions such as Key Value and Time Series inherited Casspactor’s limitations. - Large or skewed partitions caused executor memory failures and out-of-memory crashes. - Casspactor had no awareness of application-level data models, forcing downstream connectors to reconstruct them through costly post-processing. - Multiple intermediate Iceberg and snapshot tables increased storage costs and operational complexity. - Its backup composition model prevented reliable time travel to earlier backups after topology or keyspace schema changes. - The monolithic connector could not serve as a reusable foundation for specialized connectors. ## Direct S3 Metadata as the Source of Truth - The new design reads backup metadata directly from the S3 storage layer. - This removes the chain of external metadata dependencies. - Backup existence and completeness are determined from the files that actually contain the data. - Direct backup access also enables restoration of historical backup states. ## A Layered Connector Architecture - The Cassandra Analytics Wrapper builds on open-source Cassandra Analytics and Netflix’s internal backup format. - It uses an S3 client to read Cassandra backup files and convert them into standard Spark DataFrames. - A Connector Factory, implemented through Java UDFs and transforms, lets each abstraction define its own optimized connector. - Key Value, Time Series, and other models can transform generic DataFrames according to their own semantics. - Improvements to the shared reading engine automatically benefit every connector. ## Performance and Operational Improvements - Mutation compaction and processing run at Spark executor level, allowing better handling of wide and highly skewed partitions. - Reduced data shuffling helps prevent memory failures on large datasets. - Direct DataFrame output eliminates costly intermediary Iceberg tables. - Automatic job sizing adjusts resource usage based on source-table characteristics, reducing manual tuning. - Fewer dependencies improve reliability and make the system easier to maintain. Netflix’s new engine provides a shared, backup-native foundation while keeping data-model-specific logic in separate connectors. This approach is better suited to expanding Cassandra abstractions and large-scale data movement than maintaining another monolithic connector.
Read original(opens in new tab)