Dynamic Repartitioning for Time Series Workloads

Netflix’s TimeSeries Abstraction uses Cassandra to ingest and query petabytes of temporal data with millisecond-scale latency, but growing partitions can cause seconds-long reads, timeouts, and resource exhaustion. Its initial time-based partitioning works well when workload estimates are accurate, yet traffic changes and outlier IDs can make partitions too large or too small. Netflix therefore developed automated time-slice repartitioning and, for isolated hot IDs, asynchronous dynamic partitioning at the individual-ID level.

Cassandra and the Wide-Partition Problem

• Cassandra provides:
  • High-throughput, low-latency reads and writes
  • Cost-effective operation at scale
  • Strong operational familiarity within Netflix
• TimeSeries datasets accumulate events over time, creating potentially very wide partitions.
• Wide partitions can lead to:
  • Read latencies increasing from milliseconds to seconds
  • Request timeouts
  • Garbage-collection pauses
  • High CPU utilization and thread queueing
• Scaling Cassandra clusters can help, but Netflix sought more targeted solutions.

Initial Time-Based Partitioning

• TimeSeries divides data into discrete time slices to keep partitions manageable.
• This structure also makes it efficient to:
  • Query data by time
  • Drop old data without creating large tombstone problems
• At dataset creation, users provide expected workload characteristics.
• Netflix’s provisioning pipeline uses those inputs, along with Monte Carlo simulations, to select infrastructure and partition settings.

Why Static Provisioning Falls Short

• Workloads may be unknown or inaccurately estimated during initial provisioning.
• Traffic patterns, client behavior, and product needs can change over time.
• A small number of TimeSeries IDs may generate far more events than the rest.
• Time slices provide a way to change partitioning for future data, but manually updating thousands of datasets is impractical.

Repartitioning Entire Time Slices

• Cassandra introspection tools, such as nodetool tablehistograms, expose partition-size distributions.
• Netflix added a background worker that:
  • Monitors partition histograms for time slices
  • Publishes observations through a Cassandra virtual table
  • Detects partitions that are too large or too small
  • Calculates a new partitioning adjustment factor
• Target partition density is typically between 2 MiB and 10 MiB, depending on workload.
• The worker updates the strategy for future time slices. For example, it may expand a time_bucket interval from 60 seconds to 604,800 seconds when partitions are too small.
• This approach reduced read latency and timeouts caused by thread queueing.
• Its limitation is that it changes partitioning broadly and is ineffective when only a minority of IDs produce oversized partitions.

Handling Isolated Problem IDs

Netflix considers several responses when only some IDs are problematic:

• Do nothing: Appropriate when wide partitions do not affect application-level metrics.
• Partial returns: Abort a request after it exceeds a latency SLO while returning data already collected; useful when latency matters more than completeness.
• Block IDs: Prevent exceptionally bad test, spam, or otherwise harmful IDs from destabilizing the system.
• These options are inadequate when valid, important IDs must return all their data despite generating large partitions.

Dynamic Partitioning per ID

Dynamic partitioning addresses outliers by splitting partitions for individual TimeSeries IDs rather than modifying an entire table.

The asynchronous pipeline has three stages:

• Detection: The read path identifies partitions that exceed a configured size threshold.
• Planning and splitting: The system asynchronously plans and executes splits into appropriately sized partitions.
• Serving reads: Once splits are available, read requests are transparently rerouted to them.

During each read, the server tracks the bytes retrieved for a partition. If usage exceeds the threshold, it emits a detection event to Kafka containing information such as:

• The Cassandra time-slice table
• The affected TimeSeries ID
• The existing time and event bucket
• Whether the partition is immutable
• A version identifier

Practical Recommendation

Use whole-time-slice repartitioning when an entire dataset is systematically over- or under-partitioned. For isolated but important high-volume IDs, dynamic per-ID partitioning provides a more precise way to control latency without disrupting the rest of the dataset.