StarRocks Operations: Isolating Multi-tenant Workloads with Resource Groups

Toss adopted StarRocks as a real-time OLAP engine to consolidate service queries, analytics, validation, and dashboard workloads on one platform. As different workloads began competing within the same clusters, the key operational challenge became deciding which queries to protect during CPU contention. The article describes a gradual strategy: classify workloads, use cpu_weight by default, and introduce exclusive_cpu_cores only when stronger isolation is required.

Why StarRocks

• Toss previously relied on separate MySQL and Hadoop-based paths for serving, validation, monitoring, and analytics.
• StarRocks reduced this duplication by providing:
  • A MySQL-compatible SQL interface
  • Large-scale analytical processing
  • Real-time service-oriented reads
• Workloads eventually included:
  • Advertising and loan-underwriting services
  • Dashboards and monitoring tools
  • Kafka Connect ingestion
  • Batch jobs and backfills
• Average traffic varied by cluster:
  • Service cluster: approximately 69 QPS over 24 hours and 87 QPS over a week
  • Monitoring and batch cluster: approximately 20 QPS, plus heavier batch workloads
• Peak contention between different workloads mattered more than average QPS.

Workload Classification

Toss prioritized workloads in the following order:

1. Service queries
2. Server-side batch jobs
3. Large-scale ingestion and backfills
4. Monitoring and user query tools such as Grafana, Tableau, and Redash

• Service queries required strict SLA protection.
• Batch jobs needed to finish reliably but did not require real-time responses.
• Ingestion and backfills could overwhelm the cluster and therefore needed explicit limits.
• Monitoring queries received the lowest priority.

Using cpu_weight for Shared Capacity

• cpu_weight distributes CPU proportionally when workloads compete.
• Higher-weight groups receive more CPU during contention.
• When the cluster is idle, all groups can use available CPU regardless of weight.
• Toss used this as the default mechanism for multi-tenant workload control.
• Example priorities:
  • service_wg: weight 50
  • batch_wg: weight 10
  • dashboard_wg: weight 5
• Resource groups could also specify mem_limit and concurrency_limit.
• StarRocks uses a scheduler inspired by Linux CFS, with pipeline drivers yielding in roughly 100 ms time slices.

Using exclusive_cpu_cores for Strong Isolation

• exclusive_cpu_cores reserves physical CPU cores for a resource group.
• StarRocks binds worker threads to those cores using pthread_setaffinity_np.
• The group receives separate pools for:
  • DriverExecutor
  • ScanExecutor
  • ConnectorScanExecutor
• This prevents the protected workload from competing with shared thread pools.
• exclusive_cpu_cores and cpu_weight cannot be used together within the same resource group, although both types can coexist in one cluster.
• The setting is limited to (0, min_be_cpu_cores - 1].
• Because it is more rigid and consumes dedicated capacity, Toss recommends using it only when relative priority is insufficient.

Toss Shopping Case

• A cluster handled both real-time queries from shopping_service and heavy workloads from commerce_batch.
• Initially, both workloads had similar priority, allowing large batch queries to degrade service latency.
• First adjustment:
  • Increase shopping_service’s cpu_weight
  • Lower commerce_batch’s weight
• This improved prioritization but did not eliminate latency spikes when heavy batch work overlapped with roughly 1,500 service requests per minute.
• Second adjustment:
  • Place shopping_service in its own resource group
  • Assign dedicated CPU cores with exclusive_cpu_cores
• Afterward, service latency remained stable even during heavy batch execution.
• The operational approach was therefore incremental: begin with weights and escalate to dedicated cores only when necessary.

Classifier Design and Resource Controls

• Resource Groups control how resources are allocated; Classifiers determine which queries enter each group.
• Classifiers can match attributes such as:
  • User
  • Role
  • Query type
  • Source IP
  • Database
• The article recommends using stable identifiers such as user or db for reliable production behavior.
• Examples include mapping service SELECT queries by service account and assigning server-side batch queries according to their dedicated user.
• CPU isolation alone is insufficient for memory-heavy full scans or sudden spikes involving hundreds of concurrent queries, so memory and concurrency limits are also important.

Toss’s practical recommendation is to start with clear workload classification and cpu_weight, then add memory and concurrency limits. Use exclusive_cpu_cores selectively for latency-sensitive workloads whose SLAs cannot be protected through proportional CPU scheduling alone.