How we minimized the overhead of Kubernetes in our job system

Kubernetes can improve machine management and scalability, but its scheduling and runtime overhead can significantly reduce job throughput if configured poorly. Datadog found its Kubernetes-based job system used more CPU and completed jobs 40–50% more slowly than the previous VM-based system. By designing a controlled experiment, choosing better metrics, and tuning pod resource requests, the team recovered performance to roughly VM parity while investigating the overhead of running one parent process per pod.

Designing a Comparable Experiment

• The initial comparison was difficult because the Kubernetes and VM deployments differed in:
  • Number of nodes
  • Number of worker-parent clusters
  • Workload and enqueue rate
• A controlled experiment was created using:
  • Identical c5.2xlarge machines
  • The same kernel version, 3.13.0-141
  • Both systems repeatedly running a simple Python job
• Each Kubernetes pod contained one parent process and its worker processes, making pod count equivalent to parent-process count per node.
• The older kernel did not include CPU mitigations, avoiding that variable in the comparison.

Choosing Useful Performance Metrics

Measuring node effort

• Load average initially appeared useful for measuring machine utilization.
• Kubernetes background processes—such as cluster polling and pod-state checks—artificially increased load average.
• Load average counts runnable processes rather than the amount of CPU time they actually consume.
• The team therefore used CPU idle time instead:
  • It measures unused CPU capacity.
  • It reflects actual CPU work rather than the number of active processes.

Measuring system performance

• The job system optimized for throughput rather than latency.
• Throughput was measured by the number of jobs completed within 30 seconds.
• Latency remained useful for detecting queueing problems, but throughput was the primary success metric.

Tuning Kubernetes Resource Requests

• The main performance gains came from improving pod scheduling.
• The target was six pods per c5.2xlarge node.
• Initially, each pod requested:
  • One full CPU core
  • More memory than necessary
• Since the node had eight cores and approximately 1.5 GiB of memory consumed by Kubernetes and system services, only four pods could be scheduled.
• Requests were reduced to:
  • 100m CPU, or 100 millicores
  • 500 MB memory
• CPU tuning generally enabled six pods per node, although some nodes still scheduled only five.
• Further memory reduction was needed because system daemons consumed enough memory to prevent six pods from fitting on some nodes.
• Resource requests affect scheduling minimums, while limits constrain containers after they start.
• These request changes did not slow jobs because the pods still received sufficient resources to operate.

One Parent Process per Pod

• The team considered placing multiple parent processes in each pod to reduce potential pod overhead.
• One parent plus its workers was a natural application unit and simplified orchestration.
• The decision depended on how much overhead each pod introduced:
  • High overhead would favor fewer, larger pods.
  • Low overhead would favor one parent per pod for simpler management.
• Using pstree, the team identified six job-system instances per node and traced their process trees through components such as:
  • containerd-shim
  • tini
  • The application process
• They estimated that each pod included overhead associated with three containers, particularly containerd-shim.
• CPU overhead was then investigated using perf sched.

The practical lesson is to compare equivalent workloads, measure actual CPU consumption rather than relying blindly on load average, and tune Kubernetes requests for the desired packing density. Resource requests should be large enough for reliable operation but not so large that they unnecessarily prevent pods from being scheduled together.