A one-line Kubernetes fix that saved 600 hours a year

Atlantis restarts were taking about 30 minutes, blocking infrastructure changes and consuming more than 50 engineering hours monthly. The delay was caused by Kubernetes recursively changing ownership on a large Ceph-backed PersistentVolume containing millions of files. Setting fsGroupChangePolicy: OnRootMismatch avoided unnecessary recursive ownership changes and reduced restart time dramatically.

The Restart Bottleneck

• Atlantis runs as a singleton Kubernetes StatefulSet.
• Its PersistentVolume stores repository and Terraform state.
• Credential rotations, onboarding, and offboarding required restarting Atlantis.
• With roughly 100 restarts per month, each 30-minute delay created more than 600 hours of annual lost engineering time.
• The volume had grown large enough to exhaust inodes, making storage expansion and pod restarts necessary.

Kubernetes Made the Delay Look Like a Scheduling Problem

• kubectl rollout restart statefulset atlantis terminated the old pod and created a replacement.
• The new pod was scheduled quickly but remained stuck in Init:0/1.
• Kubernetes events showed the image pulling successfully, but revealed no obvious cause for the long gap.
• Kubelet logs showed the PersistentVolume mounting successfully, followed by repeated context deadline exceeded errors while syncing the pod.

The Hidden Cost of fsGroup

• Searching logs using the PersistentVolume name exposed the relevant message:
  • Kubernetes was “setting volume ownership” because an fsGroup was configured.
  • Kubernetes warned that ownership changes could be slow when a volume contained many files.
• The default behavior recursively changed ownership across the entire mounted volume.
• As Atlantis’s volume accumulated millions of files, this initialization step became the 30-minute bottleneck.

The One-Line Fix

• The volume configuration was changed to:

fsGroupChangePolicy: OnRootMismatch

• With this policy, Kubernetes checks the root directory’s ownership and only performs recursive changes when necessary.
• Existing volumes with the correct ownership no longer require a full filesystem traversal during every restart.

The practical lesson is to inspect kubelet and volume logs when a pod appears scheduled but remains stuck before initialization. For large persistent volumes, explicitly setting fsGroupChangePolicy: OnRootMismatch can eliminate costly recursive ownership changes and prevent substantial operational downtime.