Issue

• A big amount of files in a volume causes a timeout when trying to attach the said volume
• There’s a timeout when waiting for volumes to attach when using Kubernetes
• There’s a timeout when trying to attach volumes to a Jenkins instance or Master
• I am seeing a warning or an event similar to the one below when trying to attach a volume:

  Unable to mount volumes for pod "<POD_IDENTIFIER>": timeout expired waiting for volumes to attach or mount for pod "jenkins"/"<POD_NAME>". list of unmounted volumes=[output]. list of unattached volumes=[output <VOLUMES>]

Environment

• CloudBees CI (CloudBees Core)
• CloudBees CI (CloudBees Core) on modern cloud platforms - Managed Master
• CloudBees CI (CloudBees Core) on modern cloud platforms - Operations Center
• CloudBees CI (CloudBees Core) on traditional platforms - Client Master
• CloudBees CI (CloudBees Core) on traditional platforms - Operations Center
• CloudBees Jenkins Enterprise
• CloudBees Jenkins Enterprise - Managed Master
• CloudBees Jenkins Enterprise - Operations Center
• CloudBees Jenkins Platform - Client Master
• CloudBees Jenkins Platform - Operations Center
• CloudBees Jenkins Distribution
• Jenkins LTS

Workaround

The solution for the problem is removing fsGroup from the Pod template.

Explanation: When mounting the volume k8s won’t try to change ownership of many different files from the volume and then won’t trigger the timeout.

Tested environment

• AKS - Azure Kubernetes Service
• EKS - Amazon Kubernetes Service

References

• https://github.com/kubernetes/kubernetes/issues/67014