Issue

• Kubernetes Agent provisioning is slow / stuck causing the build queue to grow
• Many threads are BLOCKED on org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision waiting on a lock on org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud:

"Computer.threadPoolForRemoting [#11599]" #350850 daemon prio=5 os_prio=0 tid=0x00007f5e581bf800 nid=0x4f9a waiting for monitor entry [0x00007f5da6db9000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:539)
	- waiting to lock <0x00000002f33e6f38> (a org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud)
	[...]

Environment

• CloudBees CI (CloudBees Core) on Modern Cloud Platforms
• CloudBees CI (CloudBees Core)
• CloudBees CI (CloudBees Core) on Modern Cloud Platforms - Managed controller
• CloudBees CI (CloudBees Core) on Modern Cloud Platforms - Operations Center
• CloudBees CI (CloudBees Core) on Traditional Platforms - Client controller
• CloudBees CI (CloudBees Core) on Traditional Platforms - Operations Center
• CloudBees Jenkins Platform - Client controller
• CloudBees Jenkins Platform - Operations Center
• CloudBees Jenkins Distribution
• Jenkins LTS
• Kubernetes Plugin

Related Issue(s)

• kubernetes-plugin #902
• kubernetes-plugin #904
• kubernetes-plugin #905

Resolution

The lock contention problem has been fixed in version 1.28.3 of the kubernetes plugin.

The recommended solution is to upgrade CloudBees CI to version 2.263.1.2 that includes this version of the plugin.

Workaround

If a core upgrade is not an option in short term, the workaround is to upgrade the kubernetes plugin to a fixed version. The fix has been backported to different baselines of the plugin:

• If using kubernetes plugin version 1.28.x, upgrade the kubernetes plugin to version 1.28.3
• If using kubernetes plugin version 1.27.x, upgrade the kubernetes plugin to version 1.27.7
• If using kubernetes plugin version 1.26.x, upgrade the kubernetes plugin to version 1.26.6