- After upgrading CloudBees Core to release 126.96.36.199, Kubernetes planned agents are stuck in “Pending” and builds are hanging forever, waiting for the planned agents to come online with “Waiting for next available executor”.
- In such cases, the agent pod are not even scheduled
- CloudBees CI (CloudBees Core) 188.8.131.52
- CloudBees CI (CloudBees Core) on Modern Cloud Platforms - Managed Master 184.108.40.206
- CloudBees CI (CloudBees Core) on Modern Cloud Platforms - Operations Center 220.127.116.11
- CloudBees CI (CloudBees Core) on Traditional Platforms - Client Master 18.104.22.168
- CloudBees CI (CloudBees Core) on Traditional Platforms - Operations Center 22.214.171.124
- CloudBees Jenkins Platform - Client Master 126.96.36.199
- CloudBees Jenkins Platform - Operations Center 188.8.131.52
- CloudBees Jenkins Distribution 184.108.40.206
- Jenkins LTS
- Kubernetes Plugin from version 1.19.1 to 1.21.2 (excluded)
This is a bug in the kubernetes plugin introduced by JENKINS-56307 in version 1.19.1 of the kubernetes plugin.
This version introduces a new Node Provisioner strategy
NoDelayProvisionerStrategy that is enabled by default. The strategy provisions a node as soon as the Node Provisioner detects a need for more agents. As opposed to the default strategy that makes his decision based on load estimates.
There is a bug in the implementation that causes agents to never be provisioned and builds to hang.
This issue has been fixed in version 1.21.2 of the kubernetes plugin.
The solution is to upgrade CloudBees CI to version 220.127.116.11 or later.
The workaround is to disable the
NoDelayProvisionerStrategy. This can be done by adding the system property
-Dio.jenkins.plugins.kubernetes.disableNoDelayProvisioning=true to the Master’s startup. This requires to restart the Master in order to take effect. See How to add Java arguments to Jenkins to under stand how to do this.