- Managed master fails during provisioning and show an error similar to:
Pod : Pod currently Pending. Reason: Unschedulable -> 0/4 nodes are available: 1 Insufficient cpu, 1 node(s) had taints that the pod didn't tolerate, 2 node(s) had no available volume zone.
- CloudBees Core on Modern Cloud Platforms
- Master Provisioning Plugin
- Kubernetes version , 1.12
- AWS EBS
- Azure Disk
- GCE PD
- #34583: PersistentVolume on EBS can be created in availability zones with no nodes
- Kubernetes Topology Awareness Support
- Azure Topology Awareness Support
- AWS Topology Awareness Support
- GCE PD Topology Awareness Support
This is caused by an issue in Kubernetes prior to version 1.12. The Kubernetes scheduler does not handle zone constraints when using dynamic provisioning of zone specific storage - such as for example EBS. First, the PersistentVolumeClaim is created and it is only then that the volume binding / dynamic provisioning occurs. This can lead to unexpected behavior where the PVC is created in a zone where there are actually no resources available for scheduling the pod.
Kubernetes 1.12 has released a new feature called Volume Binding Mode to control that behavior.
It requires that the VolumeScheduling feature gate is enabled - which it is by default since 1.10. You need to add the field
volumeBindingMode: WaitForFirstConsumer to the default storage class.
Here is an example:
kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: test-storage volumeBindingMode: WaitForFirstConsumer [...]
Note: Supported plugins support are listed at Storage Classes - Volume Binding Mode.
The solution is to upgrade to Kubernetes 1.12.
If this is not possible, an obvious workaround is to use a single zone environment until access to Kubernetes 1.12 is possible.
Another workaround is to create single zone storage classes (one per Availability Zone). The Managed Master can be configured to use a specific storage class.
1) Create Storage Classes
To create a Storage Class and bind it to an availability zone, the
zone parameter can be used. Following is a YAML example of a storage class using the
AWS EBS provisioner bound to
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: gp2-1a labels: k8s-addon: storage-aws.addons.k8s.io parameters: type: gp2 zone: us-east-1a provisioner: kubernetes.io/aws-ebs
Note: Different provisioners accept different parameters, more details can be found at Storage Class - Parameters.
2) Configure Managed Masters
In the Managed Master configuration, set the storage class name in the “Storage Class Name” field:
If a persistent volume claim already exists for the managed master, it must be deleted to be able to change the storage class. In that case you can follow these steps:
IMPORTANT NOTE: deleting a master PVC will delete the corresponding master volume.
- Stop the master from the CJOC UI
- Delete the master’s PVC with
kubectl delete pvc <master-pod-name> -n build
- Set the value of the “Storage Class Name” to use
- Start the master from the CJOC UI
Note: A default storage class for master provisioning can be set under Manage Jenkins > Configure System > Kubernetes Master Provisioning > Advanced > Default Storage Class Name. This setting would apply to newly created master only.