Issue

Managed master fails during provisioning and show an error similar to:

Pod : Pod currently Pending. Reason: Unschedulable -> 0/4 nodes are available: 1 Insufficient cpu, 1 node(s) had taints that the pod didn't tolerate, 2 node(s) had no available volume zone.

Environment

Related Issue

Explanation

This is caused by an issue in Kubernetes prior to version 1.12. The Kubernetes scheduler does not handle zone constraints when using dynamic provisioning of zone specific storage - such as for example EBS. First, the PersistentVolumeClaim is created and it is only then that the volume binding / dynamic provisioning occurs. This can lead to unexpected behavior where the PVC is created in a zone where there are actually no resources available for scheduling the pod.

Kubernetes 1.12 has released a new feature called Volume Binding Mode to control that behavior.

Resolution

Kubernetes 1.12 and later

It requires that the VolumeScheduling feature gate is enabled - which it is by default since 1.10. You need to add the field volumeBindingMode: WaitForFirstConsumer to the default storage class.

Here is an example:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: test-storage
volumeBindingMode: WaitForFirstConsumer
[...]

Note: Supported plugins support are listed at Storage Classes - Volume Binding Mode.

Kubernetes < 1.12

The solution is to upgrade to Kubernetes 1.12.

If this is not possible, an obvious workaround is to use a single zone environment until access to Kubernetes 1.12 is possible.

Another workaround is to create single zone storage classes (one per Availability Zone). The Managed Master can be configured to use a specific storage class.

1) Create Storage Classes

To create a Storage Class and bind it to an availability zone, the zone parameter can be used. Following is a YAML example of a storage class using the AWS EBS provisioner bound to us-east-1a:

apiVersion: storage.k8s.io/v1 
kind: StorageClass 
metadata: 
  name: gp2-1a
  labels: 
    k8s-addon: storage-aws.addons.k8s.io 
parameters: 
  type: gp2 
  zone: us-east-1a
provisioner: kubernetes.io/aws-ebs

Note: Different provisioners accept different parameters, more details can be found at Storage Class - Parameters.

2) Configure Managed Masters

In the Managed Master configuration, set the storage class name in the “Storage Class Name” field:

If a persistent volume claim already exists for the managed master, it must be deleted to be able to change the storage class. In that case you can follow these steps:

IMPORTANT NOTE: deleting a master PVC will delete the corresponding master volume.

Stop the master from the CJOC UI
Delete the master’s PVC with kubectl delete pvc <master-pod-name> -n build
Set the value of the “Storage Class Name” to use
Start the master from the CJOC UI

Note: A default storage class for master provisioning can be set under Manage Jenkins > Configure System > Kubernetes Master Provisioning > Advanced > Default Storage Class Name. This setting would apply to newly created master only.

Autoscaling issue when provisioning Masters in Multi AZ Environment