Issue

• I have deployed CloudBees Core with Terraform but CJOC cannot create masters. The master page shows Operation: [get] for kind: [StatefulSet] with name: [masterName] in namespace: [namespaceName] failed:

• Evidences shows that the cjoc pod does not have a volume mounted for the cjoc serviceAccount. kubectl get pod cjoc-0 shows only the following:

[...]
  volumes:
  - name: jenkins-home
    persistentVolumeClaim:
      claimName: jenkins-home-cjoc-0
  - configMap:
      defaultMode: 420
      name: cjoc-configure-jenkins-groovy
    name: jenkins-configure-jenkins-groovy

• The Jenkins logs shows PKIX exception:

2019-07-04 04:41:41.245+0000 [id=56]	WARNING	c.c.m.k.KubernetesMasterProvisioning$DescriptorImpl#lambda$getState$2: Caught an exception while retrieving state for KubernetesMasterResource [namespace=null, fsGroup=null, getName()=mm1, getEndpoint()=http://cloudbees-core.example.com/mm1/, getCpus()=1.0, getMemory()=3072.0, getRatio()=0.7, getImage()=DockerImageDefinition{imageTag='cloudbees/cloudbees-core-mm:2.164.3.2', name='CloudBees Core - Managed Master 2.164.3.2'}]
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
	at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
	at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
Caused: sun.security.validator.ValidatorException: PKIX path building failed
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
	at sun.security.validator.Validator.validate(Validator.java:262)
	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621)
Caused: javax.net.ssl.SSLHandshakeException

Environment

• CloudBees Core on Modern Cloud Platforms
• Jenkins LTS
• Terraform Kubernetes Provider

Related Issues

• terraform-providers/terraform-provider-kubernetes #263

Explanation

CloudBees Core CJOC and Masters are attached to service accounts so that they can interact with the Kubernetes API to create resources (pods, services, ingresses, etc…). The way this works is that the ServiceAccount credentials are automatically mounted as a Volume to the pod when it starts. The CJOC pod for example should have a volume like the following automatically mounted:

    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: cjoc-token-xxxxx
      readOnly: true

This is the default behavior in Kubernetes if the Service Account admission controller is activated (and it usually is). This behavior can be controlled by a property automountServiceAccountToken as explained in Configure Service Accounts for Pods

The behavior of the Terraform Kubernetes Provider however is different and opt out the automounting API credentials by default on the ServiceAccount object.

Resolution

When using terraform to deploy CloudBees Core, ensure that the cjoc and jenkins service accounts are properly configured with automount_service_account_token = true. For example, like the following assuming CJOC is deployed in the cje namespace

resource "kubernetes_service_account" "cjoc" {
  metadata {
    name = "cjoc"
    namespace = "cje"
  }
  automount_service_account_token = true
}

Refer to the terraform kubernetes provider documentation for more details.

Workaround

A workaround would be to add a volume explicitly to the CJOC / Master StatefulSet that mounts the service account secret.

CJOC

(Note: In those commands, replace <namespace> by the namespace where CJOC is deployed)

1) Get the name of the secret attached to the cjoc service account

kubectl get sa cjoc -n <namespace> -o jsonpath="{.secrets[0].name}"

2) Create a file automount-statefulset-patch.yaml (replace <secret-name> with the name of the secret we just retrieved):

apiVersion: "apps/v1"
kind: "StatefulSet"
spec:
  template:
    spec:
      containers:
      - name: "jenkins"
        volumeMounts:
        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          name: workaround-automount
          readOnly: true
      volumes:
      - name: workaround-automount
        secret:
          defaultMode: 420
          secretName: <secret-name>

3) Apply the patch with:

kubectl patch -p "$(cat automount-statefulset-patch.yaml)" statefulset cjoc -n <namespace>

Master

(Note: In those commands, replace <namespace> by the namespace where the Master is deployed)

1) Get the name of the secret attached to the jenkins service account

kubectl get sa jenkins -n <namespace> -o jsonpath="{.secrets[0].name}"

2) In the configuration of an existing Managed Master, add the following snippet to the YAML field (replace <secret-name> with the name of the secret we just retrieved):

apiVersion: "apps/v1"
kind: "StatefulSet"
spec:
  template:
    spec:
      containers:
      - name: "jenkins"
        volumeMounts:
        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          name: workaround-automount
          readOnly: true
      volumes:
      - name: workaround-automount
        secret:
          defaultMode: 420
          secretName: <secret-name>

Then restart the master from CJOC’s UI.

Note: The same configuration can be applied under Manage Jenkins > Configure System > Kubernetes Master Provisioning > Advanced > Default Storage Class Name. This setting would apply to newly created master only.