Issue

• I would like to add / import self signed certificates to JVM truststore of Masters and / or Agents
• I would like to add / import self signed certificates to use with tool like git and curl
• Container images have ca-certificates installed

NOTE: This article describes the process to follow on Kubernetes based installation, for non Kubernetes based installations please refer to this article

Environment

• CloudBees Core on Modern Cloud Platforms
• Kubernetes Plugin
• Kubernetes v1.10 and later

Resolution

Since version 2.138.3.1, the recommended solution is to use a sidecar-injector as explained Using self-signed certificates in CloudBees Jenkins Enterprise.

Otherwise, there is another solution using ConfigMaps like below.

Solution with ConfigMaps

For version lower than 2.138.3.1, the following solution can be used:

• 1) Create a ConfigMap that holds cacerts and ca-certificates.crt
• 2) Mount the ConfigMaps to the Containers that require them - i.e. CJOC, Masters, agents - and set up the pod / containers to point tools to the location of the certificates bundle.

The advantage of this approach is that whenever a new certificates need to be added, it can be done by updating the ConfigMap and restarting the resource that mounts it.

Pre-requisites

• ca-certificates is installed in the image / containers where certificates need to be imported. At the moment of writing this article, cloudbees-core images are based on openjdk over Debian stretch and therefore have ca-certificates installed`
• Kubernetes version 1.10 which provide support for binaries in ConfigMaps - necessary for cacerts

1) Create the ConfigMap

First, a ConfigMap needs to be created to hold:

• a preconfigured JVM keystore cacerts
• a preconfigured CA bundle ca-certificates.crt

Create a directory:

mkdir ca-bundle

Prepare the cacerts

Create / Copy a JVM truststore. You can copy the truststore from an existing JVM installation, for example from the CJOC container:

kubectl cp $CJE_NAMESPACE/cjoc-0:/etc/ssl/certs/java/cacerts $(pwd)/ca-bundle/cacerts

Then import certificates to the cacerts truststore - in this example cje.example.com.pem:

keytool -import -alias cje.example.com -keystore ca-bundle/cacerts -file cje.example.com.pem

Prepare the ca-certificates.crt

Create / Copy a ca-certificates.crt. You can copy the ca-certificates.crt from an existing container, for example from the CJOC container:

kubectl cp $CJE_NAMESPACE/cjoc-0:/etc/ssl/certs/ca-certificates.crt $(pwd)/ca-bundle/ca-certificates.crt

Then add certificates to the ca-certificates.crt. The format should look like the following:

[...]
-----BEGIN CERTIFICATE-----
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-----END CERTIFICATE-----

Create the ConfigMap

The directory ca-bundle should have the following files:

ls ca-bundle/
ca-certificates.crt	cacerts

Create a ConfigMap from that directory:

kubectl create configmap ca-bundle --from-file=ca-bundle -n $CJE_NAMESPACE
configmap "ca-bundle" created

2) Deploy the Certificates

To deploy the certificate, the ConfigMap needs to be mounted to the containers.

With ca-certificates installed, the cacerts / ca-certificates.crt can be injected in specific locations which depends on the underlying distribution in the container:

• Alpine / Debian / Ubuntu / Gentoo etc.: /etc/ssl/certs/ca-certificates.crt and /etc/ssl/certs/java/cacerts for
• CentOS 7 / RHEL 7: /etc/pki/java/cacerts and /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
• CentOS 6 / RHEL 6: /etc/pki/java/cacerts and /etc/ssl/certs/ca-bundle.crt

Deploy Certificates in the CJOC

In the cloudbees-core.yaml edit the cjoc statefulset and add the volume / volumes mounts similar to the following:

---
apiVersion: "apps/v1beta1"
kind: "StatefulSet"
metadata:
  name: cjoc
  labels:
    com.cloudbees.cje.type: cjoc
    com.cloudbees.cje.tenant: cjoc
spec:
  serviceName: cjoc
  replicas: 1
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      name: cjoc
      labels:
        com.cloudbees.cje.type: cjoc
        com.cloudbees.cje.tenant: cjoc
    spec:
      serviceAccountName: cjoc
      terminationGracePeriodSeconds: 10
      containers:
      - name: jenkins
        image: cloudbees/cloudbees-cloud-core-oc:2.138.2.2
        [...]
        volumeMounts:
        - name: volume-ca-bundle
          mountPath: /etc/ssl/certs/java/cacerts # Alpine / Debian / Ubuntu / Gentoo* etc.
          subPath: cacerts
        - name: volume-ca-bundle
          mountPath: /etc/ssl/certs/ca-certificates.crt # Alpine / Debian / Ubuntu / Gentoo etc.
          subPath: ca-certificates.crt
        - name: jenkins-home
          mountPath: /var/jenkins_home
        - name: jenkins-configure-jenkins-groovy
          mountPath: /var/jenkins_config/configure-jenkins.groovy.d
        livenessProbe:
          httpGet:
            path: /cjoc/login
            port: 8080
          initialDelaySeconds: 300
          timeoutSeconds: 5
      volumes:
      - name: jenkins-configure-jenkins-groovy
        configMap:
          name: cjoc-configure-jenkins-groovy
      - name: volume-ca-bundle
        configMap:
          name: ca-bundle
          
[...]

Apply the config with:

kubectl apply -f cloudbees.core.yaml -n $CJE_NAMESPACE

CJOC needs to be redeployed for the change to take effect.

Deploy Certificates in the Managed Master

Go to Manage Jenkins > Configure System > Kubernetes Master Provisioning > Advanced and set the YAML like the following (example for Debian):

apiVersion: "apps/v1"
kind: "StatefulSet"
spec:
  template:
    spec:
      containers:
        - name: "jenkins"
          volumeMounts:
          - name: volume-ca-bundle
            mountPath: /etc/ssl/certs/java/cacerts # Alpine / Debian / Ubuntu / Gentoo* etc.
            subPath: cacerts
          - name: volume-ca-bundle
            mountPath: /etc/ssl/certs/ca-certificates.crt # Alpine / Debian / Ubuntu / Gentoo etc.
            subPath: ca-certificates.crt
      volumes:
      - name: volume-ca-bundle
        configMap:
          name: ca-bundle

Important Note: If running CloudBees Core 2.138.1.2 or 2.138.2.2, there is a known issue that requires some extra configuration for this to work. Please have a look at Master Provisioning fails due to an invalid spec.selector

Note: This configuration only applied to newly created Managed Masters. For existing Managed Masters, the same configuration needs to be applied in the Managed Master configuration and the master needs to be re-provisioned.

Deploy Certificates in the Agents

Go to the configuration of the Pod Template set the Raw yaml for the Pod like the following (example for Debian):

apiVersion: v1
kind: Pod
spec:
    containers:
    - name: "jnlp"
      volumeMounts:
      - name: volume-ca-bundle
        mountPath: /etc/ssl/certs/java/cacerts # *Alpine* / *Debian* / *Ubuntu* / *Gentoo* etc.
        subPath: cacerts
      - name: volume-ca-bundle
        mountPath: /etc/ssl/certs/ca-certificates.crt # *Alpine* / *Debian* / *Ubuntu* / *Gentoo* etc.
        subPath: ca-certificates.crt
    volumes:
    - name: volume-ca-bundle
      configMap:
        name: ca-bundle

Note: This apply the configuration for the jnlp container. If there have several containers that need the certificates, the same can be applied to other containers, for example:

apiVersion: v1
kind: Pod
spec:
    containers:
    - name: "jnlp"
      volumeMounts:
      - name: volume-ca-bundle
        mountPath: /etc/ssl/certs/java/cacerts # *Alpine* / *Debian* / *Ubuntu* / *Gentoo* etc.
        subPath: cacerts
      - name: volume-ca-bundle
        mountPath: /etc/ssl/certs/ca-certificates.crt # *Alpine* / *Debian* / *Ubuntu* / *Gentoo* etc.
        subPath: ca-certificates.crt
    - name: "dind"
      - name: volume-ca-bundle
        mountPath: /etc/ssl/certs/java/cacerts # *Alpine* / *Debian* / *Ubuntu* / *Gentoo* etc.
        subPath: cacerts
      - name: volume-ca-bundle
        mountPath: /etc/ssl/certs/ca-certificates.crt # *Alpine* / *Debian* / *Ubuntu* / *Gentoo* etc.
        subPath: ca-certificates.crt
    volumes:
    - name: volume-ca-bundle
      configMap:
        name: ca-bundle

Template Inheritance

When such configuration needs to be applied to multiple agents, Pod Template inheritance can be used.

Troubleshooting

Issue

I see this error when deploying a master after installing the certificates:

java.io.IOException: Invalid keystore format

Solution

This is most likely due to the keystore not being saved in a binary format in the ConfigMap. Please make sure you are using Kubernetes v1.10 or later as it is required to save the keystore in binary format.

Resources

• Configure a Pod to Use a ConfigMap
• How to install a new SSL certificate?
• SSL Certificates Troubleshooting