Issue
- I would like to add / import self signed certificates to JVM truststore of Masters and / or Agents
- I would like to add / import self signed certificates to use with tool like
git
andcurl
- Container images have ca-certificates installed
NOTE: This article describes the process to follow on Kubernetes based installation, for non Kubernetes based installations please refer to this article
Environment
- CloudBees CI (CloudBees Core) on Modern Cloud Platforms
- Kubernetes Plugin
- Kubernetes v1.10 and later
Resolution
Since version 2.138.3.1, the recommended solution is to use a sidecar-injector as explained Using self-signed certificates in CloudBees Jenkins Enterprise.
Otherwise, there is another solution using ConfigMaps like below.
Solution with ConfigMaps
For version lower than 2.138.3.1, the following solution can be used:
- 1) Create a ConfigMap that holds
cacerts
andca-certificates.crt
- 2) Mount the ConfigMaps to the Containers that require them - i.e. CJOC, Masters, agents - and set up the pod / containers to point tools to the location of the certificates bundle.
The advantage of this approach is that whenever a new certificates need to be added, it can be done by updating the ConfigMap and restarting the resource that mounts it.
Pre-requisites
- ca-certificates is installed in the image / containers where certificates need to be imported. At the moment of writing this article,
cloudbees-core
images are based onopenjdk
over Debian stretch and therefore haveca-certificates
installed` - Kubernetes version 1.10 which provide support for binaries in ConfigMaps - necessary for
cacerts
1) Create the ConfigMap
First, a ConfigMap needs to be created to hold:
- a preconfigured JVM keystore
cacerts
- a preconfigured CA bundle
ca-certificates.crt
Create a directory:
mkdir ca-bundle
Prepare the cacerts
Create / Copy a JVM truststore. You can copy the truststore from an existing JVM installation, for example from the CJOC container.
For CloudBees Core Modern 2.204.1.3 and later:
kubectl cp $CJE_NAMESPACE/cjoc-0:/etc/pki/java/cacerts $(pwd)/ca-bundle/cacerts
For earlier versions:
kubectl cp $CJE_NAMESPACE/cjoc-0:/etc/ssl/certs/java/cacerts $(pwd)/ca-bundle/cacerts
Then import certificates to the cacerts
truststore - in this example cje.example.com.pem
:
keytool -import -alias cje.example.com -keystore ca-bundle/cacerts -file cje.example.com.pem
Prepare the ca-certificates.crt
Create / Copy a ca-certificates.crt
. You can copy the ca-certificates.crt
from an existing container, for example from the CJOC container.
For CloudBees Core Modern 2.204.1.3 and later:
kubectl cp $CJE_NAMESPACE/cjoc-0:/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem $(pwd)/ca-bundle/ca-certificates.crt
For earlier versions:
kubectl cp $CJE_NAMESPACE/cjoc-0:/etc/ssl/certs/ca-certificates.crt $(pwd)/ca-bundle/ca-certificates.crt
Then add certificates to the ca-certificates.crt
. The format should look like the following:
[...]
-----BEGIN CERTIFICATE-----
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-----END CERTIFICATE-----
Create the ConfigMap
The directory ca-bundle
should have the following files:
ls ca-bundle/
ca-certificates.crt cacerts
Create a ConfigMap from that directory:
kubectl create configmap ca-bundle --from-file=ca-bundle -n $CJE_NAMESPACE
configmap "ca-bundle" created
2) Deploy the Certificates
To deploy the certificate, the ConfigMap needs to be mounted to the containers.
With ca-certificates
installed, the cacerts
/ ca-certificates.crt
can be injected in specific locations which depends on the underlying distribution in the container:
- CentOS 7 / RHEL 7:
/etc/pki/java/cacerts
and/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
(such as CloudBees Core since version 2.204.1.3) - Alpine / Debian / Ubuntu / Gentoo etc.:
/etc/ssl/certs/ca-certificates.crt
and/etc/ssl/certs/java/cacerts
(such as CloudBees Core since before version 2.204.1.3) - CentOS 6 / RHEL 6:
/etc/pki/java/cacerts
and/etc/ssl/certs/ca-bundle.crt
Note: If not sure, it could well be mounted to all locations.
Deploy Certificates in the CJOC
In the cloudbees-core.yaml
edit the cjoc
statefulset and add the volume / volumes mounts similar to the following:
---
apiVersion: "apps/v1beta1"
kind: "StatefulSet"
metadata:
name: cjoc
labels:
com.cloudbees.cje.type: cjoc
com.cloudbees.cje.tenant: cjoc
spec:
serviceName: cjoc
replicas: 1
updateStrategy:
type: RollingUpdate
template:
metadata:
name: cjoc
labels:
com.cloudbees.cje.type: cjoc
com.cloudbees.cje.tenant: cjoc
spec:
serviceAccountName: cjoc
terminationGracePeriodSeconds: 10
containers:
- name: jenkins
image: cloudbees/cloudbees-cloud-core-oc:2.138.2.2
[...]
volumeMounts:
- name: volume-ca-bundle
mountPath: /etc/pki/java/cacerts # CentOS / RHEL 6-7 etc.
subPath: cacerts
- name: volume-ca-bundle
mountPath: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem # CentOS / RHEL 6-7 etc.
subPath: ca-certificates.crt
# Following for version of CloudBees Core on Modern Platform earlier than 2.204.1.3
- name: volume-ca-bundle
mountPath: /etc/ssl/certs/java/cacerts # Alpine / Debian / Ubuntu / Gentoo etc.
subPath: cacerts
- name: volume-ca-bundle
mountPath: /etc/ssl/certs/ca-certificates.crt # Alpine / Debian / Ubuntu / Gentoo etc.
subPath: ca-certificates.crt
- name: jenkins-home
mountPath: /var/jenkins_home
- name: jenkins-configure-jenkins-groovy
mountPath: /var/jenkins_config/configure-jenkins.groovy.d
livenessProbe:
httpGet:
path: /cjoc/login
port: 8080
initialDelaySeconds: 300
timeoutSeconds: 5
volumes:
- name: jenkins-configure-jenkins-groovy
configMap:
name: cjoc-configure-jenkins-groovy
- name: volume-ca-bundle
configMap:
name: ca-bundle
[...]
Apply the config with:
kubectl apply -f cloudbees.core.yaml -n $CJE_NAMESPACE
CJOC needs to be redeployed for the change to take effect.
Deploy Certificates in the Managed Master
Go to Manage Jenkins > Configure System > Kubernetes Master Provisioning > Advanced and set the YAML like the following (example for Debian):
apiVersion: "apps/v1"
kind: "StatefulSet"
spec:
template:
spec:
containers:
- name: "jenkins"
volumeMounts:
- name: volume-ca-bundle
mountPath: /etc/pki/java/cacerts # CentOS / RHEL 6-7 etc.
subPath: cacerts
- name: volume-ca-bundle
mountPath: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem # CentOS / RHEL 6-7 etc.
subPath: ca-certificates.crt
# Following for version of CloudBees Core on Modern Platform earlier than 2.204.1.3
- name: volume-ca-bundle
mountPath: /etc/ssl/certs/java/cacerts # Alpine / Debian / Ubuntu / Gentoo etc.
subPath: cacerts
- name: volume-ca-bundle
mountPath: /etc/ssl/certs/ca-certificates.crt # Alpine / Debian / Ubuntu / Gentoo etc.
subPath: ca-certificates.crt
volumes:
- name: volume-ca-bundle
configMap:
name: ca-bundle
Important Note: If running CloudBees Core 2.138.1.2 or 2.138.2.2, there is a known issue that requires some extra configuration for this to work. Please have a look at Master Provisioning fails due to an invalid spec.selector
Note: This configuration only applied to newly created Managed Masters. For existing Managed Masters, the same configuration needs to be applied in the Managed Master configuration and the master needs to be re-provisioned.
Deploy Certificates in the Agents
Go to the configuration of the Pod Template set the Raw yaml for the Pod like the following (example for Debian):
apiVersion: v1
kind: Pod
spec:
containers:
- name: "jnlp"
volumeMounts:
- name: volume-ca-bundle
mountPath: /etc/pki/java/cacerts # CentOS / RHEL 6-7 etc.
subPath: cacerts
- name: volume-ca-bundle
mountPath: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem # CentOS / RHEL 6-7 etc.
subPath: ca-certificates.crt
# Following for version of CloudBees Core on Modern Platform earlier than 2.204.1.3
- name: volume-ca-bundle
mountPath: /etc/ssl/certs/java/cacerts # Alpine / Debian / Ubuntu / Gentoo etc.
subPath: cacerts
- name: volume-ca-bundle
mountPath: /etc/ssl/certs/ca-certificates.crt # Alpine / Debian / Ubuntu / Gentoo etc.
subPath: ca-certificates.crt
volumes:
- name: volume-ca-bundle
configMap:
name: ca-bundle
Note: This apply the configuration for the jnlp
container. If there have several containers that need the certificates, the same can be applied to other containers, for example:
apiVersion: v1
kind: Pod
spec:
containers:
- name: "jnlp"
volumeMounts:
- name: volume-ca-bundle
mountPath: /etc/pki/java/cacerts # CentOS / RHEL 6-7 etc.
subPath: cacerts
- name: volume-ca-bundle
mountPath: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem # CentOS / RHEL 6-7 etc.
subPath: ca-certificates.crt
# Following for version of CloudBees Core on Modern Platform earlier than 2.204.1.3
- name: volume-ca-bundle
mountPath: /etc/ssl/certs/java/cacerts # Alpine / Debian / Ubuntu / Gentoo etc.
subPath: cacerts
- name: volume-ca-bundle
mountPath: /etc/ssl/certs/ca-certificates.crt # Alpine / Debian / Ubuntu / Gentoo etc.
subPath: ca-certificates.crt
- name: "dind"
- name: volume-ca-bundle
mountPath: /etc/pki/java/cacerts # CentOS / RHEL 6-7 etc.
subPath: cacerts
- name: volume-ca-bundle
mountPath: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem # CentOS / RHEL 6-7 etc.
subPath: ca-certificates.crt
# Following for version of CloudBees Core on Modern Platform earlier than 2.204.1.3
- name: volume-ca-bundle
mountPath: /etc/ssl/certs/java/cacerts # Alpine / Debian / Ubuntu / Gentoo etc.
subPath: cacerts
- name: volume-ca-bundle
mountPath: /etc/ssl/certs/ca-certificates.crt # Alpine / Debian / Ubuntu / Gentoo etc.
subPath: ca-certificates.crt
volumes:
- name: volume-ca-bundle
configMap:
name: ca-bundle
Template Inheritance
When such configuration needs to be applied to multiple agents, Pod Template inheritance can be used.
Troubleshooting
Issue
I see this error when deploying a master after installing the certificates:
java.io.IOException: Invalid keystore format
Solution
This is most likely due to the keystore not being saved in a binary format in the ConfigMap. Please make sure you are using Kubernetes v1.10 or later as it is required to save the keystore in binary format.
0 Comments