Deploy Self Signed Certificates in Masters and Agents (Custom Location)

Issue

  • I would like to add / import self signed certificates to JVM trustore of Masters and / or Agents
  • I would like to add / import self signed certificates to use with tool like git and curl
  • Container images do NOT have ca-certificates installed

Environment

Resolution

The recommended solution at the moment is to use ConfigMaps:

  • 1) Create a ConfigMap that holds cacerts and ca-certificates.crt
  • 2) Mount the ConfigMaps to the Containers that require them - i.e. CJOC, Masters, agents - and set up the pod / containers to point tools to the location of the certificates bundle.

The advantage of this approach is that whenever a new certificates need to be added, it can be done by updating the ConfigMap and restarting the resource that mounts it.

Pre-requisites

  • Kubernetes version 1.10 which provide support for binaries in ConfigMaps - necessary for cacerts

1) Create the ConfigMaps

First, a ConfigMap needs to be created to hold:

  • a preconfigured JVM keystore cacerts
  • a preconfigured CA bundle ca-certificates.crt

Create a directory:

mkdir ca-bundle

Prepare the cacerts

Create / Copy a JVM trustore. You can copy the trustore from an existing JVM installation, for example from the CJOC container:

kubectl cp $CJE_NAMESPACE/cjoc-0:/etc/ssl/certs/java/cacerts $(pwd)/ca-bundle/cacerts

Then import certificates to the cacerts trustore - in this example cje.example.com.pem:

keytool -import -alias cje.example.com -keystore ca-bundle/cacerts -file cje.example.com.pem

Prepare the ca-certificates.crt

Create / Copy a ca-certificates.crt. You can copy the ca-certificates.crt from an existing container, for example from the CJOC container:

kubectl cp $CJE_NAMESPACE/cjoc-0:/etc/ssl/certs/ca-certificates.crt $(pwd)/ca-bundle/ca-certificates.crt

Then add certificates to the ca-certificates.crt. The format should look like the following:

[...]
-----BEGIN CERTIFICATE-----
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-----END CERTIFICATE-----

Create the ConfigMap

The directory ca-bundle should have the following files:

ls ca-bundle/
ca-certificates.crt	cacerts

Create a ConfigMap from that directory:

kubectl create configmap ca-bundle --from-file=ca-bundle -n $CJE_NAMESPACE
configmap "ca-bundle" created

2) Deploy the Certificates

To deploy the certificates:

  • the ConfigMap needs to be mounted to the containers in a specific location - for example /var/certs/
  • environment variables / system properties need to be set at the pod / container level to tools to those locations

Note: In this article we provide example for git with GIT_SSL_CAINFO and curl with CURL_CA_BUNDLE. Other tools might require other environment variables. For example NPM uses NODE_EXTRA_CA_CERTS.

Deploy Certificates in the CJOC

In the cloudbees-core.yaml edit the cjoc statefulset and add:

  • the volume / volumes mounts
  • the system properties -Djavax.net.ssl.trustStore and -Djavax.net.ssl.trustStorePassword to the JAVA_OPTS environment variable
  • the environment variable for the tools using the

Following is an example:

---
apiVersion: "apps/v1beta1"
kind: "StatefulSet"
metadata:
  name: cjoc
  labels:
    com.cloudbees.cje.type: cjoc
    com.cloudbees.cje.tenant: cjoc
spec:
  serviceName: cjoc
  replicas: 1
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      name: cjoc
      labels:
        com.cloudbees.cje.type: cjoc
        com.cloudbees.cje.tenant: cjoc
    spec:
      serviceAccountName: cjoc
      terminationGracePeriodSeconds: 10
      containers:
      - name: jenkins
        image: cloudbees/cloudbees-cloud-core-oc:2.138.2.2
        env:
        - name: ENVIRONMENT
          value: KUBERNETES
        - name: JENKINS_OPTS
          value: --prefix=/cjoc
        - name: CURL_CA_BUNDLE
          value: /var/certs/ca-certificates.crt
        - name: GIT_SSL_CAINFO
          value: /var/certs/ca-certificates.crt
        - name: JAVA_OPTS
          # To allocate masters using a non-default storage class, add the following
          # -Dcom.cloudbees.masterprovisioning.kubernetes.KubernetesMasterProvisioning.storageClassName=some-storage-class
          value: >-
            -XshowSettings:vm
            -XX:MaxRAM=${MEM_LIMIT}m
            -XX:MaxRAMFraction=1
            -XX:+PrintGCDetails
            -Djavax.net.ssl.trustStore=/var/certs/cacerts
            -Djavax.net.ssl.trustStorePassword=changeit
            -Dcb.IMProp.warProfiles=kubernetes.json
            -Dcb.IMProp.warProfiles.cje=kubernetes.json
            -Dcom.cloudbees.opscenter.analytics.reporter.JocAnalyticsReporter.PERIOD=120
            -Dcom.cloudbees.opscenter.analytics.reporter.metrics.AperiodicMetricSubmitter.PERIOD=120
            -Dcom.cloudbees.opscenter.analytics.FeederConfiguration.PERIOD=120
            -Dcom.cloudbees.masterprovisioning.kubernetes.KubernetesMasterProvisioning.fsGroup=1000
            -Dhudson.lifecycle=hudson.lifecycle.ExitLifecycle
            -Dcom.cloudbees.jce.masterprovisioning.DockerImageDefinitionConfiguration.disableAutoConfiguration=true
            -Dcom.cloudbees.jce.masterprovisioning.DockerImageDefinitionConfiguration.masterImageName="CloudBees Core - Managed Master 2.138.2.2"
            -Dcom.cloudbees.jce.masterprovisioning.DockerImageDefinitionConfiguration.masterImage=cloudbees/cloudbees-core-mm:2.138.2.2
        volumeMounts:
        - name: volume-ca-bundle
          mountPath: /var/certs
        - name: jenkins-home
          mountPath: /var/jenkins_home
        - name: jenkins-configure-jenkins-groovy
          mountPath: /var/jenkins_config/configure-jenkins.groovy.d
        livenessProbe:
          httpGet:
            path: /cjoc/login
            port: 8080
          initialDelaySeconds: 300
          timeoutSeconds: 5
      volumes:
      - name: jenkins-configure-jenkins-groovy
        configMap:
          name: cjoc-configure-jenkins-groovy
      - name: volume-ca-bundle
        configMap:
          name: ca-bundle
[...]

Apply the config with:

kubectl apply -f cloudbees.core.yaml -n $CJE_NAMESPACE

CJOC should be redeployed automatically.

Deploy Certificates in the Managed Master

Go to Manage Jenkins > Configure System > Kubernetes Master Provisioning > Advanced

Add the Global Properties:

javax.net.ssl.trustStore=/var/certs/cacerts
javax.net.ssl.trustStorePassword=changeit

Add the Global Environment Variables:

CURL_CA_BUNDLE=/var/certs/ca-certificates.crt
GIT_SSL_CAINFO=/var/certs/ca-certificates.crt

Set the YAML:

apiVersion: "apps/v1"
kind: "StatefulSet"
spec:
  template:
    spec:
      containers:
        - name: "jenkins"
          volumeMounts:
          - name: volume-ca-bundle
            mountPath: /var/certs
      volumes:
      - name: volume-ca-bundle
        configMap:
          name: ca-bundle

Important Note: If running CloudBees Core 2.138.1.2 or 2.138.2.2, there is a known issue that requires some extra configuration for this to work. Please have a look at Master Provisioning fails due to an invalid spec.selector

Note: This configuration only applied to newly created Managed Masters. For existing Managed Masters, the same configuration needs to be applied in the Managed Master configuration and the master needs to be re-provisioned.

Deploy Certificates in the Agents

Go to the configuration of the Pod Template.

Add Environment Variables to the Pod template:

For Java:

  • Key: JAVA_OPTS
  • Value: -Djavax.net.ssl.trustStore=/var/certs/cacerts -Djavax.net.ssl.trustStorePassword=changeit

For git:

  • Key: CURL_CA_BUNDLE
  • Value: /var/certs/ca-certificates.crt

For tools curl:

  • Key: GIT_SSL_CAINFO
  • Value: /var/certs/ca-certificates.crt

Add Volumes of type ConfigMap Volume to the Pod template:

  • Config Map name: ca-bundle
  • Mount path: /var/certs/

For example:

deploy-ssl-agents-custom.png

Note: This apply the configuration for the jnlp container. If there have several containers that need the certificates, the same can be applied to other containers.

Template Inheritance

When such configuration needs to be applied to multiple agents, Pod Template inheritance can be used.

Resources

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.