Issue

• After importing certificates using the certificates-update operation, I am experiencing SSL issues in Jenkins similar to:

stderr: fatal: unable to access '<url>': error setting certificate verify locations:
  CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none

Environment

• CloudBees Jenkins Enterprise (CJE) - AWS/Anywhere

Related Issue

• https://bugs.alpinelinux.org/issues/8379

Resolution

This exception suggests that the CA certificate located at <CAfile> is wrong or not accessible.

The certificates-update operations copies the certificates in the bastion /certificates directory to the workers / controllers under /etc/ssl/certs/. All the certificates under /etc/ssl/certs/ are concatenated to produce one file /etc/ssl/certs/ca-certificates.crt (using update-ca-certificates). The directory /etc/ssl/certs/ is also mounted to the certs container on workers and controllers. And the certs container is mounted to CJE tenants containers (i.e. cjoc, masters, elasticsearch, palace). This mechanic allows the certificates to be automatically updated in the tenant’s containers by simply updating the certs container.

A common issue occurs when the certificate(s) imported are not correctly formatted. This causes the concatenation to produce a corrupted ca-certificate.crt. In which case the certificates are not correctly loaded by the tenants and the issue “unable to access ‘’: error setting certificate verify locations” arises. Some examples are:

a. Missing line break at the end of the file. If one or more certificates miss a line break at the end, the concatenated certificate /etc/ssl/certs/ca-certificates.crt on your workers/controllers would be corrupted and contains lines like the following, which is wrong:

[...]
-----END CERTIFICATE----------BEGIN CERTIFICATE-----
[...]

The expected concatenated file should show each certificate separated by a line break:

[...]
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
[...]

b. Wrong characters at the certificates END or BEGIN closure (e.g. -----END CERTIFICATE-----')

[...]
-----END CERTIFICATE`-----
-----BEGIN CERTIFICATE-----
[...]

Validation

The following example illustrates a failure with the certificates:

$> awk -v cmd='openssl x509 -noout -subject' '/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt | grep -i error

unable to load certificate
9829:error:0906D066:PEM routines:PEM_read_bio:bad end line:/BuildRoot/Library/Caches/com.apple.xbs/Sources/OpenSSL098/OpenSSL098-59.60.2/src/crypto/pem/pem_lib.c:747:
unable to load certificate
9830:error:0906D06C:PEM routines:PEM_read_bio:no start line:/BuildRoot/Library/Caches/com.apple.xbs/Sources/OpenSSL098/OpenSSL098-59.60.2/src/crypto/pem/pem_lib.c:648:Expecting: TRUSTED CERTIFICATE

Note: it has been tested with alpine. For other images, it might change (ref)

How to fix it

Fixing the formatting issues with .pem files in certificates directory of the CJE project.

Then, depending on the CJE version:

### Before CJE 1.11.11

1. First, wipe the existing custom certificates for your workers and controllers under /etc/ssl/cert/.

2. Run the certificates-update operation again.

### Since CJE 1.11.11

Just run the certificates-update operation again, because it will wipe out the old certificates for you.

(Note: this has been filed as a bug against Alpine Linux - see https://bugs.alpinelinux.org/issues/8379 - that is the base image of the certs container.)