Issue
- After importing certificates using the certificates-update operation, I am experiencing SSL issues in Jenkins similar to:
stderr: fatal: unable to access '<url>': error setting certificate verify locations:
CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: none
Environment
- CloudBees Jenkins Enterprise (CJE) - AWS/Anywhere
Related Issue
Resolution
This exception suggests that the CA certificate located at <CAfile>
is wrong or not accessible.
The certificates-update operations copies the certificates in the bastion /certificates
directory to the workers / controllers under /etc/ssl/certs/
. All the certificates under /etc/ssl/certs/
are concatenated to produce one file /etc/ssl/certs/ca-certificates.crt
(using update-ca-certificates
). The directory /etc/ssl/certs/
is also mounted to the certs
container on workers and controllers. And the certs
container is mounted to CJE tenants containers (i.e. cjoc, masters, elasticsearch, palace). This mechanic allows the certificates to be automatically updated in the tenant’s containers by simply updating the certs
container.
A common issue occurs when the certificate(s) imported are not correctly formatted. This causes the concatenation to produce a corrupted ca-certificate.crt
. In which case the certificates are not correctly loaded by the tenants and the issue “unable to access ‘
a. Missing line break at the end of the file. If one or more certificates miss a line break at the end, the concatenated certificate /etc/ssl/certs/ca-certificates.crt
on your workers/controllers would be corrupted and contains lines like the following, which is wrong:
[...]
-----END CERTIFICATE----------BEGIN CERTIFICATE-----
[...]
The expected concatenated file should show each certificate separated by a line break:
[...]
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
[...]
b. Wrong characters at the certificates END
or BEGIN
closure (e.g. -----END CERTIFICATE-----'
)
[...]
-----END CERTIFICATE`-----
-----BEGIN CERTIFICATE-----
[...]
Validation
The following example illustrates a failure with the certificates:
$> awk -v cmd='openssl x509 -noout -subject' '/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt | grep -i error
unable to load certificate
9829:error:0906D066:PEM routines:PEM_read_bio:bad end line:/BuildRoot/Library/Caches/com.apple.xbs/Sources/OpenSSL098/OpenSSL098-59.60.2/src/crypto/pem/pem_lib.c:747:
unable to load certificate
9830:error:0906D06C:PEM routines:PEM_read_bio:no start line:/BuildRoot/Library/Caches/com.apple.xbs/Sources/OpenSSL098/OpenSSL098-59.60.2/src/crypto/pem/pem_lib.c:648:Expecting: TRUSTED CERTIFICATE
Note: it has been tested with alpine. For other images, it might change (ref)
How to fix it
Fixing the formatting issues with .pem
files in certificates
directory of the CJE project.
Then, depending on the CJE version:
### Before CJE 1.11.11
-
First, wipe the existing custom certificates for your workers and controllers under
/etc/ssl/cert/
. -
Run the certificates-update operation again.
### Since CJE 1.11.11
Just run the certificates-update operation again, because it will wipe out the old certificates for you.
(Note: this has been filed as a bug against Alpine Linux - see https://bugs.alpinelinux.org/issues/8379 - that is the base image of the certs
container.)
1 Comments