- Although the CJOC certificate has been imported in the client master’s truststore, connection of the Client Master to Operations Center fails due to TLS handshake exception:
Agent discovery failed: TLS Handshake exception establishing connection to Jenkins server: <CJOC_URL>. You might need to trust server's self-signed certificate on global security configuration.
- Although the certificates presented by CJOC’s URL have been imported in the client master’s truststore, connection of the Client Master to Operations Center fails due to TLS handshake exception pointing to the “Kubernetes Ingress Controller Fake Certificate”:
TLS hostname verification failure establishing connection to Jenkins server: <CJOC_URL> Certificate subject: CN=Kubernetes Ingress Controller Fake Certificate, O=Acme Co issuer: CN=Kubernetes Ingress Controller Fake Certificate, O=Acme Co
- CJP-5946: Upgrade AsyncHttpClient Library (to support SNI)
The Kubernetes Nginx Controller supports and uses Server Name Indication (SNI). In case where the server name provided by the client (i.e. the client master) does not match the configured server name (i.e. CJOC server name) the default behavior of the Nginx Ingress Controller is to present an automatically generated SSL certificate “Kubernetes Ingress Controller Fake Certificate”. This auto-generated SSL certificate is also used if the TLS termination is not setup properly - for example if the TLS secret is wrong.
At the moment of this article, the CJOC / Master connection protocol does not support SNI, this is tracked internally as CJP-5946. For that reason, the connection of external Client Master to a CJOC hosted in Kubernetes may fail the SSL handshake if the TLS termination is set up at Ingress Level. In such case, the push of connection details should show something similar to:
This is most likely due to CJP-5946. But could well be due to an issue with the kubernetes secret. The Troubleshooting section below can help to confirm this.
To understand if the TLS handshake exception is related to the SNI limitation CJP-5946, the more straightforward way to check on this is to carry out those two checks in a terminal session outside of Kubernetes:
- Check TLS termination without SNI:
openssl s_client -showcerts -connect $CJOC_FQDN:443
- Check TLS termination with SNI:
openssl s_client -showcerts -servername $CJOC_FQDN -connect $CJOC_FQDN:443
If the server presents different certificates with SNI and without SNI, then the handshake failure is most likely related to CJP-5946. Based on the output of those command:
- If SSL termination with SNI shows the expected certificate(s) and TLS termination without SNI shows the “Kubernetes Ingress Controller Fake Certificate”: the client master cannot be connected because of CJP-5946. See the workaround below.
- If SSL termination with SNI shows the expected certificate(s) and TLS termination without SNI shows the expected certificate(s), the client master lacks the expected certificate(s) in the JVM truststore.
- If SSL termination with SNI shows the “Kubernetes Ingress Controller Fake Certificate”, the SSL Termination is not setup properly. A way to troubleshoot this further is to tail the logs of the ingress controller - with a command like
kubectl -n ingress-nginx logs -f nginx-ingress-controller-xxxxx- and then reach the CJOC URL with either
curl -IvL $CJOC_URLor a browser. The output of the inggress nginx controller should show provide enough information to understand the root cause.
(Note: This article assumes that the CJOC’s certificate has been added / imported to the Client Master’s truststore. See How to connect an HTTPS Client Master to CJOC.)
The workaround is to define a default SSL certificate for the Ingress Controller. It consists of creating a TLS secret in the default namespace and configuring the Ingress controller to use that TLS secret by default.
Depending on the Cloud Platform, the reconfiguration of the ingress controller differs. Following are links for the different solutions:
- For Kubernetes Nginx Ingress Controller: https://kubernetes.github.io/ingress-nginx/user-guide/tls/#default-ssl-certificate
- For Rancher Nginx Ingress Controller(s): https://rancher.com/docs/rke/v0.1.x/en/config-options/add-ons/ingress-controllers/#configuring-an-nginx-default-certificate