- Although the CJOC certificate has been imported in the client master’s truststore, connection of the Client Master to Operations Center fails due to TLS handshake exception:
Agent discovery failed: TLS Handshake exception establishing connection to Jenkins server: <CJOC_URL>. You might need to trust server's self-signed certificate on global security configuration.
- Although the certificates presented by CJOC’s URL have been imported in the client master’s truststore, connection of the Client Master to Operations Center fails due to TLS handshake exception pointing to the “Kubernetes Ingress Controller Fake Certificate”:
TLS hostname verification failure establishing connection to Jenkins server: <CJOC_URL> Certificate subject: CN=Kubernetes Ingress Controller Fake Certificate, O=Acme Co issuer: CN=Kubernetes Ingress Controller Fake Certificate, O=Acme Co
- The client master logs shows something similar to:
WARNING: Pre-validation discovery on <CJOC_URL> failed javax.net.ssl.SSLHandshakeException: TLS Handshake exception establishing connection to Jenkins server: https://oc.jenkins.example.com:8888/. You might need to trust server's self-signed certificate on global security configuration. at com.cloudbees.opscenter.agent.AgentProtocolEndpointLocator.locate(AgentProtocolEndpointLocator.java:530) at com.cloudbees.opscenter.client.plugin.OperationsCenterRegistrar$PushRegistrationConfirmation.<init>(OperationsCenterRegistrar.java:500) at com.cloudbees.opscenter.client.plugin.OperationsCenterRegistrar$DescriptorImpl.doPushRegistration(OperationsCenterRegistrar.java:316) [...] Caused by: java.util.concurrent.ExecutionException: java.net.ConnectException: General SSLEngine problem to <CJOC_URL>/instance-identity/
- CloudBees CI (CloudBees Core) on Modern Cloud Platforms < 18.104.22.168
- Operations Center Agent < 22.214.171.124
- Ingress TLS Termination
- Adding Client Masters Generic
- Adding Client Masters in AKS
- Adding Client Masters in EKS
- Adding Client Masters in GKE
- Adding Client Masters in Openshift
- Adding Client Masters in PKS
- CTR-1650: Support SNI in the connection from masters to OC
The Kubernetes Nginx Controller supports and uses Server Name Indication (SNI). In case where the server name provided by the client (i.e. the client master) does not match the configured server name (i.e. CJOC server name) the default behavior of the Nginx Ingress Controller is to present an automatically generated SSL certificate “Kubernetes Ingress Controller Fake Certificate”. This auto-generated SSL certificate is also used if the TLS termination is not setup properly - for example if the TLS secret is wrong.
Before CloudBees Core 126.96.36.199, the CJOC / Master connection protocol did not support SNI, which was tracked internally as CTR-1650. For that reason, the connection of external Client Master to an Operations Center hosted in Kubernetes may fail the SSL handshake if the TLS termination is set up at Ingress Level. In such a case, the push of connection details would show something similar to:
On the versions prior to 188.8.131.52, this is most likely due to CTR-1650. But could well be due to an issue with the Kubernetes secret. The Troubleshooting section below can help to confirm this.
To understand if the TLS handshake exception is related to the SNI limitation CTR-1650, the more straightforward way to check on this is to carry out those two checks in a terminal session outside of Kubernetes:
- Check TLS termination without SNI:
openssl s_client -showcerts -connect $CJOC_FQDN:443
- Check TLS termination with SNI:
openssl s_client -showcerts -servername $CJOC_FQDN -connect $CJOC_FQDN:443
If the server presents different certificates with SNI and without SNI, then the handshake failure is most likely related to CTR-1650. Based on the output of those command:
- If SSL termination with SNI shows the expected certificate(s) and TLS termination without SNI shows the “Kubernetes Ingress Controller Fake Certificate”: the client master cannot be connected because of CTR-1650. See the workaround below.
- If SSL termination with SNI shows the expected certificate(s) and TLS termination without SNI shows the expected certificate(s), the client master lacks the expected certificate(s) in the JVM truststore.
- If SSL termination with SNI shows the “Kubernetes Ingress Controller Fake Certificate”, the SSL Termination is not setup properly. A way to troubleshoot this further is to tail the logs of the ingress controller - with a command like
kubectl -n ingress-nginx logs -f nginx-ingress-controller-xxxxx- and then reach the CJOC URL with either
curl -IvL $CJOC_URLor a browser. The output of the inggress nginx controller should show provide enough information to understand the root cause.
(Note: This article assumes that the CJOC’s certificate has been added / imported to the Client Master’s truststore. See How to connect an HTTPS Client Master to CJOC.)
The solution is to upgrade CloudBees Core to version 184.108.40.206 or later.
The workaround is to specify a default SSL certificate for the Ingress Controller. A default certificate can be specified to the Nginx Ingress Controller as a fallback for requests that do not use SNI. This is documented at NGINX Ingress Controller - Default SSL Certificate.
With Helm use the attribute
controller.extraArgs.default-ssl-certificate of the
nginx-ingress chart to specify the TLS secret to use as a default certificate.
The following custom values file is an example based on examples in cloudbees-examples GitHub repository.
nginx-ingress: Enabled: true controller: extraArgs: default-ssl-certificate: "$(POD_NAMESPACE)/core-example-com-tls" OperationsCenter: HostName: 'cloudbees-core.example.com' ServiceType: ClusterIP Ingress: tls: Enable: true SecretName: core-example-com-tls Host: jenkins.cluster.local
The following custom values file is an example if the Nginx Ingress Controller is installed separately in a different namespace
controller: extraArgs: default-ssl-certificate: "<namespace>/<secretName>"
<namespace>is the namespace where the Operations Center is deployed
<secretName>is the name of the TLS secret used in