Issue

Although the CJOC certificate has been imported in the client controller’s truststore, connection of the Client controller to Operations Center fails due to TLS handshake exception:

Agent discovery failed: TLS Handshake exception establishing connection to Jenkins server: <CJOC_URL>. You might need to trust server's self-signed certificate on global security configuration.

Although the certificates presented by CJOC’s URL have been imported in the client controller’s truststore, connection of the Client controller to Operations Center fails due to TLS handshake exception pointing to the “Kubernetes Ingress Controller Fake Certificate”:

TLS hostname verification failure establishing connection to Jenkins server: <CJOC_URL> Certificate subject: CN=Kubernetes Ingress Controller Fake Certificate, O=Acme Co issuer: CN=Kubernetes Ingress Controller Fake Certificate, O=Acme Co

The client controller logs shows something similar to:

WARNING: Pre-validation discovery on <CJOC_URL> failed
javax.net.ssl.SSLHandshakeException: TLS Handshake exception establishing connection to Jenkins server: https://oc.jenkins.example.com:8888/. You might need to trust server's self-signed certificate on global security configuration.
	at com.cloudbees.opscenter.agent.AgentProtocolEndpointLocator.locate(AgentProtocolEndpointLocator.java:530)
	at com.cloudbees.opscenter.client.plugin.OperationsCenterRegistrar$PushRegistrationConfirmation.<init>(OperationsCenterRegistrar.java:500)
	at com.cloudbees.opscenter.client.plugin.OperationsCenterRegistrar$DescriptorImpl.doPushRegistration(OperationsCenterRegistrar.java:316)
	[...]
Caused by: java.util.concurrent.ExecutionException: java.net.ConnectException: General SSLEngine problem to  <CJOC_URL>/instance-identity/

Environment

Related Issue

CTR-1650: Support SNI in the connection from masters to OC

Explanation

The Kubernetes Nginx Controller supports and uses Server Name Indication (SNI). In case where the server name provided by the client (i.e. the client controller) does not match the configured server name (i.e. CJOC server name) the default behavior of the Nginx Ingress Controller is to present an automatically generated SSL certificate “Kubernetes Ingress Controller Fake Certificate”. This auto-generated SSL certificate is also used if the TLS termination is not setup properly - for example if the TLS secret is wrong.

Before CloudBees Core 2.222.4.3, the CJOC / Master connection protocol did not support SNI, which was tracked internally as CTR-1650. For that reason, the connection of external Client controller to an Operations Center hosted in Kubernetes may fail the SSL handshake if the TLS termination is set up at Ingress Level. In such a case, the push of connection details would show something similar to:

On the versions prior to 2.222.4.3, this is most likely due to CTR-1650. But could well be due to an issue with the Kubernetes secret. The Troubleshooting section below can help to confirm this.

Troubleshooting

To understand if the TLS handshake exception is related to the SNI limitation CTR-1650, the more straightforward way to check on this is to carry out those two checks in a terminal session outside of Kubernetes:

Check TLS termination without SNI: openssl s_client -showcerts -connect $CJOC_FQDN:443
Check TLS termination with SNI: openssl s_client -showcerts -servername $CJOC_FQDN -connect $CJOC_FQDN:443

If the server presents different certificates with SNI and without SNI, then the handshake failure is most likely related to CTR-1650. Based on the output of those command:

If SSL termination with SNI shows the expected certificate(s) and TLS termination without SNI shows the “Kubernetes Ingress Controller Fake Certificate”: the client controller cannot be connected because of CTR-1650. See the workaround below.
If SSL termination with SNI shows the expected certificate(s) and TLS termination without SNI shows the expected certificate(s), the client controller lacks the expected certificate(s) in the JVM truststore.
If SSL termination with SNI shows the “Kubernetes Ingress Controller Fake Certificate”, the SSL Termination is not setup properly. A way to troubleshoot this further is to tail the logs of the ingress controller - with a command like kubectl -n ingress-nginx logs -f nginx-ingress-controller-xxxxx - and then reach the CJOC URL with either curl -IvL $CJOC_URL or a browser. The output of the inggress nginx controller should show provide enough information to understand the root cause.

Resolution

(Note: This article assumes that the CJOC’s certificate has been added / imported to the Client controller’s truststore. See How to connect an HTTPS Client controller to CJOC.)

Solution

The solution is to upgrade CloudBees Core to version 2.222.4.3 or later.

Workaround

The workaround is to specify a default SSL certificate for the Ingress Controller. A default certificate can be specified to the Nginx Ingress Controller as a fallback for requests that do not use SNI. This is documented at NGINX Ingress Controller - Default SSL Certificate.

With Helm use the attribute controller.extraArgs.default-ssl-certificate of the nginx-ingress chart to specify the TLS secret to use as a default certificate.

The following custom values file is an example based on examples in cloudbees-examples GitHub repository.

 nginx-ingress:
   Enabled: true
   controller:
     extraArgs:
       default-ssl-certificate: "$(POD_NAMESPACE)/core-example-com-tls"
 OperationsCenter:
   HostName: 'cloudbees-core.example.com'
   ServiceType: ClusterIP
   Ingress:
     tls:
       Enable: true
       SecretName: core-example-com-tls
       Host: jenkins.cluster.local

The following custom values file is an example if the Nginx Ingress Controller is installed separately in a different namespace

controller:
  extraArgs:
    default-ssl-certificate: "<namespace>/<secretName>"

where:

<namespace> is the namespace where the Operations Center is deployed
<secretName> is the name of the TLS secret used in cjoc ingress

Issue connecting External Client controller with TLS setup at Ingress