How to troubleshoot client master connections?

Issue

Client master cannot be connected with the CJOC instance

Environment

  • CloudBees Operations Center (CJOC)
  • CloudBees Jenkins Enterprise (CJE)

Resolution

These are the steps you should follow to understand why your client master doesn’t connect correctly with the CJOC instance.

[CJOC] Ensure that Jenkins URL is correctly set-up

In the CJOC instance ensure that under Manage Jenkins -> Configure System the Jenkins URL is the right one. In case CJOC is accessible from a F5 or an ha-proxy ensure that this is correctly configure to forward the packages to CJOC from an external machine.

[CJOC] Ensure that a JNLP port is configured

In the CJOC instance ensure that under Manage Jenkins -> Configure Global security a JNLP port is configured. If your CJOC instance is running behind a F5 or ha-proxy then a fixed port is required. In any case, a fixed JNLP port is recommended.

[CJE] Ensure that the JNLP port is different

In the CJE instance ensure that under Manage Jenkins -> Configure Global security if a JNLP port is configured it is different from the one configured in CJOC. In any case, a fixed JNLP port is recommended.
Note: Use a different JNLP port for each of the CJE instances connected to the CJOC.

[CJE] Check the HTTP Proxy configuration

Go to $CJE_URL > Manage Jenkins > Plugin Manager > Advanced tab > HTTP Proxy Configuration section and check if there is any Server and Port configured. In the case they have been set up, FQN of the URL (without the port) must be included into No Proxy Host field.

Note: Alternatively, No Proxy Host can be configured by the JAVA property http.nonProxyHosts of the CJE instance.

[CJE] Check if CJOC instance is reachable through HTTP/(S)

In case you have access to the groovy console you need to go to http:///script and run:

def url = new URL("http://<CJOC_URL>/");
def connection = url.openConnection();
println("Response Headers");
println("================");
for (def e in connection.getHeaderFields()) {
  println("${e.key}: ${e.value}");
}
println("\nResponse status: HTTP/${connection.responseCode}\n");

Note: Possible expected responses status are:

  • Response status: HTTP/200 (OK)
  • Response status: HTTP/403 (*Forbidden*) is due to the CJOC Security Option is enabled and it is also expected.

That gives hints as to what the JVM proxy config is doing

In case you don’t have access to the Groovy console, then need to perform the curl command as explained on the sub-sections below:

CJE is not a TLS end-point

From the CJE instance perform curl -I -v http://<CJOC_INSTANCE>/

  • Check if the JNLP port is exposed

CJE is a TLS end-point

From the CJE instance perform curl -I -v --insecure https://<CJOC_INSTANCE>/

[CJE] Check if CJOC instance JNLP port is open for CJE

From the CJE instance perform telnet <CJOC_IP_ADDRESS> <CJOC_JNLP_PORT>

[CJE] Check if CJOC can answer to CJE instance through jenkins-cli

In the CJE instance download http/(s)://<CJOC_URL>/jnlpJars/jenkins-cli.jar. Then, on the client master execute the following command - depending if CJOC is or not a TLS end-point - to see if it is correctly executed.
As an example, for Linux OS: wget http/(s)://<CJOC_URL>/jnlpJars/jenkins-cli.jar

[CJE] CJOC is not a TLS end-point

From client master execute:

java -jar jenkins-cli.jar -s http://<CJOC_URL>/ --username=<USERNAME> --password=<PASSWORD> help

[CJE] CJOC is a TLS end-point

From client master execute:

java -Djavax.net.ssl.trustStore=<PATH_TO_CACERTS> -jar jenkins-cli.jar -s http://<CJOC_URL>/ --username=<USERNAME> --password=<PASSWORD> help

In case you are running behind a F5 or an ha-proxy

Ensure that:

  1. The F5 is using raw tcp mode for the CLI port
  2. The F5 is not doing any funky keep-alive stuff on the F5
  3. the F5 is not doing any round robin on the CLI port

[CJOC] In case your CJOC instance is a TLS end-point

Notice that this is not needed in case you are running oc-server 1.7.17+ / 1.8.5+ and oc-client 1.7.6+ / 1.8.5+

In those cases, you ensure that the Jenkins Location for CJOC is configured as an URL using https and then add the certs to the global security settings on CJOC. Once you do that, CJOC will generate connection details with the cert embedded and the clients will connect using that cert

For the rest of the cases:

If the CJOC instance is deployed on a TLS end-point, you must import the SSL certificate in the Java Keystore of the Client Master. In case the Client Master is deployed on a Tomcat web container, you might need to tell what keystore Jenkins is using. This should verify that Tomcat is using the correct keystore.
If it is not in the standard location ( $JAVA_HOME/jre/lib/security/cacerts), add it as part of the Java arguments:

-Djavax.net.ssl.keyStore=$TOMCAT_LOCATION/cacert
-Djavax.net.ssl.keyStorePassword=password

Check the following KB article for mor information:

The following are possible troubleshooting checks:

Ensure that the certificates are correctly imported in both CJOC and client master.

keytool -keystore /$JRE_HOME/lib/security/cacerts -v -list
  • The cacert file should have user read permission. This user read permission is the OS’s file system permission and the user is the OS user that Jenkins’s JVM is running.

What should I attach in a CloudBees support ticket in case client master cannot be tied to a CJOC instance?

  • Support bundles from both CJOC and CJE
  • Output of [CJE] Check if CJOC instance is reachable through HTTP/(S)
  • Output of [CJE] Check if CJOC instance JNLP port is open for CJE
  • Output of [CJE] Check if CJOC can answer to CJE instance
  • If CJOC is a TLS end-point output of keytool -keystore /$JRE_HOME/lib/security/cacerts -v -list from both CJOC and client master
  • Quick diagram of your environment so CloudBees can understand your achitecture
Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.