Issue

• Customer Environment Validation Tests for a CJOC environment

Environment

• CloudBees Jenkins Operations Center (CJOC)
• CloudBees Jenkins Enterprise (CJE)

Resolution

This document provides a mechanism to validate customers’s environments before running CloudBees Jenkins Operations Center. The KB proposes a number of tests that should be carried out to assert the underlying infrastructure works as intended.

Introduction: Connection flow between client masters and the OC instance

A simple connection flow looks like the image below:

1. Client master sends HEAD request to CJOC root URL
2. Client master parses HEAD response headers looking for X-Jenkins and X-Jenkins-CLI-Port.
3. Client master initiates a TCP connection to the port specified in X-Jenkins-CLI-Port against the host specified in X-Jenkins-CLI-Host (or the host from the CJOC root URL if the X-Jenkins-CLI-Host header is absent)

If from CJE instance under http://<CJE_URL>/script we launch the script below we can see those headers exposed by CJOC.

def url = new URL("http://<CJOC_URL>/");
def connection = url.openConnection();
println("Response Headers");
println("================");
for (def e in connection.getHeaderFields()) {
  println("${e.key}: ${e.value}");
}
println("\nResponse status: HTTP/${connection.responseCode}\n");

An output like the one below is expected:

Response Headers
================
null: [HTTP/1.1 200 OK]
X-Jenkins: [1.625.16.2 (CloudBees Jenkins Operations Center 1.8)]
X-Hudson: [1.395]
Content-Length: [16451]
Expires: [0]
Set-Cookie: [JSESSIONID.d35196d9=x9jsjuq6ugw3x5t38e6ecfjt;Path=/;HttpOnly]
X-Jenkins-CLI-Port: [6587]
X-Hudson-CLI-Port: [6587]
X-SSH-Endpoint: [127.0.0.1:49691]
Server: [Jetty(winstone-2.8)]
X-Content-Type-Options: [nosniff]
Cache-Control: [no-cache,no-store,must-revalidate]
X-Frame-Options: [sameorigin]
X-Hudson-Theme: [default]
X-Instance-Identity: [MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEApZe+Wd2zoAiWalHu/wgZaFzJS56XYxv1XejB/ml8k4tb+AutXvUhDvxlRYvmkp7CV/NvF3e2YfgH5WLnt5IXXxn5a3FsDQ3xDtN/+kv4+jwTqbZnOORtJB0xET7Pak7fpjrTkUpsR8lGhHdHVf2iQxHRJoeHFblYajNHPx9U7zkrXgLA4xUqqwCPmUys/QXAcVEl6IL/1jmxtdLFQb38NNPr6CDfhrH7DNkwvTX6EDmn1Zljw+AsFBmDbfm/uoPqUCNrrQk/OvIrKebABPdJL0YoIY7naehd8ezNiICrf92TkZNopCrTUNuq6kpe8rhHW0oQf5o0KSrSi869nXG8FwIDAQAB]
X-Jenkins-Session: [5e3b53cb]
Content-Type: [text/html;charset=UTF-8]
X-Jenkins-CLI2-Port: [6587]

Response status: HTTP/200

In case you get Response status: HTTP/403 (*Forbidden*) is due to the CJOC Security Option is enabled and it is also expected.

After the explanation above, we should be able to understand how the flow looks like when pushing the connection details from CJOC to CJE

1. Browser POST to JOC
2. JOC redirect via browser to CJE
3. CJE HTTP HEAD to JOC
4. CJE extracts X-Jenkins-CLI-Port
5. CJE establish remoting protocol to CJOC through TCP on JNLP port exposed by CJOC

Summary:

1. Communication between CJOC and CJE happens at HTTP level - on the in which CJOC is accessible through the browser - and at TCP level through the JNLP port exposed by CJOC
2. Communication is ALWAYS initiated by CJE trying to connect with CJOC using HTTP and TCP protocols

Customer CJOC Configuration Validation

On this section, we need to ensure that we have set-up everything correctly at Jenkins level. The image below shows a typical customer environment. A similar environment (plus a firewall configuration) can be found on this tutorial for demo purposes.

In both CJOC and CJE level we need to ensure that:

1. The Jenkins URL was set-up under Manage Jenkins -> Configure System
2. Jenkins is exposing a JNLP under Manage Jenkins -> Configure Global Security. The best practice is to set-up a fixed port in both CJOC and CJE instances, but this port should be different in CJOC and CJE
   Note: Use a different JNLP port for each of the CJE instances connected to the CJOC.

[Support] Customer CJOC Configuration Validation

If you are going to open a support ticket because you cannot figure out why CJE cannot connect to CJOC, then ensure that you provide us a table similar to the one below so we can understand how your environment looks like.

Note: CJOC/CJE backup are available for High Availability (HA) set-up environments

Customer CJOC Networking Validation

On this section we will continue with the same typical customer’s architecture we used on the section before:

At networking level CJE needs to be able to access CJOC through HTTP - on the port in which CJOC is accessible- and TCP on the JNLP port exposed by CJOC.

CJOC connectivity tests

Please, provide Test ID and Pass/Fail results for the following table:

Master connectivity tests

Please, provide Test ID and Pass/Fail results for the following table:

[Support] Customer CJOC Networking Validation

In case you plan to open a support ticket, please ensure that all these tests are done before opening a ticket and provide the output on a .txt so we can check that the outputs are the expected.

JENKINS_HOME tests

Ssh to each server and cat the $JENKINS_HOME/config.xml.

CJE validation for non-fresh instances

Typical scenarios are:

1. CJE instance that was never managed by any CJOC and now moves to a CJE-CJOC architecture.
2. CJE instance that was previous managed by one CJOC and now migrates to another CJOC instance (CJE Migration between CJOC instances)

CJE Admin Login validation (Optional)

Just in the case that a user with Administer permission is not able to login into the CJE instance to connect to a CJOC, CJE Security needs to be disable by following this instructions.
For instance, when CJE Jenkins Dashboard ($CJE_URL) redirects to registration page ($CJE_URL/registration).

Disable previous CJOC connection

Before connecting a CJE to a CJOC, the former CJOC Connector in the CJE configurations needs to be disabled.

Go to $CJE_URL/configure > Operations Center Connector > Disable

Other relevant documents:

• How to troubleshoot client master connections