Customer Environment Validation Tests for a CJOC environment

Issue

  • Customer Environment Validation Tests for a CJOC environment

Environment

  • CloudBees Jenkins Operations Center (CJOC)
  • CloudBees Jenkins Enterprise (CJE)

Resolution

This document provides a mechanism to validate customers’s environments before running CloudBees Jenkins Operations Center. The KB proposes a number of tests that should be carried out to assert the underlying infrastructure works as intended.

Introduction: Connection flow between client masters and the OC instance

A simple connection flow looks like the image below:

  1. Client master sends HEAD request to CJOC root URL
  2. Client master parses HEAD response headers looking for X-Jenkins and X-Jenkins-CLI-Port.
  3. Client master initiates a TCP connection to the port specified in X-Jenkins-CLI-Port against the host specified in X-Jenkins-CLI-Host (or the host from the CJOC root URL if the X-Jenkins-CLI-Host header is absent)

If from CJE instance under http://<CJE_URL>/script we launch the script below we can see those headers exposed by CJOC.

def url = new URL("http://<CJOC_URL>/");
def connection = url.openConnection();
println("Response Headers");
println("================");
for (def e in connection.getHeaderFields()) {
  println("${e.key}: ${e.value}");
}
println("\nResponse status: HTTP/${connection.responseCode}\n");

An output like the one below is expected:

Response Headers
================
null: [HTTP/1.1 200 OK]
X-Jenkins: [1.625.16.2 (CloudBees Jenkins Operations Center 1.8)]
X-Hudson: [1.395]
Content-Length: [16451]
Expires: [0]
Set-Cookie: [JSESSIONID.d35196d9=x9jsjuq6ugw3x5t38e6ecfjt;Path=/;HttpOnly]
X-Jenkins-CLI-Port: [6587]
X-Hudson-CLI-Port: [6587]
X-SSH-Endpoint: [127.0.0.1:49691]
Server: [Jetty(winstone-2.8)]
X-Content-Type-Options: [nosniff]
Cache-Control: [no-cache,no-store,must-revalidate]
X-Frame-Options: [sameorigin]
X-Hudson-Theme: [default]
X-Instance-Identity: [MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEApZe+Wd2zoAiWalHu/wgZaFzJS56XYxv1XejB/ml8k4tb+AutXvUhDvxlRYvmkp7CV/NvF3e2YfgH5WLnt5IXXxn5a3FsDQ3xDtN/+kv4+jwTqbZnOORtJB0xET7Pak7fpjrTkUpsR8lGhHdHVf2iQxHRJoeHFblYajNHPx9U7zkrXgLA4xUqqwCPmUys/QXAcVEl6IL/1jmxtdLFQb38NNPr6CDfhrH7DNkwvTX6EDmn1Zljw+AsFBmDbfm/uoPqUCNrrQk/OvIrKebABPdJL0YoIY7naehd8ezNiICrf92TkZNopCrTUNuq6kpe8rhHW0oQf5o0KSrSi869nXG8FwIDAQAB]
X-Jenkins-Session: [5e3b53cb]
Content-Type: [text/html;charset=UTF-8]
X-Jenkins-CLI2-Port: [6587]

Response status: HTTP/200

In case you get Response status: HTTP/403 (*Forbidden*) is due to the CJOC Security Option is enabled and it is also expected.

After the explanation above, we should be able to understand how the flow looks like when pushing the connection details from CJOC to CJE

  1. Browser POST to JOC
  2. JOC redirect via browser to CJE
  3. CJE HTTP HEAD to JOC
  4. CJE extracts X-Jenkins-CLI-Port
  5. CJE establish remoting protocol to CJOC through TCP on JNLP port exposed by CJOC

Summary:

  1. Communication between CJOC and CJE happens at HTTP level - on the in which CJOC is accessible through the browser - and at TCP level through the JNLP port exposed by CJOC
  2. Communication is ALWAYS initiated by CJE trying to connect with CJOC using HTTP and TCP protocols

Customer CJOC Configuration Validation

On this section, we need to ensure that we have set-up everything correctly at Jenkins level. The image below shows a typical customer environment. A similar environment (plus a firewall configuration) can be found on this tutorial for demo purposes.

In both CJOC and CJE level we need to ensure that:

  1. The Jenkins URL was set-up under Manage Jenkins -> Configure System
  2. Jenkins is exposing a JNLP under Manage Jenkins -> Configure Global Security. The best practice is to set-up a fixed port in both CJOC and CJE instances, but this port should be different in CJOC and CJE
    Note: Use a different JNLP port for each of the CJE instances connected to the CJOC.


[Support] Customer CJOC Configuration Validation

If you are going to open a support ticket because you cannot figure out why CJE cannot connect to CJOC, then ensure that you provide us a table similar to the one below so we can understand how your environment looks like.

Node Type IP Address DNS Name HTTP(S) PORT JNLP Port $JENKINS_HOME
CJOCa CJOC primary joc-a.acme.com
CJOCb CJOC backup joc-b.acme.com
CJOC CJOC VIP jenkins.acme.com/operations-center
Master1a Master1 primary je1-a.acme.com
Master1b Master1 backup je1-b.acme.com
Master1 Master1 VIP jenkins.acme.com/je1

Note: CJOC/CJE backup are available for High Availability (HA) set-up environments

Customer CJOC Networking Validation

On this section we will continue with the same typical customer’s architecture we used on the section before:

At networking level CJE needs to be able to access CJOC through HTTP - on the port in which CJOC is accessible- and TCP on the JNLP port exposed by CJOC.

CJOC connectivity tests

Please, provide Test ID and Pass/Fail results for the following table:

Test ID Details Notes
cjoc-desktop-http From desktop: browser access to HTTP(s) port on <CJOC VIP>.
Expected: Jenkins home page to display 		As shown on the above figure: jenkins.acme.com/operations-center
cjoc-desktop-http-primary-instance From desktop: browser access HTTP(s) port on <CJOC primary>.
Expected: Jenkins home page to display 		As shown on the above figure: joc-a.acme.com
cjoc-desktop-http-backup-instance From desktop: browser access HTTP(s) port on <CJOC backup>.
Expected: “This node is standing by in case the primary node fails over” message 		As shown on the above figure: joc-b.acme.com
cjoc-dns-resolution From desktop: ping <CJOC_VIP>, ping <joc-a>, ping <joc-b> - check resolved IPs as per table above
cjoc-http-tcp-vip From desktop:
a) curl -I -v <CJOC_VIP>:<HTTP_PORT>
Obtain value from X-Jenkins-CLI-Port
b) telnet <CJOC_VIP> <CLI PORT>
Expected:
Trying ::1... Connected to localhost Escape character is ...
cjoc-http-tcp-primary-instance Shutdown CJOC-b and log on CJOC-a machine, and then:
a) curl -I -v <CJOC_VIP>:<HTTP_PORT>
Obtain value from X-Jenkins-CLI-Port
b) telnet <CJOC_VIP> <CLI PORT>
Expected:
Trying ::1... Connected to localhost Escape character is ...
cjoc-http-tcp-backup-instance Shutdown CJOC-a and log on CJOC-b machine, and then:
a) curl -I -v <CJOC_VIP>:<HTTP_PORT>
Obtain value from X-Jenkins-CLI-Port
b) telnet <CJOC_VIP> <CLI PORT>
Expected:
Trying ::1... Connected to localhost Escape character is ...

Master connectivity tests

Please, provide Test ID and Pass/Fail results for the following table:

Test ID Details Notes
cje-desktop-http From desktop: browser access to HTTP(s) port on <CJE VIP>.
Expected: Jenkins home page to display 		As shown on the above figure: jenkins.acme.com/je1
cje-desktop-http-primary-instance From desktop: browser access HTTP(s) port on <Master1 primary>.
Expected: Jenkins home page to display 		As shown on the above figure: je1-a.acme.com
cje-desktop-http-backup-instance From desktop: browser access HTTP(s) port on <Master1 backup>.
Expected: “This node is standing by in case the primary node fails over” message 		As shown on the above figure: je1-b.acme.com
cje-http-tcp-primary-instance Shutdown CJEa and log on CJEb machine, and then:
a) curl -I -v <CJOC_VIP>:<HTTP_PORT>
Obtain value from X-Jenkins-CLI-Port
b) telnet <CJOC_VIP> <CLI PORT>
Expected:
Trying ::1... Connected to localhost Escape character is ...
cje-http-tcp-backup-instance Shutdown CJEb and log on CJEa machine, and then:
a) curl -I -v <CJOC_VIP>:<HTTP_PORT>
Obtain value from X-Jenkins-CLI-Port
b) telnet <CJOC_VIP> <CLI PORT>
Expected:
Trying ::1... Connected to localhost Escape character is ...

[Support] Customer CJOC Networking Validation

In case you plan to open a support ticket, please ensure that all these tests are done before opening a ticket and provide the output on a .txt so we can check that the outputs are the expected.

JENKINS_HOME tests

Ssh to each server and cat the $JENKINS_HOME/config.xml.

CJE validation for non-fresh instances

Typical scenarios are:

  1. CJE instance that was never managed by any CJOC and now moves to a CJE-CJOC architecture.
  2. CJE instance that was previous managed by one CJOC and now migrates to another CJOC instance (CJE Migration between CJOC instances)

CJE Admin Login validation (Optional)

Just in the case that a user with Administer permission is not able to login into the CJE instance to connect to a CJOC, CJE Security needs to be disable by following this instructions.
For instance, when CJE Jenkins Dashboard ($CJE_URL) redirects to registration page ($CJE_URL/registration).

Disable previous CJOC connection

Before connecting a CJE to a CJOC, the former CJOC Connector in the CJE configurations needs to be disabled.

Go to $CJE_URL/configure > Operations Center Connector > Disable

Other relevant documents:

Was this article helpful?
1 out of 1 found this helpful
Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.