Issue
- Customer Environment Validation Tests for a CJOC environment
Environment
- CloudBees Jenkins Operations Center (CJOC)
- CloudBees Jenkins Enterprise (CJE)
Resolution
This document provides a mechanism to validate customers’s environments before running CloudBees Jenkins Operations Center. The KB proposes a number of tests that should be carried out to assert the underlying infrastructure works as intended.
Introduction: Connection flow between client masters and the OC instance
A simple connection flow looks like the image below:
- Client master sends
HEADrequest to CJOC root URL - Client master parses
HEADresponse headers looking forX-JenkinsandX-Jenkins-CLI-Port. - Client master initiates a TCP connection to the port specified in
X-Jenkins-CLI-Portagainst the host specified inX-Jenkins-CLI-Host(or the host from the CJOC root URL if theX-Jenkins-CLI-Hostheader is absent)
If from CJE instance under http://<CJE_URL>/script we launch the script below we can see those headers exposed by CJOC.
def url = new URL("http://<CJOC_URL>/");
def connection = url.openConnection();
println("Response Headers");
println("================");
for (def e in connection.getHeaderFields()) {
println("${e.key}: ${e.value}");
}
println("\nResponse status: HTTP/${connection.responseCode}\n");
An output like the one below is expected:
Response Headers
================
null: [HTTP/1.1 200 OK]
X-Jenkins: [1.625.16.2 (CloudBees Jenkins Operations Center 1.8)]
X-Hudson: [1.395]
Content-Length: [16451]
Expires: [0]
Set-Cookie: [JSESSIONID.d35196d9=x9jsjuq6ugw3x5t38e6ecfjt;Path=/;HttpOnly]
X-Jenkins-CLI-Port: [6587]
X-Hudson-CLI-Port: [6587]
X-SSH-Endpoint: [127.0.0.1:49691]
Server: [Jetty(winstone-2.8)]
X-Content-Type-Options: [nosniff]
Cache-Control: [no-cache,no-store,must-revalidate]
X-Frame-Options: [sameorigin]
X-Hudson-Theme: [default]
X-Instance-Identity: [MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEApZe+Wd2zoAiWalHu/wgZaFzJS56XYxv1XejB/ml8k4tb+AutXvUhDvxlRYvmkp7CV/NvF3e2YfgH5WLnt5IXXxn5a3FsDQ3xDtN/+kv4+jwTqbZnOORtJB0xET7Pak7fpjrTkUpsR8lGhHdHVf2iQxHRJoeHFblYajNHPx9U7zkrXgLA4xUqqwCPmUys/QXAcVEl6IL/1jmxtdLFQb38NNPr6CDfhrH7DNkwvTX6EDmn1Zljw+AsFBmDbfm/uoPqUCNrrQk/OvIrKebABPdJL0YoIY7naehd8ezNiICrf92TkZNopCrTUNuq6kpe8rhHW0oQf5o0KSrSi869nXG8FwIDAQAB]
X-Jenkins-Session: [5e3b53cb]
Content-Type: [text/html;charset=UTF-8]
X-Jenkins-CLI2-Port: [6587]
Response status: HTTP/200
In case you get Response status: HTTP/403 (*Forbidden*) is due to the CJOC Security Option is enabled and it is also expected.
After the explanation above, we should be able to understand how the flow looks like when pushing the connection details from CJOC to CJE
- Browser POST to JOC
- JOC redirect via browser to CJE
- CJE
HTTP HEADto JOC - CJE extracts
X-Jenkins-CLI-Port - CJE establish remoting protocol to CJOC through TCP on JNLP port exposed by CJOC
Summary:
- Communication between CJOC and CJE happens at HTTP level - on the in which CJOC is accessible through the browser - and at TCP level through the JNLP port exposed by CJOC
- Communication is ALWAYS initiated by CJE trying to connect with CJOC using HTTP and TCP protocols
Customer CJOC Configuration Validation
On this section, we need to ensure that we have set-up everything correctly at Jenkins level. The image below shows a typical customer environment. A similar environment (plus a firewall configuration) can be found on this tutorial for demo purposes.
In both CJOC and CJE level we need to ensure that:
- The Jenkins URL was set-up under Manage Jenkins -> Configure System
- Jenkins is exposing a JNLP under Manage Jenkins -> Configure Global Security. The best practice is to set-up a fixed port in both CJOC and CJE instances, but this port should be different in CJOC and CJE
Note: Use a different JNLP port for each of the CJE instances connected to the CJOC.
[Support] Customer CJOC Configuration Validation
If you are going to open a support ticket because you cannot figure out why CJE cannot connect to CJOC, then ensure that you provide us a table similar to the one below so we can understand how your environment looks like.
| Node | Type | IP Address | DNS Name | HTTP(S) PORT | JNLP Port | $JENKINS_HOME |
|---|---|---|---|---|---|---|
| CJOCa | CJOC primary | joc-a.acme.com | ||||
| CJOCb | CJOC backup | joc-b.acme.com | ||||
| CJOC | CJOC VIP | jenkins.acme.com/operations-center | ||||
| Master1a | Master1 primary | je1-a.acme.com | ||||
| Master1b | Master1 backup | je1-b.acme.com | ||||
| Master1 | Master1 VIP | jenkins.acme.com/je1 |
Note: CJOC/CJE backup are available for High Availability (HA) set-up environments
Customer CJOC Networking Validation
On this section we will continue with the same typical customer’s architecture we used on the section before:
At networking level CJE needs to be able to access CJOC through HTTP - on the port in which CJOC is accessible- and TCP on the JNLP port exposed by CJOC.
CJOC connectivity tests
Please, provide Test ID and Pass/Fail results for the following table:
| Test ID | Details | Notes |
|---|---|---|
| cjoc-desktop-http | From desktop: browser access to HTTP(s) port on <CJOC VIP>. Expected: Jenkins home page to display |
As shown on the above figure: jenkins.acme.com/operations-center |
| cjoc-desktop-http-primary-instance | From desktop: browser access HTTP(s) port on <CJOC primary>. Expected: Jenkins home page to display |
As shown on the above figure: joc-a.acme.com |
| cjoc-desktop-http-backup-instance | From desktop: browser access HTTP(s) port on <CJOC backup>. Expected: “This node is standing by in case the primary node fails over” message |
As shown on the above figure: joc-b.acme.com |
| cjoc-dns-resolution | From desktop: ping <CJOC_VIP>, ping <joc-a>, ping <joc-b> - check resolved IPs as per table above |
|
| cjoc-http-tcp-vip | From desktop: a) curl -I -v <CJOC_VIP>:<HTTP_PORT> Obtain value from X-Jenkins-CLI-Port b) telnet <CJOC_VIP> <CLI PORT> Expected: Trying ::1... Connected to localhost Escape character is ... |
|
| cjoc-http-tcp-primary-instance | Shutdown CJOC-b and log on CJOC-a machine, and then: a) curl -I -v <CJOC_VIP>:<HTTP_PORT> Obtain value from X-Jenkins-CLI-Port b) telnet <CJOC_VIP> <CLI PORT> Expected: Trying ::1... Connected to localhost Escape character is ... |
|
| cjoc-http-tcp-backup-instance | Shutdown CJOC-a and log on CJOC-b machine, and then: a) curl -I -v <CJOC_VIP>:<HTTP_PORT> Obtain value from X-Jenkins-CLI-Port b) telnet <CJOC_VIP> <CLI PORT> Expected: Trying ::1... Connected to localhost Escape character is ... |
Master connectivity tests
Please, provide Test ID and Pass/Fail results for the following table:
| Test ID | Details | Notes |
|---|---|---|
| cje-desktop-http | From desktop: browser access to HTTP(s) port on <CJE VIP>. Expected: Jenkins home page to display |
As shown on the above figure: jenkins.acme.com/je1 |
| cje-desktop-http-primary-instance | From desktop: browser access HTTP(s) port on <Master1 primary>. Expected: Jenkins home page to display |
As shown on the above figure: je1-a.acme.com |
| cje-desktop-http-backup-instance | From desktop: browser access HTTP(s) port on <Master1 backup>. Expected: “This node is standing by in case the primary node fails over” message |
As shown on the above figure: je1-b.acme.com |
| cje-http-tcp-primary-instance | Shutdown CJEa and log on CJEb machine, and then: a) curl -I -v <CJOC_VIP>:<HTTP_PORT> Obtain value from X-Jenkins-CLI-Port b) telnet <CJOC_VIP> <CLI PORT> Expected: Trying ::1... Connected to localhost Escape character is ... |
|
| cje-http-tcp-backup-instance | Shutdown CJEb and log on CJEa machine, and then: a) curl -I -v <CJOC_VIP>:<HTTP_PORT> Obtain value from X-Jenkins-CLI-Port b) telnet <CJOC_VIP> <CLI PORT> Expected: Trying ::1... Connected to localhost Escape character is ... |
[Support] Customer CJOC Networking Validation
In case you plan to open a support ticket, please ensure that all these tests are done before opening a ticket and provide the output on a .txt so we can check that the outputs are the expected.
JENKINS_HOME tests
Ssh to each server and cat the $JENKINS_HOME/config.xml.
CJE validation for non-fresh instances
Typical scenarios are:
- CJE instance that was never managed by any CJOC and now moves to a CJE-CJOC architecture.
- CJE instance that was previous managed by one CJOC and now migrates to another CJOC instance (CJE Migration between CJOC instances)
CJE Admin Login validation (Optional)
Just in the case that a user with Administer permission is not able to login into the CJE instance to connect to a CJOC, CJE Security needs to be disable by following this instructions.
For instance, when CJE Jenkins Dashboard ($CJE_URL) redirects to registration page ($CJE_URL/registration).
Disable previous CJOC connection
Before connecting a CJE to a CJOC, the former CJOC Connector in the CJE configurations needs to be disabled.
Go to $CJE_URL/configure > Operations Center Connector > Disable
0 Comments