Issue

A specific job or your Jenkins environment hangs or responds slowly.

Required Data: Hang Issue On Linux Jenkins

This article describes how to collect the minimum required information for performance issues on a Jenkins environment so that it can be efficiently troubleshooted.

If the required data is bigger than 50 MB you will not be able to use ZenDesk to upload all the information. On this case we would like to encourage you to use our upload service in order to attach all the required information.

Environment

• Operating System: Linux
• CloudBees CI (CloudBees Core) on modern cloud platforms - Managed controller
• CloudBees CI (CloudBees Core) on modern cloud platforms - Operations Center
• CloudBees CI (CloudBees Core) on traditional platforms - Client controller
• CloudBees CI (CloudBees Core) on traditional platforms - Operations Center
• CloudBees Jenkins Enterprise - Managed controller
• CloudBees Jenkins Enterprise - Operations Center
• CloudBees Jenkins Platform - Client controller
• CloudBees Jenkins Platform - Operations Center
• CloudBees Jenkins Distribution
• Jenkins LTS

Automatic Data collection

This is the preferred method if you are using a product supported by the cbsupport CLI.
Current products supporting collecting performance data are:

• CloudBees CI (CloudBees Core) on modern cloud platforms - Managed controller
• CloudBees CI (CloudBees Core) on modern cloud platforms - Operations Center

Steps to follow are:

1. Install and configure cbsupport following Using cbsupport CLI to collect the requested data
2. Run cbsupport required-data performance
3. Collect the archive generated in the working directory of cbsupport and attach it to the ticket using our upload service

Manual Data collection - Required Data check list

Ideally, your environment is configured as documented in Prepare Jenkins For Support.

• Support bundle
• Output of the script collectPerformanceData.sh
• GC log file for review
• Optional (if an I/O issue is suspected): output of the script listDProcessesNativeStacks.sh

Support bundle

A support bundle from the problematic instance ideally when the issue is happening or in the worst case right after the issue is exposed. Please, follow the KB below in case you don’t know how to generate a support bundle.

Please, enable at minimum the following checks: System properties, Controller Log Recorders, Garbage Collection Logs, Slow Request Records, Out Of Memory Errors, Controller Heap Histogram, Deadlock Records, Thread dumps of running Pipeline builds and Thread dumps.

Find more details about the Support Bundle in What is a support-bundle and how to create one.

Output of the script collectPerformanceData.sh

Depending on the type of products you have different approaches:

CloudBees CI on traditional platforms

This process is also valid for CloudBees Jenkins Platform (Older product) and Jenkins LTS.

Steps

1. - Check you have the required tools to run the script included in the $PATH

The script must run as the same user as Jenkins.
In the next steps we will consider this user to be named jenkins.
The following checks must be performed with this users.

The collectPerformanceData.sh script collects thread dumps using either the jstack, jcmd or jattach commands.

There are 2 options here:

• either there is a JDK installed on the machine, in which case you need to make sure that either jstack or jcmd are available either in your PATH or in the path pointed to by the JAVA_HOME variable.
• or the machine only has a JRE installed, in which case you will need to download jattach from here.
  Once jattach is downloaded, make sure it is executable for the user jenkins and either add it to your PATH or make the variable JATTACH_HOME to the directory containing it.

The script works optimally when the following optional commands are available top, vmstat, netstat, nfsiostat, nfsstat and iostat.

2. - Download the collectPerformanceData.sh script.

3. - Make the script executable

The collectPerformanceData.sh script will need to be made executable by running: chmod +x collectPerformanceData.sh.

4. - Get the $JENKINS_USER and $JENKINS_PID

You can retrieve the PID running ps ax | grep -i java | grep -i -e jenkins -e cloudbees | grep war | awk '{print $1}'

5. - Check that the jenkins user has permissions in the directory where collectPerformanceData.sh is running

The collectPerformanceData.sh script will generate data in this directory.
Alternatively you can make the script generate its output elsewhere by setting up the JENKINS_HANG_OUTPUT_DIR variable.
In this case, the jenkins user should have rights to write in JENKINS_HANG_OUTPUT_DIR.

6. - If you are experiencing a hang or a poor performance issue please execute the script while the issue is occurring:

Run the collectPerformanceData.sh script as the jenkins user by running:

sudo -u jenkins sh collectPerformanceData.sh $JENKINS_PID

7. - Collect the archive generated and attach it to the ticket using our upload service

Example

Running ps ax | grep -i java | grep -i -e jenkins -e cloudbees | grep war | awk '{print $1}' we get:

50946

We can now run the script, the user running Jenkins being jenkins

sudo -u jenkins sh collectPerformanceData.sh 50946

You can run the script without any arguments to validate your setup.

sudo -u jenkins sh collectPerformanceData.sh

Note: If the collectPerformanceData.sh script does not run as expected please upload the output the script generates from the terminal window for review.

CloudBees CI on modern platforms

Steps

In addition to the process described for traditional platform, take into account the following considerations CloudBees CI on Modern: Performance Issues - High CPU / Blocked Threads.

CloudBees Jenkins Enterprise (Older product)

Steps

1. Run cje run support-performance <TENANT> [DURATION] [FREQUENCY] as explained on How to use cje command line tools
2. Collect the archive generated in the support/performance directory of CJE_HOME and attach it to the ticket using our upload service

GC log file for review.

If you followed Prepare Jenkins For Support then the gc logs file should be at -Xlog:gc[...]:file=<some-log-file>:[...] for JDK 11+ or -Xloggc:<some-log-file> for JDK 8:

Output of the script listDProcessesNativeStacks.sh

See the What is listDProcessesNativeStacks.sh and how does it help? KB.