Issue

A specific job or your Jenkins environment hangs or responds slowly.

Required Data: Hang Issue On Linux Jenkins

This article describes how to collect the minimum required information for performance issues on a Jenkins environment so that it can be efficiently troubleshooted.

If the required data is bigger than 20 MB you will not be able to use ZenDesk to upload all the information. On this case we would like to encourage you to use our upload service in order to attach all the required information.

Environment

• Operating System: Linux
• CloudBees Core on modern cloud platforms - Managed Master
• CloudBees Core on modern cloud platforms - Operations Center
• CloudBees Core on traditional platforms - Client Master
• CloudBees Core on traditional platforms - Operations Center
• CloudBees Jenkins Enterprise - Managed Master
• CloudBees Jenkins Enterprise - Operations Center
• CloudBees Jenkins Platform - Client Master
• CloudBees Jenkins Platform - Operations Center
• CloudBees Jenkins Distribution
• Jenkins LTS

Automatic Data collection

This is the preferred method if you are using a product supported by the cbsupport CLI.
Current products supporting collecting performance data are:

• CloudBees Core on modern cloud platforms - Managed Master
• CloudBees Core on modern cloud platforms - Operations Center

Steps to follow are:

1. Install and configure cbsupport following Using cbsupport CLI to collect the requested data
2. Run cbsupport required-data performance
3. Collect the archive generated in the working directory of cbsupport and attach it to the ticket using our upload service

Manual Data collection - Required Data check list

Ideally, your environment is configured as documented in Prepare Jenkins For Support.

• Support bundle
• Output of the script jenkinshangWithJstack.sh
• GC log file for review
• Optional (if an I/O issue is suspected): output of the script listDProcessesNativeStacks.sh

Support bundle

A support bundle from the problematic instance ideally when the issue is happening or in the worst case right after the issue is exposed. Please, follow the KB below in case you don’t know how to generate a support bundle.

Please, enable at minimum the following checks: System properties, Master Log Recorders, Garbage Collection Logs, Slow Request Records, Out Of Memory Errors, Master Heap Histogram, Deadlock Records, Thread dumps of running Pipeline builds and Thread dumps.

• What is a support-bundle and how to create one

Output of the script jenkinshangWithJstack.sh

Pre-requisites

1. - Check you have the required tools to run the script included in the $PATH

See CloudBees Jenkins Platform Supported Java Versions > Recommended installations

Note: jstack must be recognized as command in your Jenkins Host.

The jenkinshangWithJstack.sh script collects thread dumps using the jstack command. It also needs to be able to run top, vmstat, netstat, nfsiostat, nfsstat and iostat. Please make sure that the user Jenkins is running as can execute all of these commands.

2. - Get the script

Download the jenkinshangWithJstack.sh.

3. - Make executable the script

The jenkinshangWithJstack.sh script will need to be made executable by running: chmod +x jenkinshangWithJstack.sh.

4. - Get the $JENKINS_USER and $JENKINS_PID

If CloudBees Jenkins Enterprise or CloudBees Jenkins Operations Center is running on Ubuntu you will need to confirm the PID via the command line by running ps -ef | grep jenkins. This is required because the CloudBees Jenkins PID File contains the PID of the daemon process itself not the CloudBees Jenkins PID.*

5. - Check that <JENKINS_USER> has permissions in the same folder where jenkinshangWithJstack.sh is running

The jenkinshangWithJstack.sh script will generate data in the current directory where it is run from.

6. - If you are experiencing a hang or a poor performance issue please execute the script while the issue is occurring:

Run the jenkinshangWithJstack.sh script as the same user that starts Jenkins by running:

sudo -u $JENKINS_USER sh jenkinshangWithJstack.sh $JENKINS_PID

Example

From ps aux | grep jenkins

avahi 1115 0.0 0.0 32344 1328 ? Ss 2017 0:47 avahi-daemon: running [jenkins.local] 
exampleUser 3045 236 16.2 10286700 2645660 ? Ssl Jan02 6592:33 /etc/alternatives/java -Dhudson.slaves.WorkspaceList=- -Djava.awt .headless=true -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/ var/cache/jenkins/war --httpPort=8086 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20 
root 10366 0.0 0.0 112652 964 pts/0 S+ 03:01 0:00 grep --color=auto jenkins

Then

sudo -u exampleUser sh jenkinshangWithJstack.sh 3045

You can run the script without any arguments to validate if it will work as expected.

sudo -u exampleUser sh jenkinshangWithJstack.sh

Note: If the jenkinshangWithJstack.sh script does not run as expected please upload the output the script generates from the terminal
window for review.

GC log file for review.

If you followed Prepare Jenkins For Support then the gc logs file should be under -Xloggc:$path/gc-%t.log Note that the slash should be adjusted for your operating system, i.e. Windows uses \

Output of the script listDProcessesNativeStacks.sh

See the What is listDProcessesNativeStacks.sh and how does it help? KB.