Issue

• Jenkins hangs (may be anytime, including startup)
• Jenkins is not responsive or coming up, the process is running and Jenkins logs are stale

Required Data: Hang Issue On Linux Jenkins

This article describes how to collect the minimum required information for deadlock issues on a Jenkins environment so that it can be efficiently troubleshooted.

If the required data is bigger than 50 MB you will not be able to use ZenDesk to upload all the information. On this case we would like to encourage you to use our upload service in order to attach all the required information.

Environment

• Operating System: Linux
• CloudBees CI (CloudBees Core) on modern cloud platforms - Managed Master
• CloudBees CI (CloudBees Core) on modern cloud platforms - Operations Center
• CloudBees CI (CloudBees Core) on traditional platforms - Client Master
• CloudBees CI (CloudBees Core) on traditional platforms - Operations Center
• CloudBees Jenkins Enterprise - Managed Master
• CloudBees Jenkins Enterprise - Operations Center
• CloudBees Jenkins Platform - Client Master
• CloudBees Jenkins Platform - Operations Center
• CloudBees Jenkins Distribution
• Jenkins LTS

Automatic Data collection

This is the preferred method if you are using a product supported by the cbsupport CLI.
Current products supporting collecting performance data are:

• CloudBees CI (CloudBees Core) on modern cloud platforms - Managed Master
• CloudBees CI (CloudBees Core) on modern cloud platforms - Operations Center

Steps to follow are:

1. Install and configure cbsupport following Using cbsupport CLI to collect the requested data
2. Run cbsupport required-data performance
3. Collect the archive generated in the working directory of cbsupport and attach it to the ticket using our upload service

Manual Data collection - Required Data check list

Ideally, your environment is configured as documented in [Prepare Jenkins For Support][].

• Output of the script collectPerformanceData.sh
• Support bundle
• List of File Descriptors
• Optional (if an I/O issue is suspected): output of the script listDProcessesNativeStacks.sh

Output of the script collectPerformanceData.sh

Thread dumps are the best way to pinpoint the root cause of this issue.

See Required Data: Jenkins Hang Issue On Linux KB.

Support bundle

A support bundle from the problematic instance ideally when the issue is happening or in the worst case right after the issue is exposed. Please, follow the KB below in case you don’t know how to generate a support bundle.

Please, enable at minimum the following checks: System properties, Master Log Recorders, Garbage Collection Logs, Slow Request Records, Out Of Memory Errors, Master Heap Histogram, Deadlock Records, Thread dumps of running Pipeline builds and Thread dumps.

• What is a support-bundle and how to create one
• How to get a support bundle when Jenkins will not start? (if Jenkins is stuck while starting)

List of File Descriptors

The list of file descriptor hold by the Jenkins process.

Find the PID of the Jenkins process. For example the following output of ps aux | grep jenkins show that Jenkins PID is 2487:

[ec2-user@ip-10-240-0-112 ~]$ ps -ef | grep jenkins
ec2-user  2487  2435  1 May07 ?        00:26:30 java -Duser.home=/var/jenkins_home -Xmx2150m -Xms2150m -jar /usr/share/jenkins/jenkins.war --webroot=/tmp/jenkins/war --pluginroot=/tmp/jenkins/plugins
ec2-user 14540 13480  0 03:20 pts/0    00:00:00 grep --color=auto jenkins

Collect the list of file descriptor of that process with lsof (or sudo lsof if sudo privileges are needed):

lsof -a -p $JENKINS_PID > /tmp/file-descriptors.txt

If lsof is not available on the host, you may use the following command instead:

ls -l /proc/$JENKINS_PID/fd/ > /tmp/file-descriptors.txt

Collect the file produced /tmp/file-descriptors.txt

Output of the script listDProcessesNativeStacks.sh

See the What is listDProcessesNativeStacks.sh and how does it help? KB.