Issue
- Jenkins hangs (may be anytime, including startup)
- Jenkins is not responsive or coming up, the process is running and Jenkins logs are stale
Required Data: Hang Issue On Linux Jenkins
This article describes how to collect the minimum required information for deadlock issues on a Jenkins environment so that it can be efficiently troubleshooted.
If the required data is bigger than 50 MB you will not be able to use ZenDesk to upload all the information. On this case we would like to encourage you to use our upload service in order to attach all the required information.
Environment
- Operating System: Linux
- CloudBees CI (CloudBees Core) on modern cloud platforms - Managed Master
- CloudBees CI (CloudBees Core) on modern cloud platforms - Operations Center
- CloudBees CI (CloudBees Core) on traditional platforms - Client Master
- CloudBees CI (CloudBees Core) on traditional platforms - Operations Center
- CloudBees Jenkins Enterprise - Managed Master
- CloudBees Jenkins Enterprise - Operations Center
- CloudBees Jenkins Platform - Client Master
- CloudBees Jenkins Platform - Operations Center
- CloudBees Jenkins Distribution
- Jenkins LTS
Automatic Data collection
This is the preferred method if you are using a product supported by the cbsupport
CLI.
Current products supporting collecting performance data are:
- CloudBees CI (CloudBees Core) on modern cloud platforms - Managed Master
- CloudBees CI (CloudBees Core) on modern cloud platforms - Operations Center
Steps to follow are:
- Install and configure
cbsupport
following Using cbsupport CLI to collect the requested data - Run
cbsupport required-data performance
- Collect the archive generated in the working directory of cbsupport and attach it to the ticket using our upload service
Manual Data collection - Required Data check list
Ideally, your environment is configured as documented in [Prepare Jenkins For Support][].
- Output of the script collectPerformanceData.sh
- Support bundle
- List of File Descriptors
- Optional (if an I/O issue is suspected): output of the script listDProcessesNativeStacks.sh
Output of the script collectPerformanceData.sh
Thread dumps are the best way to pinpoint the root cause of this issue.
See Required Data: Jenkins Hang Issue On Linux KB.
Support bundle
A support bundle from the problematic instance ideally when the issue is happening or in the worst case right after the issue is exposed. Please, follow the KB below in case you don’t know how to generate a support bundle.
Please, enable at minimum the following checks: System properties
, Master Log Recorders
, Garbage Collection Logs
, Slow Request Records
, Out Of Memory Errors
, Master Heap Histogram
, Deadlock Records
, Thread dumps of running Pipeline builds
and Thread dumps
.
- What is a support-bundle and how to create one
- How to get a support bundle when Jenkins will not start? (if Jenkins is stuck while starting)
List of File Descriptors
The list of file descriptor hold by the Jenkins process.
Find the PID of the Jenkins process. For example the following output of ps aux | grep jenkins
show that Jenkins PID is 2487
:
[ec2-user@ip-10-240-0-112 ~]$ ps -ef | grep jenkins
ec2-user 2487 2435 1 May07 ? 00:26:30 java -Duser.home=/var/jenkins_home -Xmx2150m -Xms2150m -jar /usr/share/jenkins/jenkins.war --webroot=/tmp/jenkins/war --pluginroot=/tmp/jenkins/plugins
ec2-user 14540 13480 0 03:20 pts/0 00:00:00 grep --color=auto jenkins
Collect the list of file descriptor of that process with lsof
(or sudo lsof
if sudo privileges are needed):
lsof -a -p $JENKINS_PID > /tmp/file-descriptors.txt
If lsof
is not available on the host, you may use the following command instead:
ls -l /proc/$JENKINS_PID/fd/ > /tmp/file-descriptors.txt
Collect the file produced /tmp/file-descriptors.txt
Output of the script listDProcessesNativeStacks.sh
See the What is listDProcessesNativeStacks.sh and how does it help? KB.
0 Comments