Required Data: Jenkins: High CPU On Linux

Issue

A specific job or your Jenkins environment is experiencing high CPU usage.

Required Data: High CPU On Linux

This article describes how to collect the minimum required information for high CPU issues on a Jenkins environment to aid in troubleshooting.

If the required data is bigger than 20 MB you will not be able to use ZenDesk to upload all the information. In this case, please use our upload service in order to attach all the required information. The upload service works best in Chrome or Firefox.

Environment

Required Data check list

Ideally, your environment is configured as documented in Prepare Jenkins For Support.

  • Support bundle
  • Output of the script jenkinshangWithJstack.sh
  • GC log file for review

Support Bundle

Please capture a support bundle from the problem instance, ideally when the issue is happening, or else right after the issue is exposed. This knowledge base article describes how to create a support bundle.

Please enable at minimum the following checks: System properties, Master Log Recorders, Garbage Collection Logs, Slow Request Records, Out Of Memory Errors, Master Heap Histogram, Deadlock Records, Thread dumps of running Pipeline builds and Thread dumps.

Output of the script jenkinshangWithJstack.sh

Pre-requisites

1. Check you have the required tools to run the script included in the $PATH

See CloudBees Jenkins Platform Supported Java Versions and ensure that you are using a supported Java environment.

The jenkinshangWithJstack.sh script collects thread dumps using the jstack command. It also needs to be able to run top, vmstat, netstat, nfsiostat, nfsstat and iostat. Please make sure that the user Jenkins is running as can execute all of these commands.

2. Download jenkinshangWithJstack.sh.

3. Make the script executable by running: chmod +x jenkinshangWithJstack.sh.

4. Determine the $JENKINS_USER and $JENKINS_PID

Run ps -ef | grep jenkins. You will see output similar to:

jenkins 12345 17347 0 Mar17 ? 00:00:17 /usr/bin/java -jar jenkins.war

The first two columns show the user and process ID. In this case, $JENKINS_USER is jenkins and $JENKINS_PID is 12345. It is best to use ps to determine these values, rather than looking at the Jenkins service PID file, because on some systems the PID file contains the process ID of the daemon/service process, rather than Jenkins itself.

5. Check that the Jenkins user can write to the same folder where jenkinshangWithJstack.sh is running. The script will generate data in that directory.

6. If you are experiencing a high CPU issue please execute the script while the issue is occurring. Run the jenkinshangWithJstack.sh script as the same user that starts Jenkins by running:

sudo -u $JENKINS_USER sh jenkinshangWithJstack.sh $JENKINS_PID 300 5

Example

From ps aux | grep jenkins

avahi 1115 0.0 0.0 32344 1328 ? Ss 2017 0:47 avahi-daemon: running [jenkins.local] 
exampleUser 3045 236 16.2 10286700 2645660 ? Ssl Jan02 6592:33 /etc/alternatives/java -Dhudson.slaves.WorkspaceList=- -Djava.awt .headless=true -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/ var/cache/jenkins/war --httpPort=8086 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20 
root 10366 0.0 0.0 112652 964 pts/0 S+ 03:01 0:00 grep --color=auto jenkins

Then

sudo -u exampleUser sh jenkinshangWithJstack.sh 3045 300 5

You can run the script without any arguments to validate if it will work as expected.

sudo -u exampleUser sh jenkinshangWithJstack.sh

If the jenkinshangWithJstack.sh script does not run as expected please upload the output the script generates from the terminal window for review.

GC log file for review

If you followed Prepare Jenkins For Support then the gc logs file should be under -Xloggc:$path/gc-%t.log

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.