Jenkins instance presents slowness

Symptoms

Jenkins instance is slow at some point. For example, pages that are usually fast to load suddenly take several seconds.
The UI is always displayed although not usable at some point.

Diagnosis/Treatment

Precondition Ensure before taking any further action that Prepare Jenkins for support is done on customer side

  • When the problem is exposed we need to use this script in order to take several thread dumps when the problem is exposed and correlate them with the output of a process list.
  • Only a thread dump is NOT ENOUGH because you are not able to know what is the percentage of CPU used by each thread in the thread dump - you need also the process list with CPU usage listed.
  • Only one thread dump is NOT ENOUGH as it only represent a punctual static photograph. We need several sequential thread dumps to understand behavior over time.
  • Attach at least one support bundle when the issue is exposed or right after.
  • Output of the garbage collector when the issue is exposed. Remember that you are specifying this file in Prepare Jenkins for support -Xloggc:<some-log-file>

JVM Tuning has been known to improve performance, especially with a large java heap size. For this we recommend:

note: Most users need 4G or less of heap memory for Jenkins. It is best to ensure you really need additional heap (either from low garbage collection throughput and excessive garbage collection activity, measured by GC logs, or from getting OutOfMemory errors in Jenkins).

The way to run the script is as follow:

# Syntax
sudo -u jenkins-oc sh /home/jenkins/jenkinshangWithJstack.sh <PID> <TOTAL_TIME> <SLOTS>
 
# Example
sudo -u jenkins-oc sh /home/jenkins/jenkinshangWithJstack.sh <PID> 300 30

Notice that the script needs to be run either by root or the user running the jenkins process ps -auxgrep jenkins (sudo -u will not work)
We recommend to firstly run the script when the issue is not exposed and check that all the elements generated has content.

This will provide this file structure:

generated files sampledescription

images/TG-jenkinsinstancepresentsslowness-01.pngContent:
* iostat Monitoring the performance of storage devices
* jstack thread Dump
* mode.txt Script output
* netstat.out Network connections for the Transmission Control Protocol (both incoming and outgoing), routing tables, and a number of network interfaces.
* topdashHOutput Sub-process list with multiple information like CPU usage, memory consumption, …
* topOutput Process running on the machine
* vmstat Display summary information about operating system memory, processes, interrupts, paging and block I/O

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.