Jenkins instance presents slowness

Symptoms

Jenkins instance is slow at some point. For example, pages that are usually fast to load suddenly take several seconds.
The UI is always displayed although not usable at some point.

Diagnosis/Treatment

Precondition Before taking any further action ensure that the recommended JVM parameters are in place per the guide Prepare Jenkins for support.

  • When the problem is exposed we need to use the script jenkinshangWithJstack.sh in order to take several thread dumps when the problem is exposed and correlate them with the output of a process list.
    • Only a thread dump is NOT ENOUGH because you are not able to know what is the percentage of CPU used by each thread in the thread dump - you need also the process list with CPU usage listed.
    • Only one thread dump is NOT ENOUGH as it only represent a punctual static photograph. We need several sequential thread dumps to understand behavior over time.
  • Attach at least one support bundle when the issue is exposed or right after.
  • Output of the garbage collector when the issue is exposed. Remember that you are specifying this file in Prepare Jenkins for support -Xloggc:<some-log-file>

JVM Tuning has been known to improve performance, especially with a large java heap size. For this we recommend:

note: Most users need 4G or less of heap memory for Jenkins. It is best to ensure you really need additional heap (either from low garbage collection throughput and excessive garbage collection activity, measured by GC logs, or from getting OutOfMemory errors in Jenkins).

The script is run as follows:

  # Syntax
  jenkinshangWithJstack.sh <PID> <TOTAL_TIME> <SLOTS>

  # Example
  sudo -i -u jenkins-oc /home/jenkins/jenkinshangWithJstack.sh <PID> 300 30

Notice that the script needs to be run by the user running the jenkins process ps -aux | grep jenkins.
We recommend running the script prior to troubleshooting any issue to confirm that all the elements generated have content.

Generated bundle’s directory structure:

images/TG-jenkinsinstancepresentsslowness-01.png

Contents:
* iostat Monitoring the performance of storage devices
* jstack thread Dump
* mode.txt Script output
* netstat.out Network connections for the Transmission Control Protocol (both incoming and outgoing), routing tables, and a number of network interfaces.
* nfsiostat Monitoring the performance of NFS mounts
* nfsstat NFS statistics for client and server activity
* topdashHOutput Sub-process list with multiple information like CPU usage, memory consumption, …
* topOutput Process running on the machine
* vmstat Display summary information about operating system memory, processes, interrupts, paging and block I/O

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.