Prepare Jenkins for Support

Issue

  • Best Practices to Support Jenkins

Environment

Resolution

The following settings are known Best Practices, and should be used in your Jenkins environment to allow for easier support in the event of an issue.

A. Java Parameters

CloudBees Supported Java Version

JVM Administration and Tuning is necessary for any Enterprise Java Application Deployment. As Jenkins is an Enterprise Java Application, the following arguments are considered to be best practices, based on historical data analysis of Garbage Collection Logs, Thread Dump Analysis, and Heap Dump Analysis.

In the last several years with Java 1.8.0, fine tuning Garbage Collection settings was considered a common practice, but ultimately masks underlying problems within the JDK. Because of this, it was also common practice to allocate more RAM and CPU resources to the JVM to keep it running smoothly. While this will help in the short term, it still masks underlying problems.

In today’s methodologies, it is more important to ensure you are running a current JDK version, and keep the JVM Arguments simple. There have been several years of bug fixes, memory leak fixes, threading improvements, and Garbage Collection enhancements, and therefore we recommend JDK 1.8.0_202 or newer.

Additionally, keeping your JVM Heap size below 16GB per our JVM Best Practices encourages horizontal scaling and prevents you from creating a monolithic JVM. Following this methodology will ensure you achieve a performant and stable instance of Jenkins, and will ultimately improve the experience for the end-users.

Please ensure you are using ONLY the following arguments unless directed by a Support Engineer, and if you have a question about adding an argument, please engage our Support Team.

Heap Size

Heap Size -Xmx= and -Xms= should be set to the same value, and should be determined by the current needs of the application. Per our JVM Best Practices we recommend a minimum of 2GB (4GB for Production Instances) and a maximum of 16GB (Anything larger should be scaled horizontally.) It is important, as a JVM Administrator, to plan for heap to be increased as usage increases, and scale horizontally when the recommended maximum heap size of 16GB is reached. Monitoring Garbage Collection logging is the recommended way to predict how much heap you need.

JVM Arguments

The following parameters are explained in-depth in Oracle Documentation as well as our JVM Best Practice. Please implement the following arguments ensuring ${PATH} is replaced by a valid path on your operating system in which Jenkins has permission to write:

-XX:+AlwaysPreTouch
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=${PATH}
-XX:+UseG1GC
-XX:+UseStringDeduplication
-XX:+ParallelRefProcEnabled
-XX:+DisableExplicitGC
-XX:+UnlockDiagnosticVMOptions
-XX:+UnlockExperimentalVMOptions 
-verbose:gc
-Xloggc:${PATH}/gc-%t.log  *Note that the slash should be adjusted for your operating system, i.e. Windows uses* \
-XX:NumberOfGCLogFiles=2
-XX:+UseGCLogFileRotation
-XX:GCLogFileSize=100m
-XX:+PrintGC
-XX:+PrintGCDateStamps
-XX:+PrintGCDetails
-XX:+PrintHeapAtGC 
-XX:+PrintGCCause
-XX:+PrintTenuringDistribution 
-XX:+PrintReferenceGC 
-XX:+PrintAdaptiveSizePolicy
-XX:ErrorFile=${PATH}/hs_err_%p.log  *Note that the slash should be adjusted for your operating system, i.e. Windows uses* \
-XX:+LogVMOutput    (requires -XX:+UnlockDiagnosticVMOptions)
-XX:LogFile=${PATH} (requires -XX:+UnlockDiagnosticVMOptions)             
* *Example: `-XX:LogFile=/var/log/jenkins/jvm.log` (Linux)* 
* *Example: `-XX:LogFile="C:\Program Files (x86)\Jenkins\jvm.log"` (Windows)*

If you are unsure how to add Java arguments to Jenkins, please see: How to add Java arguments to Jenkins.

Note that Jenkins will require a restart for the JVM parameters to be picked up by the JVM.

B. Ulimit Settings [just for Linux OS]

ulimit -c and ulimit -f should be set to unlimited for the user that starts Jenkins. ulimit -c set to unlimited will allow core files to be generated successfully. The core files include full thread dumps and core files generated by the JVM in the event of a JVM crash. ulimit -f should be set to unlimited to ensure that files are not truncated during their generation.

ulimit -n should be set to 4096 (soft) and 8192 (hard)

ulimit -u should be set to 30654 (soft) and 30654 (hard)

The ulimit -n and ulimit -u setting recommendations are taken from: Memory problem: ‘unable to create new native thread’

Please see this KB Article for setting ulimits in Docker, Too many open files.

To have the ulimit settings persist for the user that runs Jenkins they will need to be added to /etc/security/limits.conf.

Your /etc/security/limits.conf file should like similar to the example below in regards to the
jenkins user (or the user that starts jenkins in your environment):

limits.confFileExample.png

C. Java Home Environment Variable

It is recommended to set the JAVA_HOME environment variable in both Linux and Windows environments. The Java JDK’s bin directory should also be in the environment’s PATH environment variable. This will allow for easier access to Java JDK commands such as jstack and jmap. Please note that the jstack command is only
available in the Java JDK not the Java JRE.

Please also note that core files and heapdump files can be large and are generated in the working directory of the JVM. Thus, it is recommended that there is adequate disk space for that directory.

For example, on Windows the PATH environment variable will include: C:\Program Files\Java\jdk1.8.0_202\bin

Running jstack in terminal window or command prompt should generate the following output:

jstackInTerminal.png

D. Log Startup Timing Info

Per our Startup Troubleshooting Guide it is recommended to set -Djenkins.model.Jenkins.logStartupPerformance=true in your Jenkins environment to have Jenkins output additional
data to the jenkins.log showing how much time is being spent in a specific component during startup.

E. Jenkins Admin Should Be Able To Generate A Support Bundle via jenkins-cli.jar

This will allow Jenkins admins to generate support bundles in the event the Jenkins User Interface (UI) is not available
even though Jenkins is running.

Please note that Jenkins accounts must have the Overall/Read account permission to access the CLI.

The jenkins-cli.jar can be downloaded from JENKINS_URL/cli.

More information about the jenkins-cli.jar support command can be viewed by navigating to JENKINS_URL/cli/command/support .

Additional information about the jenkins-cli.jar can be found at jenkins-cli.jar wiki page.

Please the screen shot below showing a support bundle generated via jenkins-cli.jar using the username and password.
The jenkins-cli.jar authentication can be set up to use ssh keys if you prefer as documented in the jenkins-cli.jar wiki page
linked above.

jenkinsCLISupportBundle.png

Have more questions? Submit a request

3 Comments

  • 0
    Avatar
    Brian Hedlund

    Looks like I want to add this in for GC

    -XX:+UseG1GC

  • 0
    Avatar
    Carlos Rodriguez Lopez

    Please, Can you elaborate more your question?

  • 0
    Avatar
    Seu Support

    Please be aware that if Jenkins runs under control of systemd, then defining the ulimits as shown above within /etc/security/limits.conf has no effect on Jenkins. Parameters like LimitNoFile and LimitNoProc must be set within the systemd service unit file.

Please sign in to leave a comment.