Prepare Jenkins for support

Issue

Best practices to support Jenkins

Environment

  • CloudBees Jenkins

  • Jenkins

Resolution

The following settings should be used in your Jenkins environment to allow for easier support in the event of an issue.

A. Ulimit Settings [just for Linux OS]

ulimit -c and ulimit -f should be set to unlimited for the user that starts Jenkins. ulimit -c set to unlimited will
allow core files to be generated successfully. The core files include full thread dumps and core files generated by the
JVM in the event of a JVM crash. ulimit -f should be set to unlimited to ensure that files are not truncated during
their generation.

ulimit -n should be set to 4096 (soft) and 8192 (hard)

ulimit -u should be set to 30654 (soft) and 30654 (hard)

The ulimit -n and ulimit -u setting recommendations are taken from the
Memory problem: ‘unable to create new native thread’ KB Article.

Please see this KB Article for setting ulimits in Docker, Too many open files.

To have the ulimit settings persist for the user that runs Jenkins they will need to
added to /etc/security/limits.conf.

Your /etc/security/limits.conf file should like similar to the example below in regards to the
jenkins user (or the user that starts jenkins in your environment):

limits.confFileExample.png

B. Java Parameters

Section 0.

CloudbBees Supported Java Version`

Section I.

The server and AlwaysPreTouch parameter should be enabled via:

-server

-XX:+AlwaysPreTouch

Heap Size see explained in Increasing Heap Size > Best practises

Heapdump on an Out of Memory exception should be enabled using the parameter below:

-XX:+HeapDumpOnOutOfMemoryError

-XX:HeapDumpPath= You will need to specify the directory path for the heapdump file to be written to.

Verbose gc should be enabled with the following parameters:

-verbose:gc

-Xloggc:$path/gc-%t.log Note that the slash should be adjusted for your operating system, i.e. Windows uses \

-XX:NumberOfGCLogFiles=2

-XX:+UseGCLogFileRotation

-XX:GCLogFileSize=100m

-XX:+PrintGC

-XX:+PrintGCDateStamps

-XX:+PrintGCDetails

-XX:+PrintHeapAtGC

-XX:+PrintGCCause

-XX:+PrintTenuringDistribution

-XX:+PrintReferenceGC

-XX:+PrintAdaptiveSizePolicy

Section II.

You should use one of the following garbage collection policies based on your processor and Java heap size:

Option A: Concurrent Mark Sweep (CMS) should be used for 2GB - 4GB Java heaps with multi-core systems.

-XX:+UseConcMarkSweepGC

-XX:+ExplicitGCInvokesConcurrentAndUnloadsClasses

-XX:+CMSParallelRemarkEnabled

-XX:+ParallelRefProcEnabled

-XX:+CMSClassUnloadingEnabled

-XX:+ScavengeBeforeFullGC

-XX:+CMSScavengeBeforeRemark

-XX:NewSize=512m

-XX:MaxNewSize=3g

-XX:NewRatio=2

Option B: G1GC should be used for 4GB or larger Java heaps with multi-core systems.

-XX:+UseG1GC

-XX:+UseStringDeduplication

-XX:+ParallelRefProcEnabled

-XX:+ExplicitGCInvokesConcurrent

-XX:+UnlockDiagnosticVMOptions

-XX:G1SummarizeRSetStatsPeriod=1

-XX:MaxMetaspaceExpansion=64M

-XX:+UnlockExperimentalVMOptions

-XX:G1NewSizePercent=20

Please see this KB Article for adding Java arguments to Jenkins, How to add Java arguments to Jenkins .

Note that Jenkins will require a restart for the JVM parameters to be picked up by the JVM.

Please note that your Java parameters should consist of section I and one of the options of section II (depending on your Java heap
size) from above.

The -server parameter causes the JVM to run in server mode. The Server VM has been specially tuned to maximize peak
operating speed. It is intended for executing long-running server applications, which need the fastest possible operating
speed more than a fast start-up time or smaller runtime memory footprint. (as noted in Chapter 2 in The Java HotSpot Performance Engine Architecture)

The -XX:+AlwaysPreTouch parameter pre-zeroes memory mapped pages on JVM startup – improves runtime performance.

The -XX:+HeapDumpOnOutOfMemoryError parameter tells the JVM to automatically generate a heap dump file when a heap
memory allocation can’t be satisfied.

The -XX:HeapDumpPath= parameter allows you to specify the directory path where the heapdump file should be written.
Please sure that the directory path specified can be written to by the Jenkins process and has adequate space to hold
the large heapdump files.

The -verbose:gc parameter enables verbose gc.

The Xloggc:$path/gc-%t.log parameter specifies where the verbose gc data is written to. The %t in the gc log file path
will cause the JVM to generate a new file with each JVM restart.

The -XX:NumberOfGCLogFiles=2 -XX:+UseGCLogFileRotation -XX:GCLogFileSize=100m parameters limit the number of files
and their size. Also rotates the gc log.

The -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCCause
-XX:+PrintTenuringDistribution -XX:+PrintReferenceGC -XX:+PrintAdaptiveSizePolicy parameters gather info on object
age and reference GC time for further tuning if needed.

The -XX:+PrintHeapAtGC parameter will generate a large amount of logging to the specified gc log file. Please ensure
there is adequate file system space when enabling this option. All the information produced by this option is known to
the JVM; therefore, there is no extra processing required as a result of enabling this parameter. However, there will be
a slight amount of additional I/O overhead which should amount to less than 3%.

The -XX:+UseG1GC paramter enables the G1GC gc policy.

The -XX:+UseStringDeduplication parameter is looking for the strings with
the same contents and canonicalizing the underlying char[] with string characters.

The -XX:+ParallelRefProcEnabled parallelize reference processing, reducing
young and old GC times.

The -XX:+ExplicitGCInvokesConcurrentAndUnloadsClasses avoid explicit System.gc() call
triggering full GC, instead trigger G1 and unload classes.

The -XX:+UnlockDiagnosticVMOptions -XX:G1SummarizeRSetStatsPeriod=1 parameters enable
additional logging for G1 status.

The -XX:MaxMetaspaceExpansion=64M parameter avoids triggering full GC when we just
allocate a bit more metaspace. The metaspace region gets cleaned automatically.

The -XX:+UnlockExperimentalVMOptions parameter allows the values of experimental
flags to be changed by unlocking them.

The -XX:G1NewSizePercent=20 parameter sets the percent of the heap to use as the
minimum for the young generation size.

The -XX:+UseConcMarkSweepGC parameter enables the Concurrent Mark Sweep gc policy.
The -XX:+ExplicitGCInvokesConcurrentAndUnloadsClasses parameter fixes issues with
superlong explicit GC cycles
(switches to concurrent GC, but also enables class unloading so we don’t get memory leaks).

The -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled parameters parallelize
re-marking and reference processing. The first will reduce full GC time, the latter should
reduce both GC times.

The -XX:+CMSClassUnloadingEnabled parameter allows GC of classloaders & classes.

The -XX:+ScavengeBeforeFullGC -XX:+CMSScavengeBeforeRemark parameters will force young
gen GC before full GC (reduces full GC duration).

The -XX:NewSize=512m -XX:MaxNewSize=3g -XX:NewRatio=2 parameters are a starting point, but
will differ from system to system.

The XX:+UseCompressedOops allows references to be 32-bit in a 64-bit JVM and access close to 32 GB of heap. This can save a significant amount of memory and potentially improve performance.

C. Java Home Environment Variable

It is recommended to set the JAVA_HOME environment variable in both Linux
and Windows environments. The Java JDK’s bin directory should also be in the
environment’s PATH environment variable. This will allow for easier acess to Java JDK
commands such as jstack and jmap. Please note that the jstack command is only
available in the Java JDK not the Java JRE.

Please also note that core files and heapdump files can be large and are generated
in the working directory of the JVM. Thus, it is recommended that there is adequate disk space for that directory.

For example, on Windows the PATH enviornment variable will include: C:\Program Files\Java\jdk1.8.0_102\bin

Running jstack in terminal window or command prompt should generate the following output:

jstackInTerminal.png

D. Log Startup Timing Info

Set -Djenkins.model.Jenkins.logStartupPerformance=true in your Jenkins environment to have Jenkins output additional
data to the jenkins.log showing how much time is being spent in a specific component during startup.

E. Jenkins Admin Should Be Able To Generate A Support Bundle via jenkins-cli.jar

This will allow Jenkins admins to generate support bundles in the event the Jenkins User Interface (UI) is not available
even though Jenkins is running.

Please note that Jenkins accounts must have the Overall/Read account permission to access the CLI.

The jenkins-cli.jar can be downloaded from JENKINS_URL/cli.

More information about the jenkins-cli.jar support command can be viewed by navigating to JENKINS_URL/cli/command/support .

Additional information about the jenkins-cli.jar can be found at jenkins-cli.jar wiki page.

Please the screen shot below showing a support bundle generated via jenkins-cli.jar using the username and password.
The jenkins-cli.jar authentication can be set up to use ssh keys if you prefer as documented in the jenkins-cli.jar wiki page
linked above.

jenkinsCLISupportBundle.png

Have more questions? Submit a request

3 Comments

Please sign in to leave a comment.