Issue
- Best Practice For Java Heap settings
Environment
- CloudBees CI (CloudBees Core) on Modern Platforms
- CloudBees CI (CloudBees Core) on Traditional Platforms
- CloudBees Jenkins Distribution
- CloudBees Jenkins Team (CJT)
- CloudBees Jenkins Platform - Client Master (CJP-CM)
- CloudBees Jenkins Platform - Operations Center (CJP-OC)
- CloudBees Jenkins Enterprise (CJE)
- Jenkins LTS
Resolution
Traditional Platform
Use -Xmx
and -Xms
Recommended Heap Specifications can be found in CloudBees JVM Guide.
Modern Platform
Use Container resources
spec
with -XX:MinRAMPercentage
and -XX:MaxRAMPercentage
Since JDK 8u191 XX:+UseContainerSupport
is activated by default, everything should work out of the box. It also introduces -XX:MaxRAMPercentage
which takes a value between 0 and 100 (Note: values should be double type see docker-library/openjdk/issues/350). This allows fine grained control of the amount of RAM the JVM is allowed to allocate.
So we now know that the JVM is container aware and we should set the provided amount of memory in our runtime environment. For Jenkins, in general, the same requests/limits is recommended, in case the request is too low will cause evictions if the node that is running the pod is running out of memory.
...
resources:
limits:
memory: 6Gi
requests:
memory: 6Gi
...
Defining just the limits
assumes the same requests
value.
...
resources:
limits:
memory: 6Gi
...
Notes
a. Do not set -Xmx
, -Xms
. Those options set the JVM Heap size directly and it is best to let the JVM infer these based on current limits
. Use -XX:MinRAMPercentage
or -XX:MaxRAMPercentage
instead.
- If
-Xmx
is passed, the JVM will ignore both-XX:MaxRAMPercentage
and-XX:MinRAMPercentage
as you can see in the following example.
$ docker run -m 1GB openjdk:10 java -XX:MaxRAMPercentage=25 -Xmx512m -XshowSettings:vm -version
Max. Heap Size: 512.00M
Using VM: OpenJDK 64-Bit Server VM
b. Both -XX:MaxRAMPercentage
and -XX:MinRAMPercentage
are used to determine the maximum Java heap size. even though -XX:MinRAMPercentage
’s name might make us think its argument could be used to configure minimum heap size, that is not the case. -XX:MinRAMPercentage
JVM argument will be used to compute Java heap size only if your overall available memory’s size in the physical server (or in the container) is less than 250MB (approximately). See examples below.
- 100 MB memory container example.
$ docker run -m 100MB openjdk:10 java -XX:MaxRAMPercentage=25 -XX:MinRAMPercentage=50 -XshowSettings:vm -version
Max. Heap Size (Estimated): 48.38M
Using VM: OpenJDK 64-Bit Server VM
- 1 GB memory container example.
$ docker run -m 1GB openjdk:10 java -XX:MaxRAMPercentage=25 -XX:MinRAMPercentage=50 -XshowSettings:vm -version
Max. Heap Size (Estimated): 247.50M
Using VM: OpenJDK 64-Bit Server VM
For a more detailed explanation on the meaning of these two arguments please check: Difference between InitialRAMPercentage, MinRAMPercentage, MaxRAMPercentage.
c. Do not set -XX:MaxRAMFraction
. It is deprecated in favor of MaxRAMPercentage
see JDK-8186315.
d. A ratio JVM heap / Container Memory limit higher than 0.5
is known to be unstable and may cause unexpected master restarts due to JVM off heap usage.
Memory Estimation
Use your favorite APM tool to monitor the Memomory Heap Consumption by your instance. The right heap value should be a value between 80-90% of total heap consumption when the instance is at the maximum workload. Increased heap memory will produce long Garbage Collection pauses, which can be observed via slowness in the UI and sometimes even side effects like agent disconnections.
Horizontal Scaling
If your instance is not able to support its current workload (due to number of jobs and dynamic configurations) you should consider scaling horizontally your infrastructure by adding more masters to divide the workload more efficiently.
References:
2 Comments