Java Heap settings best practice

Issue

  • Best Practice For Java Heap settings

Environment

Resolution

Traditional Platform

Use -Xmx and -Xms

Recommended Heap Specifications can be found in CloudBees JVM Guide.

Modern Platform

Use Container resources spec with -XX:MinRAMPercentage and -XX:MaxRAMPercentage

Since JDK 8u191 XX:+UseContainerSupport is activated by default, everything should work out of the box. It also introduces -XX:MaxRAMPercentage which takes a value between 0 and 100 (Note: values should be double type see docker-library/openjdk/issues/350). This allows fine grained control of the amount of RAM the JVM is allowed to allocate.

So we now know that the JVM is container aware and we should set the provided amount of memory in our runtime environment. For Jenkins, in general, the same requests/limits is recommended, in case the request is too low will cause evictions if the node that is running the pod is running out of memory.

...
resources:
    limits:
      memory: 6Gi
    requests:
      memory: 6Gi
...

Defining just the limits assumes the same requests value.

...
resources:
    limits:
      memory: 6Gi
...

Notes

a. Do not set -Xmx , -Xms in the Master Proviosioning > Java Options. Those options set the JVM Heap size directly and it is best to let the JVM infer these based on current limits. Use -XX:MaxRAMPercentage instead.

  • If -Xmx is passed, the JVM will ignore both -XX:MaxRAMPercentage (also -XX:MinRAMPercentage).

b. Do not enable Enable heap ratio because it calculates the JVM Heap size from the Jenkins Controller Memory in MB and introduces -Xmx and -Xms to the Master Statefulset Manifest JAVA_OPTS (which is not recommended). In case it is enabled, a ratio JVM heap / Container Memory limit higher than 0.7 is known to be unstable and may cause unexpected master restarts due to JVM off heap usage.

c. Use -XX:MaxRAMPercentage to determine the maximum Java heap size. -XX:MinRAMPercentage JVM argument will be used to compute Java heap size only if your overall available memory’s size in the physical server (or in the container) is less than 250MB (approximately). A Host with 250 MB of memory would not been able to support the minimal requirement of Heap Size for CloudBees CI on Production Enviroments: 4GB (See CloudBees Jenkins JVM troubleshooting - Heap Size).

d. Do not set -XX:MaxRAMFraction. It is deprecated in favor of MaxRAMPercentage see JDK-8186315 and After upgrading CloudBees CI on modern platform, the Operation Center - Controller is getting restarted often.

For a more detailed explanation on the meaning of InitialRAMPercentage, MinRAMPercentage, MaxRAMPercentage, refere to this article.

Memory Estimation

Use your favorite APM tool to monitor the Memomory Heap Consumption by your instance. The right heap value should be a value between 80-90% of total heap consumption when the instance is at the maximum workload. Increased heap memory will produce long Garbage Collection pauses, which can be observed via slowness in the UI and sometimes even side effects like agent disconnections.

Horizontal Scaling

If your instance is not able to support its current workload (due to number of jobs and dynamic configurations) you should consider scaling horizontally your infrastructure by adding more masters to divide the workload more efficiently.

References:

Have more questions?

2 Comments

  • 0
    Avatar
    Kostiantyn Gorbunov

    According to Oracle JVM ergonomic page, the maximum heap size NEVER should be 1/4 of the physical memory IN GENERAL CASE. If you read actual source it is just DEFAULT value 1/4 or mem and < 1 GB. This example never ever ever should appear on that kind of reference material.

  • 0
    Avatar
    Carlos Rodriguez Lopez

    @Kostiantyn Gorbunov Thanks for pointing this out

Please sign in to leave a comment.