Java Heap Out of Memory Exception

Issue

You see one of the following in jenkins.log:

java.lang.OutOfMemoryError
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: GC Overhead limit exceeded

Environment

  • Jenkins
  • CloudBees Jenkins Enterprise
  • CloudBees Jenkins Operations Center

Resolution

1. Increasing Heap Size

To increase the heap size, you can add or update -Xms and -Xmx parameters in the command-line argument of the JVM. Jenkins should not be started with less than -Xmx2g. In the case your instance demands more than -Xmx10g any of the following points could be happening

  • Other factors, apart from the Heap Size, are not taking into consideration for improving your performance. please review: CJP Performance Best Practices for Linux
  • Your instance is not able to support its current workload (number of jobs and configuration) so you should consider about escalating horizontally your infra (more masters) to divide this workload more efficiently.

Best Practices

  1. Set your minimum heap size ( -Xms) to at least 1/2 of your maximum size ( -Xmx)
  2. For 64 bit JVM, set -Xmx less than 32G to benefit from -XX:+UseCompressedOops (which comes activated by default in recent JVM).
  3. Jenkins running on Docker: Docker container should run with a heap memory no higher than 1/2 memory of the Host.
  4. The Jenkins Monitoring Plugin can be installed temporally (It’s not recommendable to keep it installed for Production Environment) to monitor the performance of your instance, the right heap value which should be a value between 80-90% of total heap consumption when the instance is at the maximum workload. Big heap memory values might produce big GC pauses which means slowness and sometimes even side effects like agent disconnections.

After the next occurrence of the Out of Memory Exception please upload a Support Bundle and the verbose gc log file to your ticket for review.

If the above steps do not help with diagnosing the out of memory exception, then we would most likely need a Heap Dump to obtain this information. The best practise is including to your existing JAVA_ARGS -XX:+HeapDumpOnOutOfMemoryError, then the next time an out of memory exception is thrown the JVM will generate a heap dump file. You can also set -XX:HeapDumpPath=SPECIFY_PATH to have the heap dump written to that location.

2. Tuning your Garbage Collector according to your Heap Size

Garbage collection (GC) is a dynamic approach to automatic memory management and heap allocation that processes and identifies dead memory blocks and reallocates storage for reuse.

As explained in Prepare Jenkins for Support > Java Parameters > Section II

3. Further diagnostics

If you continue to encounter issues even after the heap space increase and set correct GC settings, then we would need more diagnostic information to debug further. In order to investigate the cause of the Out of Memory Exception you will need to enable GC Verbose. As explained in Prepare Jenkins for Support > Java Parameters > Section I : Verbose gc should be enabled

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.