How to generate a heap dump?

Issue

  • How do I generate a heap dump.
  • Diagnosing memory issues: OutOfMemoryError (OOM). Analyzing possible Memory Leaks.

Resolution

A heap dump is a snapshot of the memory of a Java process at a certain point in time. There are different formats for persisting this data, and depending on the format it may contain different pieces of information, but in general the snapshot contains information about the Java objects and classes in the heap at the moment the snapshot was triggered. Usually a full GC is triggered before the heap dump is written so it contains information about the remaining objects.

1. Get the PID of the Java process where Jenkins is running

To generate a head dump the first part is to obtain the process id (or PID). There are multiple methods to obtain the PID:

  • jps - From Unix Terminal or Windows Cmd with the JDK installed into the OS. Lists the instrumented Java Virtual Machines (JVMs) on the target system.
  • ps -ef | grep java - From Unix Terminal only. It is used to obtain the process id of all java processes.
  • Process explorer - There are task manager UI applications (System Tools) which can be used to obtain the process id. Find the process and view its PID in the corresponding column.
    • On Windows and Linux (Ubuntu) named Task manager
    • On Mac OS named Activity Monitor

2. Get the Heap Dump

NOTE: You must run the following commands as the user that Jenkins is running as. Usually this user is ‘jenkins.’ No other user, including root, will be able to run the command successfully.

After obtaining the process id ($PID), the next step is to generate the heap dump using jcmd:

  • jcmd $PID GC.heap_dump /tmp/$FILENAME.hprof - From Unix Terminal or Windows Powershell with the JDK running Jenkins. Pass the full path to the desired output file. It prints shared object memory maps or heap memory details for a process, core file, or remote debug server. Please change the $FILENAME for any of the heap dumps you would like to send to us via our Upload Site and ensure you specify the .hprof file extension.

jcmd is the preferred method for generating a heap dump per Oracle Documentation

If you are experiencing a problem using jcmd please use the alternative jmap method: jmap -dump:format=b,file=$FILENAME.hprof $PID - From Unix Terminal or Windows Powershell

NOTE: The JVM will need to perform a full garbage collection cycle before it can generate a heap dump, in order to free as many unused objects as possible first. Jenkins will be paused/unavailable during this time. For very large heap sizes, the full GC could be slow, so it’s important to realize that getting a heap dump may impact users of the application.

Generating heap dumps periodically during an OOM issue is beneficial, particularly after a service restart to baseline the instance, and subsequently later to show the memory growth over a period of time. Attach the output generated from these commands to a CloudBees support ticket, so we can help diagnose further.

Heap Dumps can be quite large in size, therefore the split utility comes in handy here, as our upload site supports 10GB uploads, but can be difficult depending on your internet connection. Thus, use the split linux utility to split the generated heapdump file into 2GB segments for ease of upload to your support ticket and our team will cat them together once received.
or Windows Cmd

  • split -b 2gb $FILENAME - From Unix Terminal or Windows Powershell

To concatenate files after splitting:

  • cat ${file1} {file2} {file3} > {output_filename} -From Unix Terminal or Windows Powershell
  • copy /B {file1} + {file2} + {file3} {output_filename} -From Windows CMD

Troubleshooting

Sometimes you will face some problems while collecting the heap dump and will get errors such as:

Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding

In some cases, enabling the -F option is extremely slow, especially for large heaps. The error mentioned above is generally related to permissions issues with the user taking the heap dump. To prevent this kind of errors, the user taking the heap dump should be the same user who is running the Java process. And sometimes you will even need to call the user explicitly running the command using the following syntax:

sudo -su <JENKINS_USER> -c "jmap...."

If you are using CentOS 7 or higher, you have to take into consideration that SELinux will stop Jmap from writing heapdumps unless the directory where the dump is going to be generated (runtime user owned). The easiest way to proceed is to disable temporarily SELinux.

echo 0 > /selinux/enforce

or

setenforce 0

Notes

  1. Some of these commands might require admin permissions.
  2. A JDK is required, not JRE. To run the proposed commands you need to run them from $JDK_PATH/bin or from anywhere by including $JDK_PATH/bin into your System $PATH.
  3. For Windows > Task Manager if PID is not displayed > click View - Select Columns > Select the PID (Process Identifier).
  4. Unix Terminal includes Linux and MacOS OS.
  5. As an option, for running Unix commands into Windows Cygwin can be installed.
  6. Please check the KB CloudBees Jenkins Platform supported Java versions.

References

Jps

Jmap

Jcmd

Was this article helpful?
2 out of 2 found this helpful

Have more questions?

1 Comments

Date Votes
  • 0
    Avatar
    Kirill Gostaf

    One may face a permission denied error:

    bash-4.4$ jcmd 6 GC.heap_dump filename=/tmp/jul21.log
    6:
    Permission denied
    bash-4.4$
    bash-4.4$
    bash-4.4$ jcmd 6 GC.heap_dump /tmp/jul21.log
    6:
    Heap dump file created
    bash-4.4$
    Edited by Kirill Gostaf
Please sign in to leave a comment.