KBEC-00298 - Getting the Flow Java process thread dump under Linux

Summary

Under normal circumstances, you can get the Flow process thread dump by logging into Flow and running:

ectool getServerStatus --diagnostics 1

However, if you are trying to troubleshoot a hanging Flow process, the process will not be responsive to API commands.

Solution

You can get the thread dump directly through Linux commands.

Note: for thread activity analysis, you will need to produce multiple thread dumps spaced a short time apart (typically 30 seconds) in order to properly monitor thread activity over a time span. If support has requested thread dumps from your system, please send 5-10 dumps. 

Getting the PID

First, you need to acquire the process ID of the Java service wrapper for Flow. This PID is stored in the Flow directory (/opt/electriccloud/electriccommander by default) under server/commander.pid. E.g. if the file contains the number "964," then the wrapper PID is 964. You can confirm this is the correct PID by running:

ps -ef | grep -i  | grep -i commander | grep -i java

Your output will look something like this:

vagrant  982  964    15  21:51 ?     00:01:12 /opt/electriccloud/electriccommander/jre/bin/java ... (the rest is ommitted)

Note that 982 is the Flow child process PID and 964 is the parent wrapper PID we found in commander.pid. Using either the parent or the child PID will produce the same results, as the parent wrapper process simply passes the command to the child Commander process.

Getting the thread dump

Run:

kill -3 

The -3 option sends the signal that will tell the process to create a thread dump. DO NOT run kill with any other signal option, or you might terminate the Flow process. 

Repeat this step 5-10 times to get a full set of thread dumps over a time span.

The thread dump is now logged to logs/commander-service.log (or logs/service.log in older versions) in the Flow root directory. It can then be cleaned and opened with tools such as Samurai.

Alternative Solution

The thread dump can also be obtained with the JDK jstack utility. This utility can be used on both Linux and Windows machines but may require additional installations. See KBEC-00052 for details on this method.

See Also

KBEC-00052 - Installing JDK and taking a jstack threads dump or stack dump

Applies to

  • Product versions: all
  • OS versions: Linux

Have more questions?

0 Comments

Please sign in to leave a comment.