KBEC-00430 - Check "open file" Limit of the Flow Server Process

Problem

You may  get "Too many  open files" message when using Cloudbees products in *nix system. For example, here is a WARN reported by the Cloudbees Accelerator cluster manager:

2019-10-14T07:26:00.798 | WARN  | 0.0.0.0:8031} |                                               | ServerConnector                |

java.io.IOException: Too many open files

        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)

        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)

        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)

        at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)

        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)

        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)

        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)

        at java.lang.Thread.run(Thread.java:748)

And here is logs written by Cloudbees Flow server:

INFO | jvm 1 | 2016/02/16 02:46:10.542 | 2016-02-16 02:46:10.292:WARN:oejs.ServerConnector:qtp1921015445-28-acceptor-0@6ca61f50-ServerConnector@4f24894d{HTTP/1.1}{0.0.0.0:8000}:
INFO | jvm 1 | 2016/02/16 02:46:10.542 | java.io.IOException: Too many open files
INFO | jvm 1 | 2016/02/16 02:46:10.542 | at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
INFO | jvm 1 | 2016/02/16 02:46:10.542 | at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
INFO | jvm 1 | 2016/02/16 02:46:10.542 | at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
INFO | jvm 1 | 2016/02/16 02:46:10.542 | at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)
INFO | jvm 1 | 2016/02/16 02:46:10.542 | at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)
INFO | jvm 1 | 2016/02/16 02:46:10.542 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
INFO | jvm 1 | 2016/02/16 02:46:10.542 | at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
INFO | jvm 1 | 2016/02/16 02:46:10.542 | at java.lang.Thread.run(Thread.java:745)

This means that the process exceeds the "open file limit" from the operating system.

Solution

You can change the settings to enable the process to get a bigger limit. Please refer to the following articles for more details:

KBEC-00164 - Setting Linux server and agent file descriptor limits

KBEC-00379 - Necessary Settings in CentOS When Running ElectricFlow in CentOS

You may want to check the current limit before changing to a bigger limit. There are many ways  to do this. Here are some convenient commands to check the limits of the Cloudbees product processes.

checking the limits of the Flow server process:

ubuntu@ip-172-31-41-128:~$ cat /proc/`ps -ef | grep ElectricFlowServer | awk '{ print $2 }' | head -1`/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 64063 64063 processes
Max open files 1048576 1048576 files
Max locked memory 16777216 16777216 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 64063 64063 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
ubuntu@ip-172-31-41-128:~$

checking the limits of the Flow agent process:

ubuntu@ip-172-31-41-128:~$ cat /proc/`ps -ef | grep ElectricFlowAgent | awk '{ print $2 }' | head -1`/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 64063 64063 processes
Max open files 1048576 1048576 files
Max locked memory 16777216 16777216 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 64063 64063 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
ubuntu@ip-172-31-41-128:~$

checking the limits of the Flow Repository Server process:

ubuntu@ip-172-31-41-128:~$ cat /proc/`ps -ef | grep ElectricFlowRepository | awk '{ print $2 }' | head -1`/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 64063 64063 processes
Max open files 1048576 1048576 files
Max locked memory 16777216 16777216 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 64063 64063 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
ubuntu@ip-172-31-41-128:~$

checking the limits of the Accelerator cluster manager process:

Assuming that the cluster manager is run by the default user "eacmuser" with default parameters, the following command should work:

ubuntu@ip-172-31-41-128:~$ cat /proc/`ps -u eacmuser -f | grep accelerator | awk '{ print $2 }' | head -1`/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 64063 64063 processes
Max open files 1048576 1048576 files
Max locked memory 16777216 16777216 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 64063 64063 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
ubuntu@ip-172-31-41-128:~$

 

Have more questions?

0 Comments

Article is closed for comments.