Issue

You see this stacktrace in jenkins.log:

Caused by: java.lang.OutOfMemoryError: unable to create new native thread
      at java.lang.Thread.start0(Native Method)
      at java.lang.Thread.start(Thread.java:714)
      at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
      at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1371)

Environment

• Jenkins, CloudBees Jenkins Enterprise, CloudBees Jenkins Operations Center
• Linux, Solaris

Resolution

Great chances are you are hitting one of the following system OS limit:

• number of running user processes
• number of opened files

To see the current limits of your system, run ulimit -a on the command-line with the user running Jenkins (usually jenkins). You should see something like this:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 30
file size               (blocks, -f) unlimited
pending signals                 (-i) 30654
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 99
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

To increase limits, add these lines to /etc/security/limits.conf:

jenkins      soft   nofile  4096
jenkins      hard   nofile  8192
jenkins      soft   nproc   30654        
jenkins      hard   nproc   30654

Note that this assumes jenkins is the Unix user running the Jenkins process.

You can now logout and login and check that the limits are correctly modified with ulimit -a (limits are applied when the Unix user logs in).

If it’s not, check that one PAM configuration file in /etc/pam.d/ contains this line:

session required pam_limits.so