Issue
You see this stacktrace in jenkins.log
:
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1371)
Environment
- CloudBees CI (CloudBees Core) on traditional platforms - Client controller (CORE-CM)
- CloudBees CI (CloudBees Core) on traditional platforms - Operations Center (CORE-OC)
- CloudBees Jenkins Team (CJT)
- CloudBees Jenkins Platform - Client controller (CJP-CM)
- CloudBees Jenkins Platform - Operations Center (CJP-OC)
- Jenkins LTS
- Linux, Solaris
Resolution
Great chances are you are hitting one of the following system OS limit:
- number of running user processes
- number of opened files
To see the current limits of your system, run ulimit -a
on the command-line with the user running Jenkins (usually jenkins
). You should see something like this:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 30
file size (blocks, -f) unlimited
pending signals (-i) 30654
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 99
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
To increase limits, add these lines to /etc/security/limits.conf
:
jenkins soft nofile 4096
jenkins hard nofile 8192
jenkins soft nproc 30654
jenkins hard nproc 30654
Note that this assumes jenkins
is the Unix user running the Jenkins process.
You can now logout and login and check that the limits are correctly modified with ulimit -a
(limits are applied when the Unix user logs in).
If it’s not, check that one PAM configuration file in /etc/pam.d/
contains this line:
session required pam_limits.so
4 Comments