Issue
- Jenkins suffer slowness and performance issues
branch-api
version is lower than2.5.6
- In the thread dumps you can observe
BLOCKED
threads like
"Computer.threadPoolForRemoting [#15979]" #398547 daemon prio=5 os_prio=0 tid=0x00007f0509fcf000 nid=0x1e982 waiting for monitor entry [0x00007f0246221000]
java.lang.Thread.State: BLOCKED (on object monitor)
at jenkins.branch.WorkspaceLocatorImpl.locate(WorkspaceLocatorImpl.java:158)
- waiting to lock <0x00000003c0677358> (a hudson.slaves.DumbSlave)
at jenkins.branch.WorkspaceLocatorImpl.locate(WorkspaceLocatorImpl.java:129)
at jenkins.branch.WorkspaceLocatorImpl.access$100(WorkspaceLocatorImpl.java:80)
at jenkins.branch.WorkspaceLocatorImpl$Deleter$CleanupTask.run(WorkspaceLocatorImpl.java:402)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- <0x000000049de00a48> (a java.util.concurrent.ThreadPoolExecutor$Worker)
blocked waiting for the “Workspace clean-up thread”
"Workspace clean-up thread" #220769 daemon prio=5 os_prio=0 tid=0x00007f054cb47000 nid=0x1414 in Object.wait() [0x00007f046e386000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at hudson.remoting.FastPipedInputStream.read(FastPipedInputStream.java:175)
- locked <0x000000048d215470> (a [B)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
- locked <0x000000048d227520> (a java.io.InputStreamReader)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
- locked <0x000000048d227520> (a java.io.InputStreamReader)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at jenkins.branch.WorkspaceLocatorImpl.load(WorkspaceLocatorImpl.java:222)
at jenkins.branch.WorkspaceLocatorImpl.locate(WorkspaceLocatorImpl.java:159)
- locked <0x00000003c0677358> (a hudson.slaves.DumbSlave)
at jenkins.branch.WorkspaceLocatorImpl.locate(WorkspaceLocatorImpl.java:129)
at jenkins.branch.WorkspaceLocatorImpl.locate(WorkspaceLocatorImpl.java:125)
at hudson.model.Slave.getWorkspaceFor(Slave.java:335)
at hudson.model.WorkspaceCleanupThread.execute(WorkspaceCleanupThread.java:78)
at hudson.model.AsyncPeriodicWork$1.run(AsyncPeriodicWork.java:101)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
Environment
- CloudBees CI (CloudBees Core) on modern cloud platforms - Managed controller
- CloudBees CI (CloudBees Core) on traditional platforms - Client controller
- CloudBees Jenkins Enterprise - Managed controller
- CloudBees Jenkins Platform - Client controller
- Jenkins LTS
- Branch API plugin
Explanation
Thread contention with the WorkspaceLocatorImpl
can cause Jenkins to hang. Large amounts of workspace cleanup threads, such as from deleting large Multibranch/Organization items, can cause this behaviour.
Other causes are possible like a filesystem poor performance.
Resolution
Upgrade Branch API plugin to version 2.5.6 or higher.
0 Comments