Jenkins Hangs on Workspaces Cleanup

Issue

"Computer.threadPoolForRemoting [#...]" ... waiting for monitor entry [...]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at com.thoughtworks.xstream.core.DefaultConverterLookup.lookupConverterForType(DefaultConverterLookup.java:49)
	- waiting to lock <...> (a com.thoughtworks.xstream.core.DefaultConverterLookup)
	at com.thoughtworks.xstream.XStream$1.lookupConverterForType(XStream.java:498)
	at ...
	at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
	at hudson.XmlFile.write(XmlFile.java:170)
	at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.persist(AsyncResourceDisposer.java:175)
	at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.access$300(AsyncResourceDisposer.java:65)
	at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer$WorkItem.run(AsyncResourceDisposer.java:262)
	at ...

Environment

Related Issues

Resolution

The Resource Disposer plugin has a known issue JENKINS-46460 that is causing massive contention on a single lock in XSTREAM.

When the issue is exposed:

  • the administrative monitor of the Resource Disposer is active. You can check for the details of the cleanup tasks that are hanging at $JENKINS_URL/administrativeMonitor/AsyncResourceDisposer.
  • the thread dump show AsyncResourceDisposer.persist stacktraces like the following:
"Computer.threadPoolForRemoting [#...]" ... waiting for monitor entry [...]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at com.thoughtworks.xstream.core.DefaultConverterLookup.lookupConverterForType(DefaultConverterLookup.java:49)
	- waiting to lock <...> (a com.thoughtworks.xstream.core.DefaultConverterLookup)
	at com.thoughtworks.xstream.XStream$1.lookupConverterForType(XStream.java:498)
	at ...
	at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
	at hudson.XmlFile.write(XmlFile.java:170)
	at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.persist(AsyncResourceDisposer.java:175)
	at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer.access$300(AsyncResourceDisposer.java:65)
	at org.jenkinsci.plugins.resourcedisposer.AsyncResourceDisposer$WorkItem.run(AsyncResourceDisposer.java:262)
	at ...

How to address this ?

You can stop the cleanup task by hitting “Stop Tracking” on the entries listed in the administrative monitor page. This may help to reduce the load. A restart of the Client Master is usually needed to resolve the issue but only for a short term until the next hang.

We recommend disabling the Resource Disposer Plugin (and also the Workspace Cleanup plugin which uses this functionality) until JENKINS-46460 is resolved.

If you are using Pipeline, the Workspace Cleanup plugin step is not need. You can replace the workspace cleanup with shell/batch delete commands or with the deleteDir step.

Note: It is also preferred to use the features of the various SCMs plugins to clean untracked or modified files before updating. This avoids the need for an expensive full clone.

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.