Issue

Obtaining a thread dump shows, for example:

Found one Java-level deadlock:
=============================
"AbridgedConnectivityWorker-67":
  waiting to lock monitor 0x00007f90c2459368 (object 0x00000007a7fe37c0, a hudson.DescriptorExtensionList),
  which is held by "PreventRefreshFilter.initAutoRefreshFilter"
"PreventRefreshFilter.initAutoRefreshFilter":
  waiting to lock monitor 0x00007f90b81ab2e8 (object 0x00000006c1339400, a hudson.ExtensionList$Lock),
  which is held by "JSyncArtifactManagerFactory.upgrade"
"JSyncArtifactManagerFactory.upgrade":
  waiting to lock monitor 0x00007f90c2459368 (object 0x00000007a7fe37c0, a hudson.DescriptorExtensionList),
  which is held by "PreventRefreshFilter.initAutoRefreshFilter"

Java stack information for the threads listed above:
===================================================
"AbridgedConnectivityWorker-67":
	at hudson.ExtensionList.ensureLoaded(ExtensionList.java:317)
	- waiting to lock <0x00000007a7fe37c0> (a hudson.DescriptorExtensionList)
	at hudson.ExtensionList.iterator(ExtensionList.java:172)
	at hudson.ExtensionList.getInstance(ExtensionList.java:162)
	at jenkins.model.JenkinsLocationConfiguration.get(JenkinsLocationConfiguration.java:47)
	at jenkins.model.Jenkins.getRootUrl(Jenkins.java:2255)
	at com.hpe.application.automation.tools.octane.CIJenkinsServicesImpl.getServerInfo(CIJenkinsServicesImpl.java:102)
	at com.hp.octane.integrations.services.bridge.BridgeServiceImpl$1.run(BridgeServiceImpl.java:82)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
"PreventRefreshFilter.initAutoRefreshFilter":
	at hudson.ExtensionList.ensureLoaded(ExtensionList.java:317)
	- waiting to lock <0x00000006c1339400> (a hudson.ExtensionList$Lock)
	at hudson.ExtensionList.getComponents(ExtensionList.java:183)
	at hudson.DescriptorExtensionList.load(DescriptorExtensionList.java:192)
	at hudson.ExtensionList.ensureLoaded(ExtensionList.java:318)
	- locked <0x00000007a7fe37c0> (a hudson.DescriptorExtensionList)
	at hudson.ExtensionList.iterator(ExtensionList.java:172)
	at hudson.ExtensionList.get(ExtensionList.java:149)
	at hudson.plugins.claim.ClaimConfig.get(ClaimConfig.java:202)
	at hudson.plugins.claim.http.PreventRefreshFilter.initAutoRefreshFilter(PreventRefreshFilter.java:43)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
	at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
	at jenkins.model.Jenkins$5.runTask(Jenkins.java:1083)
	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
"JSyncArtifactManagerFactory.upgrade":
	at hudson.ExtensionList.ensureLoaded(ExtensionList.java:316)
	- waiting to lock <0x00000007a7fe37c0> (a hudson.DescriptorExtensionList)
	at hudson.ExtensionList.iterator(ExtensionList.java:172)
	at hudson.ExtensionList.get(ExtensionList.java:149)
	at jenkins.security.apitoken.ApiTokenPropertyConfiguration.get(ApiTokenPropertyConfiguration.java:58)
	at jenkins.security.ApiTokenProperty$DescriptorImpl.newInstance(ApiTokenProperty.java:386)
	at jenkins.security.ApiTokenProperty$DescriptorImpl.newInstance(ApiTokenProperty.java:359)
	at hudson.model.User.allocateDefaultPropertyInstancesAsNeeded(User.java:211)
	at hudson.model.User.load(User.java:198)
	at hudson.model.User.<init>(User.java:191)
	at hudson.model.User.getOrCreateById(User.java:523)
	at hudson.model.User.getById(User.java:619)
	at hudson.model.User.get(User.java:603)
	at hudson.model.User.current(User.java:586)
	at hudson.plugins.jobConfigHistory.PluginUtils.isUserExcluded(PluginUtils.java:121)
	at hudson.plugins.jobConfigHistory.JobConfigHistorySaveableListener.onChange(JobConfigHistorySaveableListener.java:55)
	at hudson.model.listeners.SaveableListener.fireOnChange(SaveableListener.java:81)
	at hudson.model.Descriptor.save(Descriptor.java:874)
	- locked <0x00000007a87ba378> (a jenkins.telemetry.Correlator)
	at jenkins.telemetry.Correlator.<init>(Correlator.java:51)
	at jenkins.telemetry.Correlator$$FastClassByGuice$$5678e8d6.newInstance(<generated>)
	at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
	at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:61)
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:105)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145)
	at hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:432)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
	at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016)
	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
	at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012)
	at hudson.ExtensionFinder$GuiceFinder._find(ExtensionFinder.java:394)
	at hudson.ExtensionFinder$GuiceFinder.find(ExtensionFinder.java:385)
	at hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:493)
	at hudson.ExtensionList.load(ExtensionList.java:380)
	at hudson.ExtensionList.ensureLoaded(ExtensionList.java:318)
	- locked <0x00000006c1339400> (a hudson.ExtensionList$Lock)
	at hudson.ExtensionList.iterator(ExtensionList.java:172)
	at jenkins.model.Jenkins.getDescriptorByType(Jenkins.java:1539)
	at com.cloudbees.jenkins.plugins.jsync.archiver.JSyncArtifactManagerFactory.upgrade(JSyncArtifactManagerFactory.java:47)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
	at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
	at jenkins.model.Jenkins$5.runTask(Jenkins.java:1083)
	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Found 1 deadlock.

We see the plugin mentioned:

	at com.hpe.application.automation.tools.octane.CIJenkinsServicesImpl.getServerInfo(CIJenkinsServicesImpl.java:102)

Environment

Resolution

A thread dump can be obtained using the following guides:

Also see https://go.cloudbees.com/docs/solutions/jvm-troubleshooting/#_understanding_thread_dumps

Upgrading hp-application-automation-tools-plugin to >= 5.6.2 should resolve the issue.

Uninstalling hp-application-automation-tools-plugin should also resolve the issue. See https://jenkins.io/doc/book/managing/plugins/#uninstalling-a-plugin

Older versions of Micro Focus Application Automation Tools plugin can cause deadlocks

Issue

Environment

Resolution

0 Comments