Issue
Obtaining a thread dump shows, for example:
Found one Java-level deadlock:
=============================
"AbridgedConnectivityWorker-67":
waiting to lock monitor 0x00007f90c2459368 (object 0x00000007a7fe37c0, a hudson.DescriptorExtensionList),
which is held by "PreventRefreshFilter.initAutoRefreshFilter"
"PreventRefreshFilter.initAutoRefreshFilter":
waiting to lock monitor 0x00007f90b81ab2e8 (object 0x00000006c1339400, a hudson.ExtensionList$Lock),
which is held by "JSyncArtifactManagerFactory.upgrade"
"JSyncArtifactManagerFactory.upgrade":
waiting to lock monitor 0x00007f90c2459368 (object 0x00000007a7fe37c0, a hudson.DescriptorExtensionList),
which is held by "PreventRefreshFilter.initAutoRefreshFilter"
Java stack information for the threads listed above:
===================================================
"AbridgedConnectivityWorker-67":
at hudson.ExtensionList.ensureLoaded(ExtensionList.java:317)
- waiting to lock <0x00000007a7fe37c0> (a hudson.DescriptorExtensionList)
at hudson.ExtensionList.iterator(ExtensionList.java:172)
at hudson.ExtensionList.getInstance(ExtensionList.java:162)
at jenkins.model.JenkinsLocationConfiguration.get(JenkinsLocationConfiguration.java:47)
at jenkins.model.Jenkins.getRootUrl(Jenkins.java:2255)
at com.hpe.application.automation.tools.octane.CIJenkinsServicesImpl.getServerInfo(CIJenkinsServicesImpl.java:102)
at com.hp.octane.integrations.services.bridge.BridgeServiceImpl$1.run(BridgeServiceImpl.java:82)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"PreventRefreshFilter.initAutoRefreshFilter":
at hudson.ExtensionList.ensureLoaded(ExtensionList.java:317)
- waiting to lock <0x00000006c1339400> (a hudson.ExtensionList$Lock)
at hudson.ExtensionList.getComponents(ExtensionList.java:183)
at hudson.DescriptorExtensionList.load(DescriptorExtensionList.java:192)
at hudson.ExtensionList.ensureLoaded(ExtensionList.java:318)
- locked <0x00000007a7fe37c0> (a hudson.DescriptorExtensionList)
at hudson.ExtensionList.iterator(ExtensionList.java:172)
at hudson.ExtensionList.get(ExtensionList.java:149)
at hudson.plugins.claim.ClaimConfig.get(ClaimConfig.java:202)
at hudson.plugins.claim.http.PreventRefreshFilter.initAutoRefreshFilter(PreventRefreshFilter.java:43)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
at jenkins.model.Jenkins$5.runTask(Jenkins.java:1083)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"JSyncArtifactManagerFactory.upgrade":
at hudson.ExtensionList.ensureLoaded(ExtensionList.java:316)
- waiting to lock <0x00000007a7fe37c0> (a hudson.DescriptorExtensionList)
at hudson.ExtensionList.iterator(ExtensionList.java:172)
at hudson.ExtensionList.get(ExtensionList.java:149)
at jenkins.security.apitoken.ApiTokenPropertyConfiguration.get(ApiTokenPropertyConfiguration.java:58)
at jenkins.security.ApiTokenProperty$DescriptorImpl.newInstance(ApiTokenProperty.java:386)
at jenkins.security.ApiTokenProperty$DescriptorImpl.newInstance(ApiTokenProperty.java:359)
at hudson.model.User.allocateDefaultPropertyInstancesAsNeeded(User.java:211)
at hudson.model.User.load(User.java:198)
at hudson.model.User.<init>(User.java:191)
at hudson.model.User.getOrCreateById(User.java:523)
at hudson.model.User.getById(User.java:619)
at hudson.model.User.get(User.java:603)
at hudson.model.User.current(User.java:586)
at hudson.plugins.jobConfigHistory.PluginUtils.isUserExcluded(PluginUtils.java:121)
at hudson.plugins.jobConfigHistory.JobConfigHistorySaveableListener.onChange(JobConfigHistorySaveableListener.java:55)
at hudson.model.listeners.SaveableListener.fireOnChange(SaveableListener.java:81)
at hudson.model.Descriptor.save(Descriptor.java:874)
- locked <0x00000007a87ba378> (a jenkins.telemetry.Correlator)
at jenkins.telemetry.Correlator.<init>(Correlator.java:51)
at jenkins.telemetry.Correlator$$FastClassByGuice$$5678e8d6.newInstance(<generated>)
at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:61)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:105)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145)
at hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:432)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012)
at hudson.ExtensionFinder$GuiceFinder._find(ExtensionFinder.java:394)
at hudson.ExtensionFinder$GuiceFinder.find(ExtensionFinder.java:385)
at hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:493)
at hudson.ExtensionList.load(ExtensionList.java:380)
at hudson.ExtensionList.ensureLoaded(ExtensionList.java:318)
- locked <0x00000006c1339400> (a hudson.ExtensionList$Lock)
at hudson.ExtensionList.iterator(ExtensionList.java:172)
at jenkins.model.Jenkins.getDescriptorByType(Jenkins.java:1539)
at com.cloudbees.jenkins.plugins.jsync.archiver.JSyncArtifactManagerFactory.upgrade(JSyncArtifactManagerFactory.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
at jenkins.model.Jenkins$5.runTask(Jenkins.java:1083)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Found 1 deadlock.
We see the plugin mentioned:
at com.hpe.application.automation.tools.octane.CIJenkinsServicesImpl.getServerInfo(CIJenkinsServicesImpl.java:102)
Environment
- CloudBees CI (CloudBees Core)
- CloudBees CI (CloudBees Core) on modern cloud platforms - Managed Master
- CloudBees CI (CloudBees Core) on modern cloud platforms - Operations Center
- CloudBees CI (CloudBees Core) on traditional platforms - Client Master
- CloudBees CI (CloudBees Core) on traditional platforms - Operations Center
- CloudBees Jenkins Enterprise
- CloudBees Jenkins Enterprise - Managed Master
- CloudBees Jenkins Enterprise - Operations Center
- CloudBees Jenkins Platform - Client Master
- CloudBees Jenkins Platform - Operations Center
- CloudBees Jenkins Distribution
- HP Application Automation Tools
- HPE Application Automation Tools
- Micro Focus Application+Automation Tools
Resolution
A thread dump can be obtained using the following guides:
- https://support.cloudbees.com/hc/en-us/articles/205199280-How-to-generate-a-thread-dump-
- https://wiki.jenkins.io/display/JENKINS/Obtaining+a+thread+dump
Also see Understanding Thread Dumps
Upgrading hp-application-automation-tools-plugin
to >= 5.6.2
should resolve the issue.
Uninstalling hp-application-automation-tools-plugin
should also resolve the issue. See https://jenkins.io/doc/book/managing/plugins/#uninstalling-a-plugin
0 Comments