Issue
The issue can manifest in at least two different ways:
- Client Master fails to start, or
- Client Master fails to lease shared agent from Operations Center
In both cases, the messages with a stack trace similar to the following should be present in the Client Master logs:
2016-01-06 00:00:56.688-0600 [id=96366] WARNING c.c.o.s.m.ConnectedMaster$RemoteSlaveManifest#getSlaves: Could not complete remote call
java.lang.RuntimeException: Writer thread failed
at org.mapdb.AsyncWriteEngine.checkState(AsyncWriteEngine.java:262)
at org.mapdb.AsyncWriteEngine.get(AsyncWriteEngine.java:326)
at org.mapdb.Caches$HashTable.get(Caches.java:245)
at org.mapdb.HTreeMap$HashIterator.findNextLinkedNodeRecur(HTreeMap.java:1081)
at org.mapdb.HTreeMap$HashIterator.findNextLinkedNode(HTreeMap.java:1046)
at org.mapdb.HTreeMap$HashIterator.advance(HTreeMap.java:1033)
at org.mapdb.HTreeMap$HashIterator.moveToNext(HTreeMap.java:992)
at org.mapdb.HTreeMap$EntryIterator.next(HTreeMap.java:1139)
at org.mapdb.HTreeMap$EntryIterator.next(HTreeMap.java:1132)
at com.cloudbees.opscenter.client.cloud.PluginImpl$SlaveManifestImpl.getSlaves(PluginImpl.java:422)
at com.cloudbees.opscenter.server.model.ConnectedMaster$GetRemoteLeases.call(ConnectedMaster.java:2086)
at com.cloudbees.opscenter.server.model.ConnectedMaster$GetRemoteLeases.call(ConnectedMaster.java:2083)
at hudson.remoting.UserRequest.perform(UserRequest.java:120)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at org.jenkinsci.remoting.CallableDecorator.call(CallableDecorator.java:18)
at hudson.remoting.CallableDecoratorList$1.call(CallableDecoratorList.java:21)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at ......remote call to 1-jenkins2 at https://<JENKINS_URL>.com/(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
at hudson.remoting.Channel.call(Channel.java:781)
at com.cloudbees.opscenter.server.model.ConnectedMaster$RemoteSlaveManifest.getSlaves(ConnectedMaster.java:2051)
at com.cloudbees.opscenter.server.model.SharedSlave$PeriodicTidyUp$1.call(SharedSlave.java:865)
at com.cloudbees.opscenter.server.model.SharedSlave$PeriodicTidyUp$1.call(SharedSlave.java:858)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 404
at org.mapdb.StoreWAL.longStackPutSixLong(StoreWAL.java:1060)
at org.mapdb.StoreWAL.longStackPut(StoreWAL.java:1040)
at org.mapdb.StoreDirect.freePhysPut(StoreDirect.java:1055)
at org.mapdb.StoreDirect.freePhysTake(StoreDirect.java:1081)
at org.mapdb.StoreDirect.physAllocate(StoreDirect.java:655)
at org.mapdb.StoreWAL.update(StoreWAL.java:436)
at org.mapdb.EngineWrapper.update(EngineWrapper.java:63)
at org.mapdb.AsyncWriteEngine.access$101(AsyncWriteEngine.java:76)
at org.mapdb.AsyncWriteEngine.runWriter(AsyncWriteEngine.java:220)
at org.mapdb.AsyncWriteEngine$WriterRunnable.run(AsyncWriteEngine.java:172)
... 1 more
2018-05-13 02:21:10.115+0000 [id=45] SEVERE jenkins.InitReactorRunner$1#onTaskFailed: Failed PluginImpl.afterExtensionsAugmented
java.lang.AssertionError: data were not fully read, check your serializer
at org.mapdb.Store.deserialize(Store.java:299)
at org.mapdb.StoreDirect.get2(StoreDirect.java:486)
at org.mapdb.StoreWAL.get2(StoreWAL.java:336)
at org.mapdb.StoreWAL.get(StoreWAL.java:320)
at org.mapdb.EngineWrapper.get(EngineWrapper.java:58)
at org.mapdb.AsyncWriteEngine.get(AsyncWriteEngine.java:333)
at org.mapdb.Caches$HashTable.get(Caches.java:246)
at org.mapdb.EngineWrapper.get(EngineWrapper.java:58)
at org.mapdb.HTreeMap.getInner(HTreeMap.java:448)
at org.mapdb.HTreeMap.getPeek(HTreeMap.java:436)
at org.mapdb.HTreeMap.containsKey(HTreeMap.java:300)
at org.mapdb.HTreeMap$KeySet.contains(HTreeMap.java:817)
at java.util.AbstractCollection.retainAll(AbstractCollection.java:409)
at com.cloudbees.opscenter.client.cloud.PluginImpl$SlaveManifestImpl.createDb(PluginImpl.java:419)
at com.cloudbees.opscenter.client.cloud.PluginImpl$SlaveManifestImpl.<init>(PluginImpl.java:406)
at com.cloudbees.opscenter.client.cloud.PluginImpl.getSlaveManifest(PluginImpl.java:310)
at com.cloudbees.opscenter.client.cloud.PluginImpl.afterExtensionsAugmented(PluginImpl.java:330)
Caused: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
Caused: java.lang.Error
at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:110)
at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
at jenkins.model.Jenkins$7.runTask(Jenkins.java:1073)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2018-05-13 02:21:10.120+0000 [id=24] SEVERE hudson.util.BootFailure#publish: Failed to initialize Jenkins
java.lang.AssertionError: data were not fully read, check your serializer
at org.mapdb.Store.deserialize(Store.java:299)
at org.mapdb.StoreDirect.get2(StoreDirect.java:486)
at org.mapdb.StoreWAL.get2(StoreWAL.java:336)
at org.mapdb.StoreWAL.get(StoreWAL.java:320)
at org.mapdb.EngineWrapper.get(EngineWrapper.java:58)
at org.mapdb.AsyncWriteEngine.get(AsyncWriteEngine.java:333)
at org.mapdb.Caches$HashTable.get(Caches.java:246)
at org.mapdb.EngineWrapper.get(EngineWrapper.java:58)
at org.mapdb.HTreeMap.getInner(HTreeMap.java:448)
at org.mapdb.HTreeMap.getPeek(HTreeMap.java:436)
at org.mapdb.HTreeMap.containsKey(HTreeMap.java:300)
at org.mapdb.HTreeMap$KeySet.contains(HTreeMap.java:817)
at java.util.AbstractCollection.retainAll(AbstractCollection.java:409)
at com.cloudbees.opscenter.client.cloud.PluginImpl$SlaveManifestImpl.createDb(PluginImpl.java:419)
at com.cloudbees.opscenter.client.cloud.PluginImpl$SlaveManifestImpl.<init>(PluginImpl.java:406)
at com.cloudbees.opscenter.client.cloud.PluginImpl.getSlaveManifest(PluginImpl.java:310)
at com.cloudbees.opscenter.client.cloud.PluginImpl.afterExtensionsAugmented(PluginImpl.java:330)
Caused: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
Caused: java.lang.Error
at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:110)
at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
at jenkins.model.Jenkins$7.runTask(Jenkins.java:1073)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused: org.jvnet.hudson.reactor.ReactorException
at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:269)
at jenkins.InitReactorRunner.run(InitReactorRunner.java:47)
at jenkins.model.Jenkins.executeReactor(Jenkins.java:1107)
at jenkins.model.Jenkins.<init>(Jenkins.java:912)
at hudson.model.Hudson.<init>(Hudson.java:86)
at hudson.model.Hudson.<init>(Hudson.java:82)
at hudson.WebAppMain$3.run(WebAppMain.java:235)
Caused: hudson.util.HudsonFailedToLoad
at hudson.WebAppMain$3.run(WebAppMain.java:252)
Environment
Resolution
mapdb
database has become corrupted on the Jenkins client master.
- Stop the client master
-
Delete the following files
- ${JENKINS_HOME}/operations-center-cloud.db
- ${JENKINS_HOME}/operations-center-cloud.db.p
- ${JENKINS_HOME}/operations-center-cloud.db.t
- Restart the client master to see if the problem is resolved:
Note: If the problem persists after deleting those files, the messaging
databases might be corrupted too. Please review this article HA Cluster does not start when messaging database is corrupted for its resolution.
0 Comments