Issue
The issue can manifest in at least two different ways:
- Client Master fails to start, or
- Client Master fails to lease shared agent form Operations Center
In both cases, the messages with a stack trace similar to the following should be present in the Client Master logs:
016-01-06 00:00:56.688-0600 [id=96366] WARNING c.c.o.s.m.ConnectedMaster$RemoteSlaveManifest#getSlaves: Could not complete remote call
java.lang.RuntimeException: Writer thread failed
at org.mapdb.AsyncWriteEngine.checkState(AsyncWriteEngine.java:262)
at org.mapdb.AsyncWriteEngine.get(AsyncWriteEngine.java:326)
at org.mapdb.Caches$HashTable.get(Caches.java:245)
at org.mapdb.HTreeMap$HashIterator.findNextLinkedNodeRecur(HTreeMap.java:1081)
at org.mapdb.HTreeMap$HashIterator.findNextLinkedNode(HTreeMap.java:1046)
at org.mapdb.HTreeMap$HashIterator.advance(HTreeMap.java:1033)
at org.mapdb.HTreeMap$HashIterator.moveToNext(HTreeMap.java:992)
at org.mapdb.HTreeMap$EntryIterator.next(HTreeMap.java:1139)
at org.mapdb.HTreeMap$EntryIterator.next(HTreeMap.java:1132)
at com.cloudbees.opscenter.client.cloud.PluginImpl$SlaveManifestImpl.getSlaves(PluginImpl.java:422)
at com.cloudbees.opscenter.server.model.ConnectedMaster$GetRemoteLeases.call(ConnectedMaster.java:2086)
at com.cloudbees.opscenter.server.model.ConnectedMaster$GetRemoteLeases.call(ConnectedMaster.java:2083)
at hudson.remoting.UserRequest.perform(UserRequest.java:120)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at org.jenkinsci.remoting.CallableDecorator.call(CallableDecorator.java:18)
at hudson.remoting.CallableDecoratorList$1.call(CallableDecoratorList.java:21)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at ......remote call to 1-jenkins2 at https://<JENKINS_URL>.com/(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
at hudson.remoting.Channel.call(Channel.java:781)
at com.cloudbees.opscenter.server.model.ConnectedMaster$RemoteSlaveManifest.getSlaves(ConnectedMaster.java:2051)
at com.cloudbees.opscenter.server.model.SharedSlave$PeriodicTidyUp$1.call(SharedSlave.java:865)
at com.cloudbees.opscenter.server.model.SharedSlave$PeriodicTidyUp$1.call(SharedSlave.java:858)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 404
at org.mapdb.StoreWAL.longStackPutSixLong(StoreWAL.java:1060)
at org.mapdb.StoreWAL.longStackPut(StoreWAL.java:1040)
at org.mapdb.StoreDirect.freePhysPut(StoreDirect.java:1055)
at org.mapdb.StoreDirect.freePhysTake(StoreDirect.java:1081)
at org.mapdb.StoreDirect.physAllocate(StoreDirect.java:655)
at org.mapdb.StoreWAL.update(StoreWAL.java:436)
at org.mapdb.EngineWrapper.update(EngineWrapper.java:63)
at org.mapdb.AsyncWriteEngine.access$101(AsyncWriteEngine.java:76)
at org.mapdb.AsyncWriteEngine.runWriter(AsyncWriteEngine.java:220)
at org.mapdb.AsyncWriteEngine$WriterRunnable.run(AsyncWriteEngine.java:172)
... 1 more
Environment
- CloudBees Jenkins Enterprise - Managed Master (CJE-MM)
- CloudBees Jenkins Enterprise - Operations Center (CJE-OC)
- CloudBees Jenkins Platform - Client Master (CJP-CM)
- CloudBees Jenkins Platform - Operations Center (CJP-OC)
Resolution
mapdb
database has become corrupted on the Jenkins client master.
Stop the client master and delete the following files and then restart the client master to see if the problem is resolved:
- ${JENKINS_HOME}/operations-center-cloud.db
- ${JENKINS_HOME}/operations-center-cloud.db.p
- ${JENKINS_HOME}/operations-center-cloud.db.t
0 Comments