CJOC doesn't lease shared slave to a client master because of org.mapdb.AsyncWriteEngine.checkState

Issue

CJOC doesn’t lease shared slaves to a client master and the following stacktrace is repeated on the Client master instance:

016-01-06 00:00:56.688-0600 [id=96366] WARNING c.c.o.s.m.ConnectedMaster$RemoteSlaveManifest#getSlaves: Could not complete remote call
java.lang.RuntimeException: Writer thread failed
    at org.mapdb.AsyncWriteEngine.checkState(AsyncWriteEngine.java:262)
    at org.mapdb.AsyncWriteEngine.get(AsyncWriteEngine.java:326)
    at org.mapdb.Caches$HashTable.get(Caches.java:245)
    at org.mapdb.HTreeMap$HashIterator.findNextLinkedNodeRecur(HTreeMap.java:1081)
    at org.mapdb.HTreeMap$HashIterator.findNextLinkedNode(HTreeMap.java:1046)
    at org.mapdb.HTreeMap$HashIterator.advance(HTreeMap.java:1033)
    at org.mapdb.HTreeMap$HashIterator.moveToNext(HTreeMap.java:992)
    at org.mapdb.HTreeMap$EntryIterator.next(HTreeMap.java:1139)
    at org.mapdb.HTreeMap$EntryIterator.next(HTreeMap.java:1132)
    at com.cloudbees.opscenter.client.cloud.PluginImpl$SlaveManifestImpl.getSlaves(PluginImpl.java:422)
    at com.cloudbees.opscenter.server.model.ConnectedMaster$GetRemoteLeases.call(ConnectedMaster.java:2086)
    at com.cloudbees.opscenter.server.model.ConnectedMaster$GetRemoteLeases.call(ConnectedMaster.java:2083)
    at hudson.remoting.UserRequest.perform(UserRequest.java:120)
    at hudson.remoting.UserRequest.perform(UserRequest.java:48)
    at hudson.remoting.Request$2.run(Request.java:326)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
    at org.jenkinsci.remoting.CallableDecorator.call(CallableDecorator.java:18)
    at hudson.remoting.CallableDecoratorList$1.call(CallableDecoratorList.java:21)
    at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
    at ......remote call to 1-jenkins2 at https://<JENKINS_URL>.com/(Native Method)
    at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
    at hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
    at hudson.remoting.Channel.call(Channel.java:781)
    at com.cloudbees.opscenter.server.model.ConnectedMaster$RemoteSlaveManifest.getSlaves(ConnectedMaster.java:2051)
    at com.cloudbees.opscenter.server.model.SharedSlave$PeriodicTidyUp$1.call(SharedSlave.java:865)
    at com.cloudbees.opscenter.server.model.SharedSlave$PeriodicTidyUp$1.call(SharedSlave.java:858)
    at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 404
    at org.mapdb.StoreWAL.longStackPutSixLong(StoreWAL.java:1060)
    at org.mapdb.StoreWAL.longStackPut(StoreWAL.java:1040)
    at org.mapdb.StoreDirect.freePhysPut(StoreDirect.java:1055)
    at org.mapdb.StoreDirect.freePhysTake(StoreDirect.java:1081)
    at org.mapdb.StoreDirect.physAllocate(StoreDirect.java:655)
    at org.mapdb.StoreWAL.update(StoreWAL.java:436)
    at org.mapdb.EngineWrapper.update(EngineWrapper.java:63)
    at org.mapdb.AsyncWriteEngine.access$101(AsyncWriteEngine.java:76)
    at org.mapdb.AsyncWriteEngine.runWriter(AsyncWriteEngine.java:220)
    at org.mapdb.AsyncWriteEngine$WriterRunnable.run(AsyncWriteEngine.java:172)
    ... 1 more

Environment

  • CloudBees Jenkins Operations Center

Resolution

mapdb database has become corrupted on the Jenkins client master.

Stop the client master and delete the following files and then restart the client master to see if the problem is resolved:

  • ${JENKINS_HOME}/operations-center-cloud.db
  • ${JENKINS_HOME}/operations-center-cloud.db.p
  • ${JENKINS_HOME}/operations-center-cloud.db.t
Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.