Lock Contention in RBAC PersistenceHelper.getOrCreateGroupContainer

Issue

  • Jenkins job stops building, Jenkins becomes unresponsive and thread dumps shows a lot of BLOCKED threads like the following:

    "AtmostOneTaskExecutor[Periodic Jenkins queue maintenance] [#000000]" #00000000 daemon prio=5 os_prio=0 tid=0x0000000000000000 nid=0x000d waiting for monitor entry [0x0000000000000000]
    java.lang.Thread.State: BLOCKED (on object monitor)
        at nectar.plugins.rbac.groups.GroupContainerLocator$PersistenceHelper.getOrCreateGroupContainer(GroupContainerLocator.java:202)
        - waiting to lock <0x00000000000000001> (a com.cloudbees.hudson.plugins.folder.properties.FolderProxyGroupContainer$GroupContainerLocatorImpl$1)
        at com.cloudbees.hudson.plugins.folder.properties.FolderProxyGroupContainer$GroupContainerLocatorImpl.get(FolderProxyGroupContainer.java:224)
        at nectar.plugins.rbac.groups.GroupContainerLocator.locate(GroupContainerLocator.java:118)
        at nectar.plugins.rbac.strategy.RoleMatrixAuthorizationStrategyImpl.getACL(RoleMatrixAuthorizationStrategyImpl.java:180)
        at hudson.model.AbstractItem.getACL(AbstractItem.java:601)
        [...]
    

Environment

Related Issue(s)

  • CTR-2641: thread contention on PersistenceHelper.getOrCreateGroupContainer
  • CTR-430: Fix nectar-rbac with doConfigDotXml()

Explanation

This is a known issue in the CloudBees Role-Based Access Controller plugin where the PersistenceHelper.getOrCreateGroupContainer method is synchronized causing a single point of contention for all lookups. In very active instances, this can cause severe performance issues, block the queue and make the Jenkins controller unresponsive.

Workaround

There is no other workaround than restarting the Jenkins controller.

Resolution

Upgrade CloudBees CI to version 2.249.3.1 or later.

Have more questions?

0 Comments

Please sign in to leave a comment.