Performance Issues caused by the Beekeeper Report computation

Issue

  • Jenkins is getting slow
  • Many blocked threads trying to compute the Beekeeper report. For example:
"Handling GET / from 127.0.0.1 : RequestHandlerThread[#xxx] View/index.jelly AdministrativeMonitorsDecorator/footer.jelly" [...] waiting for monitor entry [...]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at com.cloudbees.jenkins.plugins.assurance.Cached.get(Cached.java:37)
	- waiting to lock <0x00000006445b5c28> (a com.cloudbees.jenkins.plugins.assurance.CloudBeesAssurance$2)
	at com.cloudbees.jenkins.plugins.assurance.CloudBeesAssurance.getReport(CloudBeesAssurance.java:327)
	at com.cloudbees.jenkins.plugins.assurance.CAPUpdateCenterFilter.getItem(CAPUpdateCenterFilter.java:51)
	at com.cloudbees.jenkins.plugins.assurance.CAPUpdateCenterFilter.getCoreSource(CAPUpdateCenterFilter.java:110)
	at com.cloudbees.jenkins.cjp.installmanager.CJPUpdateCenter.getCoreSource(CJPUpdateCenter.java:72)
	at hudson.model.UpdateCenter$CoreUpdateMonitor.getData(UpdateCenter.java:1002)
	at hudson.model.UpdateCenter$CoreUpdateMonitor.isActivated(UpdateCenter.java:997)
	at jenkins.management.AdministrativeMonitorsDecorator.getActiveAdministrativeMonitors(AdministrativeMonitorsDecorator.java:91)
	at jenkins.management.AdministrativeMonitorsDecorator.getActiveAdministrativeMonitorsCount(AdministrativeMonitorsDecorator.java:76)
	at jenkins.management.AdministrativeMonitorsDecorator.shouldDisplay(AdministrativeMonitorsDecorator.java:142)
    ...
  • Many slows requests blocked by the Beekeeper report computation. For example:
"Handling GET /job/myjob/job/master/ from 127.0.0.1 : RequestHandlerThread[#xxxx] Job/index.jelly AdministrativeMonitorsDecorator/footer.jelly" Id=xxxxxx BLOCKED on com.cloudbees.jenkins.plugins.assurance.CloudBeesAssurance$2@xxxxxx owned by "Handling GET / from 127.0.0.1 : RequestHandlerThread[#xxx] View/index.jelly AdministrativeMonitorsDecorator/footer.jelly" Id=xxxxxx
    at com.cloudbees.jenkins.plugins.assurance.Cached.get(Cached.java:37)
    -  blocked on com.cloudbees.jenkins.plugins.assurance.CloudBeesAssurance$2@7db3c53c
    at com.cloudbees.jenkins.plugins.assurance.CloudBeesAssurance.getReport(CloudBeesAssurance.java:327)
    at com.cloudbees.jenkins.plugins.assurance.EnvelopeWatch.isActivated(EnvelopeWatch.java:43)
    at jenkins.management.AdministrativeMonitorsDecorator.getActiveAdministrativeMonitors(AdministrativeMonitorsDecorator.java:91)
    at jenkins.management.AdministrativeMonitorsDecorator.getActiveAdministrativeMonitorsCount(AdministrativeMonitorsDecorator.java:76)
    at jenkins.management.AdministrativeMonitorsDecorator.shouldDisplay(AdministrativeMonitorsDecorator.java:142)
    at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    ...

Environment

Related Issue(s)

  • CJP-6953 - Several BLOCKED slow-requests while the Beekeeper report is being generated

Explanation

The release of CloudBees Jenkins Platform - Client Master 2.32.2.6 and the version 2.32.0.3 of the CloudBees Assurance Plugin introduced a performance issue when Beekeeper parses the Update Center metadata to refresh the Beekeeper report. This can slow down a Jenkins instance and can lead to serious performance issues.

Resolution

The problem has been fixed in the version 2.46.2.3 of the Cloudbees Assurance Plugin . The recommended solution is to upgrade Jenkins to the latest rolling release.

Workaround

If upgrading Jenkins is not an option and you are suffering from this issue, the workaround would be to disable the administrative monitor com.cloudbees.jenkins.plugins.assurance.EnvelopeWatch in Manage Jenkins > Configure System > Administrative monitors configuration.

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.