How to disable the weather column to resolve instance slowness?

Issue

The Jenkins instance experiences an overall slowdown. Slow-request logs from the support bundle should show some stack trace involving WeatherColumn/column.jelly. It should resemble something like this: 10410msec elapsed in Handling GET /master3/ from 127.0.0.0 : RequestHandlerThread[#17] View/index.jelly WeatherColumn/column.jelly

Environment

Explanation

The weather column is a Column that can be used in Jenkins List views to displays a weather icon that represents the health of the item listed. The health is based on an aggregated score calculated mainly from build metrics.

  • In the case of a Job, the health is calculated from the job build history.
  • In the case of a Folder, the health is calculated recursively following several configurable Health Metrics.

When an instance grows to be very large, and it’s folder structure has many levels, the generation of this weather column can cost a lot of system resources slowing down other processes.

By default in Jenkins:

  • the Jenkins “All” views display all the possible column for a List View, and consequently the Weather column
  • when a Folder is created, an “All” view is created and all the possible folder health metrics are added

Therefore by default, the weather column reports the health of all items.

Resolution

If an instance is impacted by the performance of the Weather Column, a solution is to remove the folder health metrics of all existing folders.

The only caveat of such a change is that the weather column will always report Folders as Healthy.

Existing Folders

Here are some steps to disable these health metrics at the folder level:

  1. Go to all of your top-level folders.
  2. Click the configuration page.
  3. Click Health Metrics.
  4. Delete all of the entries (except Enabled Projects, it may be left if you find it useful or just want some icon in the weather column).
  5. Save the folder configuration.

The following script automates the process:

// try dry run first, set false to apply changes
def dryRun = true

final metricToExclude = "com.cloudbees.hudson.plugins.folder.health.ProjectEnabledHealthMetric"
final verb = dryRun ? "Would remove" : "Removing"
Jenkins.instance.allItems(com.cloudbees.hudson.plugins.folder.AbstractFolder).each { folder ->
    def removed = folder.healthMetrics.removeIf { metric ->
        if (metric.class.name.equals(metricToExclude)) {
            return false
        }
        println "${verb} ${metric.class.simpleName} from ${folder.name}"
        return !dryRun
    }
    if (removed) {
        folder.save()
    }
}
return null

By default this script runs in a dry run mode, i.e., it only prints what it is going to modify. To modify the health metrics change the dryRun variable from true to false.

This will stop the weather column from constantly refreshing on all folders and the jobs within them.

Note that running this script will disable the Weather Column for existing folders only, but not for newly created folders in the future.

Newly Created Folders

Folder Plugin 6.10.1 or later

Since the release of the Folder Plugin 6.10.1, it is possible to change the folder health metrics added by default when creating a Folder.

To remove all default folder health metrics, go to Manage Jenkins > Configure System > folder and remove all the Health Metric controls like the following, then save:

This guarantee that all newly created folders do not have those folder health metrics so the weather column will not report their health.

Before Folder Plugin 6.10.1

There is no simple solution other than running periodically the script provided above that addressed existing folders.

The script may well be executed from Jenkins Script Console or as a scheduled job.

The script could even be used as a Cluster Operation and run across multiple connected masters from the Operations Center. And this Cluster Operation could be run on a regular basis.

Of course, this script should be run on a test master before running across production masters.

Have more questions?

13 Comments

  • 0
    Avatar
    Harald Göttlicher

    Hi, thanks for the hint, this really helps.

    If you only want to remove the health metrics from top level folder, you can add

      if (! (folder.getParent() instanceof com.cloudbees.hudson.plugins.folder.Folder) ) {

    So i.e. the script then is

    def folders = Jenkins.instance.getAllItems(com.cloudbees.hudson.plugins.folder.Folder.class)
    folders.each{ folder ->
      if (! (folder.getParent() instanceof com.cloudbees.hudson.plugins.folder.Folder) ) {
        folder.healthMetrics.each{ metric ->
          if (!(metric instanceof com.cloudbees.hudson.plugins.folder.health.ProjectEnabledHealthMetric)) {
            println "Removing ${metric.class.simpleName} from ${folder.name}"
            folder.healthMetrics.remove(metric)
            folder.save()
          }
        }
      }
    }
    return null

    Similarly, we could check for a certain folder depth level.

     

  • 0
    Avatar
    Harald Göttlicher

    Here's my improved script with

    • folder depth level option
    • dry run

    Have fun!

    // set max folder depth to fix - e.g. 2 will fix root and one subfolder level
    def maxDepth = 2
    // try dry run first, set false to apply changes
    def dryRun = true

    def folders = Jenkins.instance.getAllItems(com.cloudbees.hudson.plugins.folder.Folder.class)
    folders.each{ folder ->
      // check if folder is deeper than max depth
      def depthReached = false
      def parent = folder.getParent()
      for (i = 1; i <= maxDepth; i++) {
        if (!(parent instanceof com.cloudbees.hudson.plugins.folder.Folder)) {
          // top level reached
          break
        }
        if (i==maxDepth) {
          // level > max depth, do not consider this folder
          depthReached = true
        }
        else {
          parent = parent.getParent()
        }
      }

      if (!depthReached) {
        println "Checking folder $folder.name..."
        folder.healthMetrics.each{ metric ->
          if (!(metric instanceof com.cloudbees.hudson.plugins.folder.health.ProjectEnabledHealthMetric)) {
            println "- ${dryRun?'Would remove':'Removing'} ${metric.class.simpleName} from ${folder.name}"
            if (!dryRun) {
              folder.healthMetrics.remove(metric)
              folder.save()
            }
          }
        }
      }
    }
    return null
  • 0
    Avatar
    Chris-lee Chris-lee

    How would you do this in non-enterprise Jenkins? I was going to go rooting around to see if I could find the non-enterprise classes to replace, but that kinda scares me.

  • 0
    Avatar
    Allan Burdajewicz

    Hi Chris-lee, 

    The script only requires that you have the CloudBees Folder plugin installed and this plugin is open source. It should therefore work on Jenkins LTS.

     

  • 0
    Avatar
    Chris-lee Chris-lee

    As written the script executed gives me this:

     

    org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
    Script1.groovy: 4: unable to resolve class com.cloudbees.hudson.plugins.folder.health.ProjectEnabledHealthMetric 
     @ line 4, column 29.
           if (!(metric instanceof com.cloudbees.hudson.plugins.folder.health.ProjectEnabledHealthMetric))

    I presumed it was because I didn't have the cloudbees enterprise version. 

     

    I looked for the source and found "https://github.com/jenkinsci/cloudbees-folder-plugin/blob/master/src/main/java/com/cloudbees/hudson/plugins/folder/health/WorstChildHealthMetric.java"

    Which seems like it might be a renamed version of the same thing, and that was detected by the script but I don't know if that's the right thing to remove. When I remove it the column with the weather remains.

  • 0
    Avatar
    Denys Digtiar
    com.cloudbees.hudson.plugins.folder.health.ProjectEnabledHealthMetric

    On a second look, this class is indeed defined in the CloudBees proprietary plugin. What you can try is to remove this condition completely and let script remove all the Health Metrics.

  • 0
    Avatar
    Owen Wood

    To rollback:

    import com.cloudbees.hudson.plugins.folder.Folder;
    import com.cloudbees.hudson.plugins.folder.health.*;
    
    Jenkins.instance.getAllItems(Folder.class).each {
        it.healthMetrics.add(new WorstChildHealthMetric())
        it.healthMetrics.add(new AverageChildHealthMetric())
        it.healthMetrics.add(new ProjectEnabledHealthMetric())
        // cloudbees-folders-plus-plugin (this one is proprietary)
        it.healthMetrics.add(new JobStatusHealthMetric(true,true,true,true))
    
        it.save()
    }
  • 0
    Avatar
    Junyi Sun

    Hello, I followed the instruction to go to a folder configuration page and I don't see anything under Health metrics.

    But I am still seeing the weather column (see the attachment). Any ideas why?

    Thanks

    Junyi

  • 0
    Avatar
    Owen Wood

    Hi Junyi,

    This will stop the weather column from constantly refreshing on all folders and the jobs within them.

    It does not remove the icon.

    I hope this makes sense.

  • 0
    Avatar
    Joe Miller

    I don't see a clear answer on how to disable this for non-enterprise installations.  I have the Cloudbees Folder plugin installed and that doesn't appear to have the classes detailed here. Thanks!

  • 0
    Avatar
    Chen Fliesher

    I have an LTS versiob of jenkins with cloudbees-folder installed.

    I am using it on a Github Folder (Github Branch source Plugin)

    The problem that

    Jenkins.instance.getAllItems(com.cloudbees.hudson.plugins.folder.Folder.class)
    doesn't return any folders.

    I tried to drilled down the Github branch source plugin, but couldn't find any relation to cloudbees folder.

    Anyone managed to pull this on a Github organation folder ?
  • 0
    Avatar
    Denys Digtiar

    Joe, Chen

    The updated script should cover both of your use cases. 

  • 0
    Avatar
    Steven Christenson

    We have 10 Jenkins masters ranging from
    CloudBees Jenkins Enterprise 2.176.4.3-rolling (the latest)
    to 
    CloudBees Jenkins Enterprise 2.164.2.1-rolling (an old soon to be retired master)

    Unfortunately the script supplied above did not remove ANY of the weather columns on any of our masters. I see above where it explains it stops the updating, but what it the point of having a Weather column that is out of date?

    Secondarily, it would be very helpful if we could configure the "default" All view that folders are created with to: 

    Not look like this:


    Put the build button leftmost where it is easier to find.
    Not include Weather
    Not include "Favorites"
    Not include "Built On"
    Not include "# Issues"

    And finally, I'd like to understand the impact that the "Last (success/failure/duration)" times have on Jenkins master loads. If those are lightweight, I'd keep them, otherwise we need to lose them unless explicitly added to a view.


     

    Edited by Steven Christenson
Please sign in to leave a comment.