How to disable the weather column to resolve instance slowness?

Issue

The Jenkins instance experiences an overall slowdown. Slow-request logs from the support bundle should show some stack trace involving WeatherColumn/column.jelly. It should resemble something like this: 10410msec elapsed in Handling GET /master3/ from 127.0.0.0 : RequestHandlerThread[#17] View/index.jelly WeatherColumn/column.jelly

Environment

  • CloudBees Jenkins Enterprise

Resolution

The weather column reports build metrics for each build. Folders display an aggregate “score” of each build and the folders they contain, and display an icon in the weather column to represent this score. When an instance grows to be very large, and it’s folder structure has many levels, the generation of this weather column can seriously impair system performance. Here are some steps to disable these health metrics at the folder level:

  1. Go to all of your top-level folders.
  2. Click the configuration page.
  3. Click Health Metrics.
  4. Delete all of the entries(except Enabled Projects, it may be left if you find it useful or just want some icon in the weather column).
  5. Save the folder configuration.

The following script automates the process:

def folders = Jenkins.instance.getAllItems(com.cloudbees.hudson.plugins.folder.Folder.class)
folders.each{ folder ->
  folder.healthMetrics.each{ metric ->
    if (!(metric instanceof com.cloudbees.hudson.plugins.folder.health.ProjectEnabledHealthMetric)) {
      println "Removing ${metric.class.simpleName} from ${folder.name}"
      folder.healthMetrics.remove(metric)
      folder.save()
    }
  }
}
return null

The script can be executed from Jenkins Enterprise’s Script Console or as a Cluster Operation and run across multiple client masters from Jenkins Operations Center. You could even schedule the Cluster Operation to run on a regular basis. Of course, you should run this script on a test master before running across your production masters.

This will stop the weather column from constantly refreshing on all folders and the jobs within them.

Note that running this script will disable the Weather Column for existing projects/jobs only, but not for newly created projects in the future. It may be ideal to run this periodically, or perhaps as a scheduled job.

Have more questions? Submit a request

7 Comments

  • 0
    Avatar
    Harald Göttlicher

    Hi, thanks for the hint, this really helps.

    If you only want to remove the health metrics from top level folder, you can add

      if (! (folder.getParent() instanceof com.cloudbees.hudson.plugins.folder.Folder) ) {

    So i.e. the script then is

    def folders = Jenkins.instance.getAllItems(com.cloudbees.hudson.plugins.folder.Folder.class)
    folders.each{ folder ->
      if (! (folder.getParent() instanceof com.cloudbees.hudson.plugins.folder.Folder) ) {
        folder.healthMetrics.each{ metric ->
          if (!(metric instanceof com.cloudbees.hudson.plugins.folder.health.ProjectEnabledHealthMetric)) {
            println "Removing ${metric.class.simpleName} from ${folder.name}"
            folder.healthMetrics.remove(metric)
            folder.save()
          }
        }
      }
    }
    return null

    Similarly, we could check for a certain folder depth level.

     

  • 0
    Avatar
    Harald Göttlicher

    Here's my improved script with

    • folder depth level option
    • dry run

    Have fun!

    // set max folder depth to fix - e.g. 2 will fix root and one subfolder level
    def maxDepth = 2
    // try dry run first, set false to apply changes
    def dryRun = true

    def folders = Jenkins.instance.getAllItems(com.cloudbees.hudson.plugins.folder.Folder.class)
    folders.each{ folder ->
      // check if folder is deeper than max depth
      def depthReached = false
      def parent = folder.getParent()
      for (i = 1; i <= maxDepth; i++) {
        if (!(parent instanceof com.cloudbees.hudson.plugins.folder.Folder)) {
          // top level reached
          break
        }
        if (i==maxDepth) {
          // level > max depth, do not consider this folder
          depthReached = true
        }
        else {
          parent = parent.getParent()
        }
      }

      if (!depthReached) {
        println "Checking folder $folder.name..."
        folder.healthMetrics.each{ metric ->
          if (!(metric instanceof com.cloudbees.hudson.plugins.folder.health.ProjectEnabledHealthMetric)) {
            println "- ${dryRun?'Would remove':'Removing'} ${metric.class.simpleName} from ${folder.name}"
            if (!dryRun) {
              folder.healthMetrics.remove(metric)
              folder.save()
            }
          }
        }
      }
    }
    return null
  • 0
    Avatar
    Chris-lee Chris-lee

    How would you do this in non-enterprise Jenkins? I was going to go rooting around to see if I could find the non-enterprise classes to replace, but that kinda scares me.

  • 0
    Avatar
    Allan Burdajewicz

    Hi Chris-lee, 

    The script only requires that you have the CloudBees Folder plugin installed and this plugin is open source. It should therefore work on Jenkins LTS.

     

  • 0
    Avatar
    Chris-lee Chris-lee

    As written the script executed gives me this:

     

    org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
    Script1.groovy: 4: unable to resolve class com.cloudbees.hudson.plugins.folder.health.ProjectEnabledHealthMetric 
     @ line 4, column 29.
           if (!(metric instanceof com.cloudbees.hudson.plugins.folder.health.ProjectEnabledHealthMetric))

    I presumed it was because I didn't have the cloudbees enterprise version. 

     

    I looked for the source and found "https://github.com/jenkinsci/cloudbees-folder-plugin/blob/master/src/main/java/com/cloudbees/hudson/plugins/folder/health/WorstChildHealthMetric.java"

    Which seems like it might be a renamed version of the same thing, and that was detected by the script but I don't know if that's the right thing to remove. When I remove it the column with the weather remains.

  • 0
    Avatar
    Denys Digtiar
    com.cloudbees.hudson.plugins.folder.health.ProjectEnabledHealthMetric

    On a second look, this class is indeed defined in the CloudBees proprietary plugin. What you can try is to remove this condition completely and let script remove all the Health Metrics.

  • 0
    Avatar
    Owen Wood

    To rollback:

    import com.cloudbees.hudson.plugins.folder.Folder;
    import com.cloudbees.hudson.plugins.folder.health.*;
    
    Jenkins.instance.getAllItems(Folder.class).each {
        it.healthMetrics.add(new WorstChildHealthMetric())
        it.healthMetrics.add(new AverageChildHealthMetric())
        it.healthMetrics.add(new ProjectEnabledHealthMetric())
        // cloudbees-folders-plus-plugin (this one is proprietary)
        it.healthMetrics.add(new JobStatusHealthMetric(true,true,true,true))
    
        it.save()
    }
Please sign in to leave a comment.