How to disable the weather column to resolve instance slowness

Issue

The Jenkins instance experiences an overall slowdown. Slow-request logs from the support bundle should show some stack trace involving WeatherColumn/column.jelly. It should resemble something like this: 10410msec elapsed in Handling GET /master3/ from 127.0.0.0 : RequestHandlerThread[#17] View/index.jelly WeatherColumn/column.jelly

Environment

  • CloudBees Jenkins Enterprise

Resolution

The weather column reports build metrics for each build. Folders display an aggregate “score” of each build and the folders they contain, and display an icon in the weather column to represent this score. When an instance grows to be very large, and it’s folder structure has many levels, the generation of this weather column can seriously impair system performance. Here are some steps to disable these health metrics at the folder level:

  1. Go to all of your top-level folders.
  2. Click the configuration page.
  3. Click Health Metrics.
  4. Delete all of the entries(except Enabled Projects, it may be left if you find it useful or just want some icon in the weather column).
  5. Save the folder configuration.

The following script automates the process:

def folders = Jenkins.instance.getAllItems(com.cloudbees.hudson.plugins.folder.Folder.class)
folders.each{ folder ->
  folder.healthMetrics.each{ metric ->
    if (!(metric instanceof com.cloudbees.hudson.plugins.folder.health.ProjectEnabledHealthMetric)) {
      println "Removing ${metric.class.simpleName} from ${folder.name}"
      folder.healthMetrics.remove(metric)
      folder.save()
    }
  }
}
return null

The script can be executed from Jenkins Enterprise’s Script Console or as a Cluster Operation and run across multiple client masters from Jenkins Operations Center. You could even schedule the Cluster Operation to run on a regular basis. Of course, you should run this script on a test master before running across your production masters.

This will stop the weather column from constantly refreshing on all folders and the jobs within them.

Note that running this script will disable the Weather Column for existing projects/jobs only, but not for newly created projects in the future. It may be ideal to run this periodically, or perhaps as a scheduled job.

Have more questions? Submit a request

2 Comments

  • 0
    Avatar
    Harald Göttlicher

    Hi, thanks for the hint, this really helps.

    If you only want to remove the health metrics from top level folder, you can add

      if (! (folder.getParent() instanceof com.cloudbees.hudson.plugins.folder.Folder) ) {

    So i.e. the script then is

    def folders = Jenkins.instance.getAllItems(com.cloudbees.hudson.plugins.folder.Folder.class)
    folders.each{ folder ->
      if (! (folder.getParent() instanceof com.cloudbees.hudson.plugins.folder.Folder) ) {
        folder.healthMetrics.each{ metric ->
          if (!(metric instanceof com.cloudbees.hudson.plugins.folder.health.ProjectEnabledHealthMetric)) {
            println "Removing ${metric.class.simpleName} from ${folder.name}"
            folder.healthMetrics.remove(metric)
            folder.save()
          }
        }
      }
    }
    return null

    Similarly, we could check for a certain folder depth level.

     

  • 0
    Avatar
    Harald Göttlicher

    Here's my improved script with

    • folder depth level option
    • dry run

    Have fun!

    // set max folder depth to fix - e.g. 2 will fix root and one subfolder level
    def maxDepth = 2
    // try dry run first, set false to apply changes
    def dryRun = true

    def folders = Jenkins.instance.getAllItems(com.cloudbees.hudson.plugins.folder.Folder.class)
    folders.each{ folder ->
      // check if folder is deeper than max depth
      def depthReached = false
      def parent = folder.getParent()
      for (i = 1; i <= maxDepth; i++) {
        if (!(parent instanceof com.cloudbees.hudson.plugins.folder.Folder)) {
          // top level reached
          break
        }
        if (i==maxDepth) {
          // level > max depth, do not consider this folder
          depthReached = true
        }
        else {
          parent = parent.getParent()
        }
      }

      if (!depthReached) {
        println "Checking folder $folder.name..."
        folder.healthMetrics.each{ metric ->
          if (!(metric instanceof com.cloudbees.hudson.plugins.folder.health.ProjectEnabledHealthMetric)) {
            println "- ${dryRun?'Would remove':'Removing'} ${metric.class.simpleName} from ${folder.name}"
            if (!dryRun) {
              folder.healthMetrics.remove(metric)
              folder.save()
            }
          }
        }
      }
    }
    return null
Please sign in to leave a comment.