Best Strategy for Disk Space Management: Clean Up Old Builds

Issue

  • Jenkins builds can pile up and cause disk space to grow out of control.
  • Jenkins fingerprints can pile up and make your instance run out of inodes (linux).
  • Performance issues can result in large/complex instances with large number of projects building frequently, such as slowdown of the GUI as it attempts to load the job history, etc.
  • Configuring build/fingerprint cleanup following project setup can be manual and time-consuming process, especially with large number of projects.

Environment

Resolution

Note: Builds and fingerprints are correlated. Jenkins maintains a database of fingerprints, Jenkins records which builds of which projects used. This database is updated every time a build runs and files are fingerprinted.

There are a number of ways to manage build cleanup:

1) By default, you can enable the “Discard Old Builds” in each project/job’s configuration page.
This will perform basic cleanup, using functionality found in Jenkins core. To make this bulk change to large number of projects
(if you have large number of projects and it’s not feasible to do this individually to each one), please install/use the
Configuration Slicing Plugin

2) You can install/use the Discard Old Build plugin
to do the cleanup. This will extend the basic Jenkins “Discard Old Builds” functionality in each build/project’s configuration,
providing additional options.

3) You can update the configuration of all existing jobs on a Jenkins Master by running the following Groovy Script within Manage Jenkins -> Script Console which will apply a permanent build discard policy to your jobs that you can configure by passing the desired values to the listed parameters.

// NOTES:
// dryRun: to only list the jobs which would be changed
// daysToKeep:  If not -1, history is only kept up to this days.
// numToKeep: If not -1, only this number of build logs are kept.
// artifactDaysToKeep: If not -1 nor null, artifacts are only kept up to this days.
// artifactNumToKeep: If not -1 nor null, only this number of builds have their artifacts kept.

import jenkins.model.Jenkins
import hudson.model.Job
import jenkins.model.BuildDiscarderProperty
import hudson.tasks.LogRotator

Jenkins.instance.allItems(Job).each { job ->
    if (job.isBuildable() && job.supportsLogRotator() && job.getProperty(BuildDiscarderProperty) == null) {
        println "Processing \"${job.fullDisplayName}\""
        if (!"true".equals(dryRun)) {
            // adding a property implicitly saves so no explicit one
            job.addProperty(new BuildDiscarderProperty(new LogRotator ( daysToKeep, numToKeep, artifactDaysToKeep, artifactNumToKeep)))
            println "$job.name is updated"
        }
    }
}
return;

4) If you are using Templates then it is possible to add a BuildDiscarderProperty like the example below to the template configuration which will add your custom discard policy to each job already created from the template and any future jobs created using the template as well.

<jenkins.model.BuildDiscarderProperty>
  <strategy class="hudson.tasks.LogRotator">
    <daysToKeep>-1</daysToKeep>
    <numToKeep>-1</numToKeep>
    <artifactDaysToKeep>-1</artifactDaysToKeep>
    <artifactNumToKeep>-1</artifactNumToKeep>
  </strategy>
</jenkins.model.BuildDiscarderProperty>

5) If you have large number of existing builds and it’s not feasible to manually cleanup builds using one of the options above, it’s possible to perform manual cleanup of jobs from the filesystem (in $JENKINS_HOME/jobs). Navigate to $JENKINS_HOME/jobs/<jobname>/builds
folder for each project, and delete some or all of the files inside here (which comprise the build history). Following the deletion, it
will be necessary to restart Jenkins or “Reload Configuration from Disk” (both in “Manage Jenkins”) so the links to the deleted
builds will disappear.
Following this operation, install/use the Configuration Slicing Plugin
mentioned in step #1 so you don’t need to perform this bulk deletion again in the future.

 It's also possible to purge selectively from the filesystem using "find $dir -mtime +$age -rm" where age is the oldest date
 you want to keep the build (in days). This will get rid of the builds and the links. Again, you may have to restart or reload
 once this is done in order for the GUI to show that the builds are no longer available.  Be sure to backup your $JENKINS_HOME
 (or at least the `jobs` folder) prior to doing this...just in case of unintentional deletes.

WARNING: It is possible using the following Groovy Script to delete more builds than intended, these may not be recoverable.

6) You can also use a Groovy Script to automatically cleanup builds of specific jobs or folders if you can access Manage Jenkins -> Script Console then you can copy the following Groovy script and customize the properties provided within it as necessary:
Automated Groovy Build Clean Script (USE WITH CAUTION)

Have more questions? Submit a request

1 Comments

Please sign in to leave a comment.