Global Build Discarders causes exceptions after Upgrading to 2.222.x

Issue

  • After upgrading CloudBees Core to 2.222.x, the Jenkins logs shows a lot of exception like the following:

    Also: java.nio.file.NoSuchFileException: <buildPath> -> <buildPath>
      at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
      at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
      at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:396)
      at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
      at java.nio.file.Files.move(Files.java:1395)
      at hudson.model.Run.delete(Run.java:1621)
      at hudson.tasks.LogRotator.perform(LogRotator.java:166)
    jenkins.util.io.CompositeIOException: Failed to rotate logs for [<buildDisplayName>]
      at hudson.tasks.LogRotator.perform(LogRotator.java:223)
      at hudson.model.Job.logRotate(Job.java:469)
      at jenkins.model.JobGlobalBuildDiscarderStrategy.apply(JobGlobalBuildDiscarderStrategy.java:54)
      at jenkins.model.BackgroundGlobalBuildDiscarder.lambda$processJob$0(BackgroundGlobalBuildDiscarder.java:67)
      at java.lang.Iterable.forEach(Iterable.java:75)
      at jenkins.model.BackgroundGlobalBuildDiscarder.processJob(BackgroundGlobalBuildDiscarder.java:61)
      at jenkins.model.GlobalBuildDiscarderListener.onFinalized(GlobalBuildDiscarderListener.java:49)
      at hudson.model.listeners.RunListener.fireFinalized(RunListener.java:255)
      at hudson.model.Run.onEndBuilding(Run.java:2018)
      [...]
    

Environment

Related Issues

Explanation

Jenkins Core 2.222 introduce Global Build Discarders as mentioned in the release notes. Those strategies apply the discard build policy of jobs periodically every hour but also whenever a build completes - like the job build discarders do.

Whenever a build finish, the build discarder may be triggered twice for the same job and is subjected to a race condition in the build deletion logic. The race condition is captured by JENKINS-61687. Jenkins might try to delete the same build twice causing the exception shown above.

In most cases, the impact is minor. However if this happens very frequently - like in large and active environments - it can spam the Jenkins logs and may well cause performance issues on the master.

Resolution

Until JENKINS-61687 is fixed, the workaround is to delete the Global Build Discarder Policies.

This can be done from the UI:

  • Go to Manage Jenkins > Configure System
  • Navigate to “Global Build Discarders”
  • Delete any of the “Project Build Discarder” or “Specific Build Discarder” that currently exist
  • Save

This can also be done via a Groovy script like the following:

jenkins.model.GlobalBuildDiscarderConfiguration.get().configuredBuildDiscarders.clear()
jenkins.model.GlobalBuildDiscarderConfiguration.get().save()

Tested product/plugin versions

The latest update of this article has been tested and verified with:

Have more questions?

0 Comments

Please sign in to leave a comment.