Best Practices for Jenkins Updates, Patches and Maintenance

Environment

• CloudBees Jenkins Enterprise - Managed controller (CJEMM)
• CloudBees Jenkins Enterprise - Operations Center (CJEOC)
• CloudBees Jenkins Team (CJT)
• CloudBees Jenkins Platform - Client controller (CJPCM)
• CloudBees Jenkins Platform - Operations Center (CJPOC)
• Jenkins LTS

Issue

• We need to restart Jenkins on a regular basis in order to upgrade Jenkins and its plugins.
• We need to patch Linux on a regular basis. Since our Jenkins is installed on Linux, we often need to restart Jenkins that has jobs running.
• How can I make the upgrade/restart process as seamless as possible?

Resolution

safeRestart

The safeRestart function tells Jenkins what to do before restarting. It is built in Jenkins core (just navigate to /safeRestart); this will prevent any new builds from starting and will allow existing builds to complete before Jenkins restarts.

Quiet Start

The Quiet Start function tells Jenkins what to do after restarting. Normally when Jenkins is restarted, it immediately starts trying to schedule jobs to run. If you are taking the server down for maintenance, you may not want everything to start running immediately. The Quiet Start plugin adds the ability to put Jenkins in the quiet mode. Go to Manage Jenkins » Quiet Restart and just check the box you see there. When the server is restarted, it will be in the “quieting down” state. An administrator can cancel that state using the regular UI.
You may also wish to uncheck this box after maintenance is complete, so that after subsequent restarts jobs will start running.

Restarting from a Pipeline CheckPoint

All Pipelines are resumable: if Jenkins needs to be restarted (or crashes, or the server reboots) while a flow is running, it should resume at the same point in its program after Jenkins starts back up. Similarly, if a flow is running a lengthy sh or bat step when an agent unexpectedly disconnects, no progress should be lost when connectivity is restored.
However, in some cases a Pipeline will have done a great deal of work and proceeded to a point where a transient error occurred: one which does not reflect the inputs to this build, such as source code changes. For example, after completing a lengthy build and test of a software component, final deployment to a server might fail for a silly reason, such as a DNS error or low disk space. After correcting the problem you might prefer to restart just the last portion of the Pipeline, without needing to redo everything that came before.
The CloudBees-only checkpoint step makes this possible. Simply place a checkpoint at a safe point in your script, after performing some work and before doing something that might fail randomly.

Best Practice

Validation of Shared Library

If you use Pipeline Shared Library, you can create a unit test pipeline job to run through some simple shared library functions while Jenkins is in the quiet mode after a Quiet Start.