How to upgrade Jenkins?

Issue

  • Jenkins requires an upgrade

Environment

  • CloudBees Jenkins Enterprise (CJE)
  • CloudBees Jenkins Operation Center (CJOC)

Resolution

IMPORTANT !!!

  • Always test upgrades in a test environment before doing it in production (cf Architecting for Manageability).
  • If you plan to upgrade, ensure that before opening a support ticket you provide:
  • A support bundle from the previous status
  • A support bundle of the current status
  • The logs directly from /var/log/jenkins
  • Explanation of what went wrong with the Upgrade

References

Useful links before starting:

Upgrade location

It is highly recommended to upgrade Jenkins on the same location where it is running even if we are talking about the production environment. Doing a backup of $JENKINS_HOME and the $BUILD_DIR - in case it is outside the default location - should be enough to revert to the previous status.

It is not recommended to keep two different instances working at the same time to avoid downtime while performing the upgrade unless you really know what you are doing - it will be very difficult to replicate the exactly same environment.
* JNLP slave will not work correctly as the $JENKINS_URL location will be different
* Credentials might fail in case the secret is not the same on both instances
* The OS might not be configured in the same way. i.e ulimit
* The Jenkins configuration might not be correctly replicated

Before Upgrading your instance

Take a SNAPSHOT of the current state

  • Generate a support bundle with the current status. It will help in case there are problems with the upgrade or to know what plugins were upgraded in the process.
  • [IMPORTANT] Backup your instance before starting your upgrade. The minimal backup is a copy of your ${JENKINS_HOME} directory. If the BUILD_DIRECTORY is out of the ${JENKINS_HOME} you must backup this directory as well.

Check the impact of the changes

Upgrades inside 1.x (deprecated)

In the rare case you need to upgrade to a newer 1.x version: IMPORTANT !!! - Verify that there are not any plugins that are pinned. When the server is stopped, remove all files named *.pinned from ${JENKINS_HOME}/plugins (see Pinned Plugins).

Upgrades from 1.x to 2.x

  • Java 8 should be installed on the machines. Read CloudBees Jenkins Platform Supported Java Versions
  • Maven jobs might be affected as they are required now to be using the same Java version than the Master is running. The workaround is explained on this KB article. It is recommended to test this workaround on a test environment.
  • Groovy has been upgraded from 1.8.x to 2.4.x on Jenkins 2.x. This change will produce an increase of the heap memory consumption on the instance. Since the Groovy API changed, it means some of your Pipelines might need to be adapted to those changes.
  • Removal of the AJP connector from the embedded Winstone-Jetty container
  • The container must be Servlet 3.1 (Java EE 7) compliant: Apache Tomcat 8, Wildfly 8, Glassfish 4, Websphere 9
  • Off-line upgrades. When upgrading CJP some offline automatic upgrades will happen. You can check the plugins which will be upgraded taking a look at our documentation:
  • CJE plugin included
  • CJOC plugin included

Note: If your CloudBees Jenkins Enterprise is connected to a CloudBees Jenkins Operations Center (CJOC) instance, the CJOC must first be upgraded to CJP2

Upgrades inside 2.x

  • [IMPORTANT] Ensure that CAP is enabled in order to avoid plugin dependency issues or incorrect versions installed.
  • Off-line upgrades. When upgrading CJP some offline automatic upgrades will happen. You can check the plugins which will be upgraded taking a look at our documentation:
  • CJE plugin included
  • CJOC plugin included

Considerations

  • If you are using Windows agents using the “Launch slave agents via Java Web Start” option (e.g. JNLP slaves), the slave.jar on each must be updated. Please see the Updating The Windows Slave procedure

Note - This step isn’t necessary for SSH slaves, using the “Launch slave agents on Unix machines via SSH” option offered by the SSH Slaves Plugin

Also, it is very recommendable to change the JNLP protocol to version 4 (JNLP-4). You can do this from Manage Jenkins -> Configure Global Security.

  • [IMPORTANT] You should ensure that your Jenkins instance is prepared for support. The correct configuration of the GC and ulimit is critical for a good Jenkins performance and to avoid crashes. Please, take particular attention to Java Parameters, Ulimit Settings, Log Startup Timing Info.

Upgrading your instance

You are now ready to install the latest version.

All CloudBees Jenkins Enterprise files can be found at the CloudBees Network Downloads page

How to Install:

War file installations

If your CloudBees Jenkins Enterprise instance is run with the command java -jar jenkins.war, you can simply replace the jenkins.war file with the latest version.  This can be done manually or through Jenkins from the Manage Jenkins page.  At the top, you will see a link stating the most recent version.  Simply click it and Jenkins will begin the upgrade process.

Custom container installations

If you use a custom container, you will find the jenkins.war file in the deploy directory of your container.  For example, /usr/local/jboss/server/default/deploy/jenkins.war would be the location for a default JBoss installation.

Debian and other Linux distro installations

Certain systems, such as Debian, will not allow upgrades via the Manage Jenkins page.  You can run the update from your package manager in these cases.  For example, in Debian you can run aptitude update then aptitude install jenkins.

IMPORTANT: RPM/YUM with High Availability

The RPM package contains a post-install script to ensure ownership on several files including JENKINS_HOME. Therefore if High Availability is setup, this could lead to I/O Errors - for example when upgrading one node while another one is still running. More information is available in JENKINS-23273.

To workaround this problem, you can skip this script by adding the property JENKINS_INSTALL_SKIP_CHOWN="true" under /etc/sysconfig/jenkins.

CloudBees Jenkins Platform 2.X

Note: If your CloudBees Jenkins Enterprise is connected to a CloudBees Jenkins Operations Center (CJOC) instance, the CJOC must first be upgraded to CJP2

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.