Upgrade Guide: CloudBees Jenkins Platform and Team

Issue

  • We are planning to upgrade our current CloudBees Jenkins version [Origin] to a certain major version [Destination].

Environment

Resolution

A. Before Upgrading

A.1 - Critical points

The success of your upgrade plan relies on:

a. Let the Support Team knows about your Upgrade Plan by opening a ticket and providing information detailed on Required Data Upgrade a Jenkins Instance.

b. [Destination] release must be supported see Supported Product Versions and Release End of Life

c. Do not perform direct upgrade to Production Environment before not having done in a Test Environment first (cf Architecting for Manageability).

d. Ensure your CloudBees Jenkins [Destination] environment is set-up accordingly with Jenkins instance is prepared for support. The correct configuration of the GC and ulimit is critical for a good Jenkins performance and to avoid crashes. Please, take particular attention to Java Parameters, Ulimit Settings, Log Startup Timing Info.

e. Take a SNAPSHOT of CloudBees Jenkins [Origin]

  • Generate a support bundle with the current status. It will help in case there are problems with the upgrade or to know what plugins were upgraded in the process.
  • [IMPORTANT] Backup your instance before starting your upgrade. The minimal backup is a copy of your ${JENKINS_HOME} directory. If the BUILD_DIRECTORY is out of the ${JENKINS_HOME} you must backup this directory as well.

Note: In the case something unexpected happens, the rollback process in based on the Backup of your instance.

e. For CJP-CM upgrade as RPM, OpenSuse or Debian

  • Currently, Jenkins and CJP-CM have the same name package name: jenkins thus you must review your sources repositories to ensure that it only contains CloudBees repositories. Upgrades of package from the Jenkins open source project sources to a CloudBees $JENKINS_HOME could make all your CloudBees features not working.

For example, for Debian you must look at /etc/apt/sources.list and /etc/apt/sources.list.d/* to ensure it points to https://downloads.cloudbees.com/cje/rolling/debian and not to http://pkg.jenkins.io/debian-stable. Alternatively, apt-cache policy can be used to display the priorities of package sources as well as those of individual packages.

f. For CJP-CM connected to CJP-OC

  • CJP-OC must first be upgraded, then CJP-CM.
  • Starting with CloudBees Jenkins Platform 2.7, the version of CJP-OC must always be more recent or as old as the version of the client masters that are connected to the CloudBees Jenkins Operations Center (CJP-OC >= CJP-CM version)
  • The CJP-CM connected to a CJP-OC do not have to be at the same version.

A.2 - Must-read References

General

Check the impact of the changes

a. Read the CloudBees Release Notes for all the product versions from the current version [Origin] to the version you are targeting to upgrade to [Destination] (including theirs revisions).

release notes list

Each product release/revision page provides details about New Features and Known and Resolved issues fixed. Besides, there is other valuable information into tabs:

release notes

For this task, use the CloudBees Comparing Version Web tool which lists changes from [Origin] to [Destination]: Baseline and Plugins.

  • CJP-CM - https://release-notes.cloudbees.com/compare/1/[Origin]...[Destination]
  • CJP-OC - https://release-notes.cloudbees.com/compare/38/[Origin]...[Destination]
  • CJT - https://release-notes.cloudbees.com/compare/121/[Origin]...[Destination]

release notes

b. Read attentively all Jenkins LTS Upgrade Guide.

c. If you are upgrading Pipeline plugin, you might be facing some incompatibilities due to recent changes on this plugin. You can check those changes on the Wiki of this plugin.

A.3 - Recommendations

a. Enable CAP [since Jenkins 2.x]

For Jenkins 2.x, [IMPORTANT] Ensure that CAP is enabled in order to avoid plugin dependency issues or incorrect versions installed.

b. Upgrade to latest version

It is recommended always to upgrade to the latest version of your CloudBees product, ensuring a wider lifecycle support coverage plus more security patches added (CloudBees Security Advisory), more fixed issues and more new features included.

c. $JENKINS_HOME should be in the same location

It is highly recommended to upgrade Jenkins on the same location where it is running even if we are talking about the production environment. Doing a backup of $JENKINS_HOME and the $BUILD_DIR - in case it is outside the default location - should be enough to revert to the previous status.

It is not recommended to keep two different instances working at the same time to avoid downtime while performing the upgrade unless you really know what you are doing - it will be very difficult to replicate the exactly same environment.

  • JNLP slave will not work correctly as the $JENKINS_URL location will be different
  • Credentials might fail in case the secret is not the same on both instances
  • The OS might not be configured in the same way. i.e ulimit
  • The Jenkins configuration might not be correctly replicated

d. Stop using Apache Maven builds (if it is possible)

Maven plugin plugin is not recommended as is often considered evil by the Community. Therefore, we recommend to migrate job using this plugin to Pipeline Jobs with the Maven integration.

If (and only if) you are able to migrate your Maven plugin builds, please read carefully our Maven jobs and Java versions compatibility guide.

A.4 - Considerations

Upgrades inside 1.x (deprecated)

In the rare case you need to upgrade to a newer 1.x version: IMPORTANT !!! - Verify that there are not any plugins that are pinned. When the server is stopped, remove all files named *.pinned from ${JENKINS_HOME}/plugins (see Pinned Plugins).

Upgrades from 1.x to 2.x

  • Java 8 should be installed on the machines. Read CloudBees Jenkins Platform Supported Java Versions
  • Maven jobs might be affected as they are required now to be using the same Java version than the Master is running. The workaround is explained on this KB article. It is recommended to test this workaround on a test environment.
  • Groovy has been upgraded from 1.8.x to 2.4.x on Jenkins 2.x. This change will produce an increase of the heap memory consumption on the instance. Since the Groovy API changed, it means some of your Pipelines might need to be adapted to those changes.
  • Removal of the AJP connector from the embedded Winstone-Jetty container
  • The container must be Servlet 3.1 (Java EE 7) compliant: Apache Tomcat 8, Wildfly 8, Glassfish 4, Websphere 9.
  • Off-line upgrades. When upgrading CJP some offline automatic upgrades will happen. You can check the plugins which will be upgraded taking a look at the “Plugins” tab of the corresponding product version in the CloudBees Release Notes.
  • Support for all TLS versions before 1.2 has been dropped. If you’re using the embedded Jetty container’s HTTPS functionality, it will no longer accept TLS 1.0 or 1.1 connections.

Upgrades inside 2.x

  • Off-line upgrades. When upgrading CJP some offline automatic upgrades will happen.
  • Support for all TLS versions before 1.2 has been dropped in versions 2.73.1+. If you’re using the embedded Jetty container’s HTTPS functionality, it will no longer accept TLS 1.0 or 1.1 connections.

B. Upgrade

B.1 - Get the installers

All CloudBees Jenkins installers can be found in the CloudBees Release Notes. Browse to your Product (CJT, CJP-CM and/or CJP-OC), then the specific version. On the main page, on the right upper corner there is an Downloads click on it select your desired distribution type:

  • WAR
  • RPM
  • OpenSuse
  • Debian
  • Windows
  • Docker

releaseNotesDownload.png

B.2 - How to Install

Same content explained in the Upgrade section for each distribution type in CJT Installation from scratch can be applied here.

WAR

If your CloudBees Jenkins instance is run with the command java -jar jenkins.war, you can simply replace the jenkins.war file with the latest version.

This can be done manually or through Jenkins from the Manage Jenkins page. At the top, you will see a link stating the most recent version. Simply click it and Jenkins will begin the upgrade process.

Linux

Red Hat / CentOS
  • CJP-CM - yum upgrade jenkins
  • CJP-OC - yum upgrade jenkins-oc
  • CJT - yum upgrade cjt

IMPORTANT: RPM/YUM with High Availability

The RPM package contains a post-install script to ensure ownership on several files including JENKINS_HOME. Therefore if High Availability is setup, this could lead to I/O Errors - for example when upgrading one node while another one is still running. More information is available in JENKINS-23273.

To workaround this problem, you can skip this script by adding the property JENKINS_INSTALL_SKIP_CHOWN="true" under /etc/sysconfig/jenkins.

Debian / Ubuntu
  • CJP-CM - apt-get update && apt-get install jenkins
  • CJP-OC - apt-get update && apt-get install jenkins-oc
  • CJT - apt-get update && apt-get install cjt
openSUSE / SUSE Linux
  • CJP-CM - zypper install jenkins
  • CJP-OC - zypper install jenkins-oc
  • CJT - zypper install cjt

Servlet Container

Once a .war archive has been downloaded, follow the servlet container’s existing application deployment process.

When using servlet containers, CJP-CM/CJP-OC/CJT will set the JENKINS_HOME to the $APP_SERVER_USER/.jenkins/ folder. If the servlet container installation does not include write permissions to this folder for this user (sometimes done for security), you either need to grant appropriate permissions or override this setting by adding the "-DJENKINS_HOME=$MY_JENKINSPATH" argument in your servlet container startup. Refer to the servlet container’s documentation for how to add startup arguments.

Custom container installations

If you use a custom container, you will find the jenkins.war file in the deploy directory of your container. For example, /usr/local/jboss/server/default/deploy/jenkins.war would be the location for a default JBoss installation.

Tomcat

Use environment variable CATALINA_OPTS to add:

  • -Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true.
  • -Dorg.apache.catalina.connector.CoyoteAdapter.ALLOW_BACKSLASH=true which is needed for Blue Ocean.

It is recommended to configure it in the script $CATALINA_BASE/bin/setenv.sh (linux) or %CATALINA_BASE%\bin\setenv.bat (windows) that you’ll create to customize your application server.

Windows

In order to upgrade CJP-OC, CJP-CM or CJT on Windows servers, a new .zip must be downloaded and installed.

Cloud Providers

C. After Upgrading

a. If you are using Windows agents using the “Launch slave agents via Java Web Start” option (e.g. JNLP slaves), the slave.jar on each must be updated. Please see the Updating The Windows Slave procedure.

Note - This step isn’t necessary for SSH slaves, using the “Launch slave agents on Unix machines via SSH” option offered by the SSH Slaves Plugin.

b. Review all articles in our Best Practices section, specifically: Performance and Remoting.

Was this article helpful?
3 out of 4 found this helpful
Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.