Issue

• For CloudBees Jenkins Enterprise instances, the Linux kernels of the controllers and workers should be upgraded to patch the Meltdown CVE-2017-5754 and Spectre CVE-2017-5753 CVE-2017-5715 vulnerabilities

Environment

• CloudBees Jenkins Enterprise (CJE) - AWS
• CloudBees Jenkins Enterprise (CJE) - Anywhere

Resolution

Depending when your CloudBees Jenkins Enterprise instance was deployed, it may or may not be affected by these vulnerabilities, step 4 in the instructions below will help check.

1. Identify your current leader using:

   #!/bin/bash
   mesos_username=$(cje run echo-secrets router_username)
   mesos_password=$(cje run echo-secrets router_password)
   mesos_url=$(cje run display-outputs | grep mesos)
   mesos_protocol=$(echo $mesos_url | cut -d ':' -f 2)
   mesos_url=$(echo $mesos_url | cut -d ':' -f 3)
   curl -s $mesos_protocol:$mesos_url/master/state.json \
    -u "$mesos_username:$mesos_password" | \
    python -c 'import sys, json; print json.load(sys.stdin)["leader"]'
   

   With the IP address you get from that command, look up the hostname using nslookup ip-address-from-above-command
   Then get the list of all your controllers by running cje run display-outputs

2. We will upgrade every controller that is not the leader first. Starting with the first controller that is not the leader:

3. Connect to the controller by running:

   dna connect controller-N
   

4. To check if your kernel is vulnerable before upgrading, run:

   uname -a
   

   and look for the Kernel Version, then check with your OS Vendor to see which version contains the fix.

   • For Red Hat Enterprise Linux, consult: https://access.redhat.com/security/vulnerabilities/speculativeexecution
   • For CentOS, consult https://www.centos.org/forums/viewforum.php?f=51
   • For Ubuntu, consult https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown

5. Upgrade the kernel by running:

   • For CentOS or Red Hat Enterprise Linux:

   sudo yum check-update
   sudo yum upgrade kernel -y
   

   • For Ubuntu:

   sudo apt-get update
   sudo apt-get dist-upgrade -y
   

6. Restart the controller we just upgraded:

   sudo reboot now
   

7. Go back and perform steps 3 through 6 to upgrade the kernels for all the rest of the controllers that are not the leader.

8. Monitor the health of the controllers and masters in Mesos and Marathon until the controllers and associated tasks are marked as healthy and running.

9. Go back and perform steps 3 through 6 to upgrade the kernel of the leader.

10. Update all workers:

    • For CJE anywhere installations: consult with your OS vendor for the latest kernel versions with the fix (see links in step 4).
    • For CJE AWS installations:
      • Replace all workers one at a time by doing subsequent add and remove operations:
      • Add a new worker of the desired type with cje prepare worker-add, editing the worker-add.config, and running cje apply
      • Then remove the old worker with cje prepare worker-remove --server.name worker-OLD_WORKER_NUMBER followed by cje apply
      • Alternatively, you could also create your own AMIs and update the worker AMIs by following: How to change controllers and workers AMI but skip the steps regarding updating and restarting your controllers, since you have already done so above

This article was last updated on 2018-03-17.