Issue
- For CloudBees Jenkins Enterprise instances, the Linux kernels of the controllers and workers should be upgraded to patch the Meltdown CVE-2017-5754 and Spectre CVE-2017-5753 CVE-2017-5715 vulnerabilities
Environment
- CloudBees Jenkins Enterprise (CJE) - AWS
- CloudBees Jenkins Enterprise (CJE) - Anywhere
Resolution
Depending when your CloudBees Jenkins Enterprise instance was deployed, it may or may not be affected by these vulnerabilities, step 4 in the instructions below will help check.
-
Identify your current leader using:
#!/bin/bash mesos_username=$(cje run echo-secrets router_username) mesos_password=$(cje run echo-secrets router_password) mesos_url=$(cje run display-outputs | grep mesos) mesos_protocol=$(echo $mesos_url | cut -d ':' -f 2) mesos_url=$(echo $mesos_url | cut -d ':' -f 3) curl -s $mesos_protocol:$mesos_url/master/state.json \ -u "$mesos_username:$mesos_password" | \ python -c 'import sys, json; print json.load(sys.stdin)["leader"]'
With the IP address you get from that command, look up the hostname using
nslookup ip-address-from-above-command
Then get the list of all your controllers by runningcje run display-outputs
-
We will upgrade every controller that is not the leader first. Starting with the first controller that is not the leader:
-
Connect to the controller by running:
dna connect controller-N
-
To check if your kernel is vulnerable before upgrading, run:
uname -a
and look for the
Kernel Version
, then check with your OS Vendor to see which version contains the fix.- For Red Hat Enterprise Linux, consult: https://access.redhat.com/security/vulnerabilities/speculativeexecution
- For CentOS, consult https://www.centos.org/forums/viewforum.php?f=51
- For Ubuntu, consult https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown
-
Upgrade the kernel by running:
- For CentOS or Red Hat Enterprise Linux:
sudo yum check-update sudo yum upgrade kernel -y
- For Ubuntu:
sudo apt-get update sudo apt-get dist-upgrade -y
-
Restart the controller we just upgraded:
sudo reboot now
-
Go back and perform steps 3 through 6 to upgrade the kernels for all the rest of the controllers that are not the leader.
-
Monitor the health of the controllers and controllers in Mesos and Marathon until the controllers and associated tasks are marked as healthy and running.
-
Go back and perform steps 3 through 6 to upgrade the kernel of the leader.
-
Update all workers:
- For CJE anywhere installations: consult with your OS vendor for the latest kernel versions with the fix (see links in step 4).
- For CJE AWS installations:
- Replace all workers one at a time by doing subsequent add and remove operations:
- Add a new worker of the desired type with
cje prepare worker-add
, editing theworker-add.config
, and runningcje apply
- Then remove the old worker with
cje prepare worker-remove --server.name worker-OLD_WORKER_NUMBER
followed bycje apply
- Alternatively, you could also create your own AMIs and update the worker AMIs by following: How to change controllers and workers AMI but skip the steps regarding updating and restarting your controllers, since you have already done so above
This article was last updated on 2018-03-17.
1 Comments