Issue

How to enable managed controllers to pull the container image at every restart.
How to enable managed controllers to not use cached outdated container images.

Environment

Explanation

The imagePullPolicy and the tag of the image affect when the kubelet attempts to pull the specified image.

imagePullPolicy: IfNotPresent the image is pulled only if it is not already present locally.
imagePullPolicy: Always every time the kubelet launches a container, the kubelet queries the container image registry to resolve the name to an image digest. If the kubelet has a container image with that exact digest cached locally, the kubelet uses its cached image; otherwise, the kubelet downloads (pulls) the image with the resolved digest, and uses that image to launch the container.

The default pull policy is IfNotPresent which causes the kubelet to skip pulling an image if it already exists.

Resolution

imagePullPolicy

1./ Describe the Pod of the questioned managed controller. In our example, the controller name is mc1.

Run kubectl command:

$ kubectl get pod mc1-0 -o yaml -n <CloudBees CI namespace> | grep imagePullPolicy
    imagePullPolicy: IfNotPresent

As you can notice, imagePullPolicy is set to IfNotPresent.

2./ Open the questioned controller configuration page in the CloudBees Operations Center UI. Under Advanced configuration section add imagePullPolicy: Always pair.

3./ Restart the managed controller from the CloudBees Operations Center UI.

4./ Validate that the managed controller Pod and StatefulSet have the new policy applied:

$ kubectl get pod mc1-0 -o yaml -n <CloudBees CI namespace> | grep imagePullPolicy
    imagePullPolicy: Always

$ kubectl get sts mc1 -o yaml -n <CloudBees CI namespace> | grep imagePullPolicy
    imagePullPolicy: Always

Note: any time the managed controller is restarted, the provisioning log should look similar to:

[Tue Jul 07 19:39:55 UTC 2020][Normal][Pod][mc1-0][Pulling] Pulling image "cloudbees/cloudbees-core-mm:2.235.1.2"
[Tue Jul 07 19:39:57 UTC 2020][Normal][Pod][mc1-0][Pulled] Successfully pulled image "cloudbees/cloudbees-core-mm:2.235.1.2"

Admission Controllers

Alternatively, you can use AlwaysPullImages.

This admission controller modifies every new Pod to force the image pull policy to Always. This is useful in a multi-tenant cluster so that users can be assured that their private images can only be used by those who have the credentials to pull them. Without this admission controller, once an image has been pulled to a node, any pod from any user can use it simply by knowing the image’s name (assuming the Pod is scheduled onto the right node), without any authorization check against the image. When this admission controller is enabled, images are always pulled prior to starting containers, which means valid credentials are required.

References

How to enable managed controller to pull the container image at every restart