0

Does Jenkins K8s Plugin store podTemplates anywhere?

Benjamin Jaques

May 19, 2020 18:23

None

Follow

Hello,

 

I am using the Jenkins K8s plugin (version 1.21) with the cloudbees-jenkins-distribution image (2.190.3.2-rolling)

I define my podTemplate in a k8s pod yaml, and then reference my pod yaml inside my Jenkinsfile like so:

agent {
    kubernetes {
        label 'my-build'
        yamlFile 'k8s-pod.yaml'
        defaultContainer 'npm'
        idleMinutes 20
    }
} 

Note that I reference a label and a yamlFile so that my build will pick up the pod if it already exists. I am seeing some weird behavior where if I make a change to the 'k8s-pod.yaml' inside my repo, that change is not reflected inside the pod that k8s wants to spin up (it takes the outdated pod yaml). 

I can remove the label, then the new pod comes up fine (with it's own build_name_build_number pod name), but I am curious why it isn't picking up on the latest changes. 

The Jenkins build logs point to the pod yaml at the correct commit.

• Facebook
• Twitter
• LinkedIn

2 comments

• 
  Allan Burdajewicz May 21, 2020 11:26 Official comment

  Hello Benjamin,

  Let me first answer your question about the storage of the template created dynamically.

  The template created by the kubernetes directive is not stored anywhere other than in memory. The directive leverages the podTemplate, node and container steps (see the source code). It does something like the following:

  podTemplate(label: 'my-build') {

    node('my-build'){

      // Only use the container step if we provide a defaultContainer that is not 'jnlp'

      container('maven') {

         [...]

      }

    }

  }

  And each step does the following:

  • podTemplate creates a dynamic pod template that is only available within the context of that step - i.e. within the podTemplate block. That is all it does. The podTemplate step does NOT allocate a kubernetes agent. It simply make a new template available.
  • the `node` block requests the label specified in the podTemplate and allocate an agent to the build. The request is process by the Jenkins Queue and the Jenkins Node Provisioner.
  • if the defaultContainer is not jnlp, the container step is used to start execute the build inside the container specified.

  What needs to be understood here is that it is not guaranteed that the agent allocated is the podTemplate being defined. Unless the label that you provide is unique among all agents in your Jenkins instance.

  In a scenario where you have several jobs using the same label in their kubernetes directive that are executed at around the same time. When the build requests for the label, those builds would provision different agents with the same label. You are therefore in a situation where - simply based on that label request - several agents can execute each of those builds. Those build might be allocated to the wrong agent.

  > Note that I reference a label and a yamlFile so that my build will pick up the pod if it already exists. I am seeing some weird behavior where if I make a change to the 'k8s-pod.yaml' inside my repo, that change is not reflected inside the pod that k8s wants to spin up (it takes the outdated pod yaml). 

  Since you are using a version of kubernetes greater than 1.17.0, do not specify any label. A unique label will be automatically assigned and will guarantee that the agent allocated is the one provisioned for this template. See https://issues.jenkins-ci.org/browse/JENKINS-57830.

  If a new agent and pod is provisioned and it is the one used by the build, but it does not reflect the spec of the YAML file of that build, check if maybe there is a parent template involved (pod template inheritance).

  Contact CloudBees Support if you need help to troubleshoot this.

  Regards,

• 0
  
  Benjamin Jaques May 21, 2020 12:43

  Thank you so much for the helpful and informative response. After digging into the k8s logs we believe we may have found the issue. The team had another branch build that was trying to spin up a broken pod yaml over and over again. They then kicked off another build that referenced the same label -- so we believe that build was trying to run its task on the existing, broken container, instead of spinning up the corrected pod yaml. I agree with you that the cleanest option is to leave the label blank and let the plugin generate a unique pod name every time.

  However, I have a follow-up clarification and question if you don't mind based on this.. We wanted to reuse the same label so that our bigger builds could share a pod and its contents -- build dependencies/SCM data. This is actually working out really nicely for us but there are potential errors like this one we encountered. Do you have a recommended path for caching build data so that subsequent builds don't take as long in a kubernetes environment? I reckon we could deploy a static agent pod with a PV that could live inside our namespace forever, but am wondering if there is a better way.

   

  Thanks for all the help -- really appreciate it!