Pipeline - Parallel execution of tasks

Issue

  • Want to execute parallel tasks in Pipeline

Environment

Resolution

Following a couple of options to achieve it.

Disclaimer: Please note that the above script just has been proved/tested within the specified environment. In the case a particular step does work within your environment, please go to <JENKINS_URL>/pipeline-syntax/ validate the syntax of the problematic step with the output from the Snippet Generator.

Option 1

def tasks = [:]

tasks["task_1"] = {
  stage ("task_1"){    
    node('label_example1') {  
        sh 'echo $NODE_NAME'
    }
  }
}
tasks["task_2"] = {
  stage ("task_2"){    
    node('label_example2') {  
        sh 'echo $NODE_NAME'
    }
  }
}

parallel tasks

Option 2

def testList = ["a", "b", "c", "d"]
def branches = [:] 

for (int i = 0; i < 4 ; i++) {
       int index=i, branch = i+1
       stage ("branch_${branch}"){ 
            branches["branch_${branch}"] = { 
                node ('label_example'){
                    sh "echo 'node: ${NODE_NAME},  index: ${index}, i: ${i}, testListVal: " + testList[index] + "'"
                }
            }
      }
}

parallel branches

Note: Variables define outside the for block are not local, but global to the script. Testing the option 2, you will notice that variable i prints always value 4, whereas index increases from 0 to 3 and branch from 1 to 4.

References

Have more questions? Submit a request

4 Comments

  • 0
    Avatar
    Serosset Serosset

    Does "parallel" have options to control what happens in case of failure of one of the parallel tasks? What is the default behavior? Fail the "parallel" statement as soon as one of the tasks fail?

    We are using a pipeline job with a combination of "parallel" and downstream jobs, what happens when a downstream job fails? (similar to the example at https://support.cloudbees.com/hc/en-us/articles/221400287-How-to-pass-parameter-to-downstream-job-in-Pipeline-job)

    Edited by Serosset Serosset
  • 0
    Avatar
    Pushpa Buzz

    I have used option-1. I am not passing any branch details or anything. Just defined the ubuntu image and running scripts in stage under sh''' .. '''. 

    I see "Failed in branch" and hung the job as well.

    Can you help me?

     

    [task_1] [Pipeline] [task_1] // stage
    [Pipeline] [task_1] }
    [task_1] Failed in branch task_1
  • 0
    Avatar
    Anand Sridharan

    With option1 I am getting the following error

    ERROR: Build step failed with exception groovy.lang.MissingMethodException: No signature of method: Script1.parallel() is applicable for argument types: (java.util.LinkedHashMap) values: [[task_1:Script1$_run_closure1@535b4b7, task_2:Script1$_run_closure2@73eb3b5a]]

    I am running Jenkins 2.89.4 with the latest version of pipeline. I am running this as a "Execute System groovy script" in a job. 

     EDIT: 

    Executing this code as "Pipeline script" works. I reckon libraries relating to pipeline doesn't load when i execute this as groovy script. 

    Edited by Anand Sridharan
  • 0
    Avatar
    Dineshk7sg Dineshk7sg

    Hi I have 2 questions.

    No 1: Can I create and run tasks individually instead parallel

       - for example  

    def tasks
    tasks["task_1"] = { stage ("task_1"){ node('label_example1') { sh 'echo $NODE_NAME' } } }

    run tasks

    Why I need task to be run separately? - to visualize a task in blue ocean.

     

    No 2: How can I run parallel stages with multi platform pipeline? below code does. However locking the node is an issue.

    when having the node block inside the stage block in a parallel tasks (why node block inside stage block, because we don't want to create duplicate stage for each task), the node get freed after the first stage and it become available for other jobs. Please find the below code, where the node get occupied by another job after the stage `first`.

    def stages = ["first","second"]
    
    def tasks = [:]
    
    for (item in stages) {
      stage (item) {
        tasks["win"] = {
          node(mywinnode) {
            bat 'echo "check"'
          }
        }
        tasks["mac"] = {
          node(mymacnode) {
            sh 'echo "check"'
          }
        }
        parallel task
      }
    }

     

Please sign in to leave a comment.