Pipeline - Parallel execution of tasks

Issue

  • I want to execute parallel tasks in Pipeline

Environment

Resolution

You need to use the parallel directive.
The directive takes a map of String and Closure.
The string is the display name of the parallel execution and the Closure the actual Pipeline code you wish to execute.
Here are two examples to get you started:

def tasks = [:]

tasks["task_1"] = {
  stage ("task_1"){    
    node('label_example1') {  
        sh 'echo $NODE_NAME'
    }
  }
}
tasks["task_2"] = {
  stage ("task_2"){    
    node('label_example2') {  
        sh 'echo $NODE_NAME'
    }
  }
}

parallel tasks
def testList = ["a", "b", "c", "d"]
def branches = [:] 

for (int i = 0; i < 4 ; i++) {
       int index=i, branch = i+1
       stage ("branch_${branch}"){ 
            branches["branch_${branch}"] = { 
                node ('label_example'){
                    sh "echo 'node: ${NODE_NAME},  index: ${index}, i: ${i}, testListVal: " + testList[index] + "'"
                }
            }
      }
}

parallel branches

In case you are doing declarative Pipeline, make sure to review its documentation.

References

Have more questions?

6 Comments

  • 0
    Avatar
    Serosset Serosset

    Does "parallel" have options to control what happens in case of failure of one of the parallel tasks? What is the default behavior? Fail the "parallel" statement as soon as one of the tasks fail?

    We are using a pipeline job with a combination of "parallel" and downstream jobs, what happens when a downstream job fails? (similar to the example at https://support.cloudbees.com/hc/en-us/articles/221400287-How-to-pass-parameter-to-downstream-job-in-Pipeline-job)

    Edited by Serosset Serosset
  • 0
    Avatar
    Pushpa Buzz

    I have used option-1. I am not passing any branch details or anything. Just defined the ubuntu image and running scripts in stage under sh''' .. '''. 

    I see "Failed in branch" and hung the job as well.

    Can you help me?

     

    [task_1] [Pipeline] [task_1] // stage
    [Pipeline] [task_1] }
    [task_1] Failed in branch task_1
  • 0
    Avatar
    Anand Sridharan

    With option1 I am getting the following error

    ERROR: Build step failed with exception groovy.lang.MissingMethodException: No signature of method: Script1.parallel() is applicable for argument types: (java.util.LinkedHashMap) values: [[task_1:Script1$_run_closure1@535b4b7, task_2:Script1$_run_closure2@73eb3b5a]]

    I am running Jenkins 2.89.4 with the latest version of pipeline. I am running this as a "Execute System groovy script" in a job. 

     EDIT: 

    Executing this code as "Pipeline script" works. I reckon libraries relating to pipeline doesn't load when i execute this as groovy script. 

    Edited by Anand Sridharan
  • 0
    Avatar
    Dineshk7sg Dineshk7sg

    Hi I have 2 questions.

    No 1: Can I create and run tasks individually instead parallel

       - for example  

    def tasks
    tasks["task_1"] = { stage ("task_1"){ node('label_example1') { sh 'echo $NODE_NAME' } } }

    run tasks

    Why I need task to be run separately? - to visualize a task in blue ocean.

     

    No 2: How can I run parallel stages with multi platform pipeline? below code does. However locking the node is an issue.

    when having the node block inside the stage block in a parallel tasks (why node block inside stage block, because we don't want to create duplicate stage for each task), the node get freed after the first stage and it become available for other jobs. Please find the below code, where the node get occupied by another job after the stage `first`.

    def stages = ["first","second"]
    
    def tasks = [:]
    
    for (item in stages) {
      stage (item) {
        tasks["win"] = {
          node(mywinnode) {
            bat 'echo "check"'
          }
        }
        tasks["mac"] = {
          node(mymacnode) {
            sh 'echo "check"'
          }
        }
        parallel task
      }
    }

     

  • 0
    Avatar
    Imran Malik

    Serosset Serosset yes use as `failFast: true` when invoking `parallel`. 

    ```

    parallel someparalleltasks, failFast:true 

    ```

    Edited by Imran Malik
  • 0
    Avatar
    Ramkumar Bangaru

    Is it possible to read the "branchname" inside the closure?

     

    for example,

    ```

    branch["a"] = {

    echo "${branchname}"

    }

     

    branch["b"] = {

    echo "${branchname}"

    }

     

    parallel branch

    ```

     

Please sign in to leave a comment.