Issue

I would like to create a Pipeline that runs parallel task in the same way a Matrix job does

Environment

Jenkins
CloudBees Jenkins Enterprise
Pipeline Plugin
Pipeline Stage View Plugin
[Blue Ocean]

Related Issues

Resolution

You can run a pipeline that mocks the behavior of a Matrix jobs. You can trivially run the different combinations in parallel and express the structure like a Matrix.

In terms of Visualization however, Pipeline Stage View is not able to display the result in a Matrix structure. There are long standing improvement issues about this: JENKINS-27395 and JENKINS-33185.

Blue Ocean Plugin offer better visualization capabilities and is the way forward.

Example with “Flat” structure:

The following pipeline has 2 axes: Nodes and JDKs. Therefore there should be 4 tasks running in parallel:

osx-agent-1 / jdk7: Runs on node “osx-agent-1” and use the tool “jdk7”
osx-agent-1 / jdk8: Runs on node “osx-agent-1” and use the tool “jdk8”
osx-agent-2 / jdk7: Runs on node “osx-agent-2” and use the tool “jdk7”
osx-agent-2 / jdk8: Runs on node “osx-agent-2” and use the tool “jdk8”

def axisNode = ["osx-agent-1","osx-agent-2"]
def axisTool = ["jdk7","jdk8"]
def tasks = [:]

stage("Before") {
    node {
        echo "before"
    }
}

for(int i=0; i< axisNode.size(); i++) {
    def axisNodeValue = axisNode[i]
    for(int j=0; j< axisTool.size(); j++) {
        def axisToolValue = axisTool[j]
        tasks["${axisNodeValue}/${axisToolValue}"] = {
            node(axisNodeValue) {
                def javaHome = tool axisToolValue
                println "Node=${env.NODE_NAME}"
                println "Java=${javaHome}"
            }
        }
    }
}

stage ("Matrix") {
    parallel tasks
}

stage("After") {
    node {
        echo "after"
    }
}

Note that the “Matrix stage” is a simple parallel task and the matrix structure can be part of a bigger pipeline (as shown by this pipeline that is creating stages before and after the matrix structure).

Visualization in Blue Ocean

Example with “Nested” structure:

Another example is to run Nested Parallel tasks. The following pipeline do the exact same things except that we create a nest parallel tasks:

“osx-agent-1”: Parent stage
- osx-agent-1 / jdk7: Runs on node “osx-agent-1” and use the tool “jdk7”
- osx-agent-1 / jdk8: Runs on node “osx-agent-1” and use the tool “jdk8”
“osx-agent-2”: Parent stage
- osx-agent-2 / jdk7: Runs on node “osx-agent-2” and use the tool “jdk7”
- osx-agent-2 / jdk8: Runs on node “osx-agent-2” and use the tool “jdk8”

def axisNode = ["osx-agent-1","osx-agent-2"]
def axisTool = ["jdk7","jdk8"]
def tasks = [:]

stage("Before") {
    node {
        echo "before"
    }
}

for(int i=0; i< axisNode.size(); i++) {
    def axisNodeValue = axisNode[i]
    def subTasks = [:]
    for(int j=0; j< axisTool.size(); j++) {
        def axisToolValue = axisTool[j]
        subTasks["${axisNodeValue}/${axisToolValue}"] = {
            def javaHome = tool axisToolValue
            println "Node=${env.NODE_NAME}"
            println "Java=${javaHome}"
        }
    }
    tasks["${axisNodeValue}"] = {
        node(axisNodeValue) {
            parallel subTasks
        }
    }
}

stage ("Matrix") {
    parallel tasks
}

stage("After") {
    node {
        echo "after"
    }
}

Visualization in Blue Ocean

Note: Improvement for the visualization of Nested stages is an open issue, see JENKINS-38442

2 Comments

Date Votes

0

Michael Fowler January 15, 2019 16:36

An example for declarative would be nice

Comment actions Permalink

Allan Burdajewicz May 03, 2019 03:08

Michael,

For declarative, you would need to rely on the script { ... } block at the moment. So the solution is similar:

def getTasks(axisNodes, axisTools) {
    def tasks = [:]
    for(int i=0; i< axisNodes.size(); i++) {
        def axisNodeValue = axisNodes[i]
        for(int j=0; j< axisTools.size(); j++) {
            def axisToolValue = axisTools[j]
            tasks["${axisNodeValue}/${axisToolValue}"] = {
                node(axisNodeValue) {
                    def javaHome = tool axisToolValue
                    println "Node=${env.NODE_NAME}"
                    println "Java=${javaHome}"
                }
            }
        }
    }
    return tasks
}

pipeline {
    agent none

    stages {

        stage("Before") {
            agent any
            steps {
                echo "before"
            }
        }

        stage("Matrix") {
            steps {
                script {
                    def axisNodes = ["osx-agent-1","osx-agent-2"]
                    def axisTool = ["jdk7","jdk8"]
                    parallel getTasks(axisNodes, axisTool)
                }
            }
        }

        stage("After") {
            agent any
            steps {
                echo "after"
            }
        }
    }
}

Regards,

Please sign in to leave a comment.