Create a Matrix-like flow with Pipeline

Issue

  • I would like to create a Pipeline that runs parallel task in the same way a Matrix job does

Environment

Related Issues

Resolution

You can run a pipeline that mocks the behavior of a Matrix jobs. You can trivially run the different combinations in parallel and express the structure like a Matrix.

In terms of Visualization however, Pipeline Stage View is not able to display the result in a Matrix structure. There are long standing improvement issues about this: JENKINS-27395 and JENKINS-33185.

Blue Ocean Plugin offer better visualization capabilities and is the way forward.

Example with “Flat” structure:

The following pipeline has 2 axes: Nodes and JDKs. Therefore there should be 4 tasks running in parallel:

  • osx-agent-1 / jdk7: Runs on node “osx-agent-1” and use the tool “jdk7”
  • osx-agent-1 / jdk8: Runs on node “osx-agent-1” and use the tool “jdk8”
  • osx-agent-2 / jdk7: Runs on node “osx-agent-2” and use the tool “jdk7”
  • osx-agent-2 / jdk8: Runs on node “osx-agent-2” and use the tool “jdk8”
def axisNode = ["osx-agent-1","osx-agent-2"]
def axisTool = ["jdk7","jdk8"]
def tasks = [:]

stage("Before") {
    node {
        echo "before"
    }
}

for(int i=0; i< axisNode.size(); i++) {
    def axisNodeValue = axisNode[i]
    for(int j=0; j< axisTool.size(); j++) {
        def axisToolValue = axisTool[j]
        tasks["${axisNodeValue}/${axisToolValue}"] = {
            node(axisNodeValue) {
                def javaHome = tool axisToolValue
                println "Node=${env.NODE_NAME}"
                println "Java=${javaHome}"
            }
        }
    }
}

stage ("Matrix") {
    parallel tasks
}

stage("After") {
    node {
        echo "after"
    }
}

Note that the “Matrix stage” is a simple parallel task and the matrix structure can be part of a bigger pipeline (as shown by this pipeline that is creating stages before and after the matrix structure).

Visualization in Blue Ocean

Example with “Nested” structure:

Another example is to run Nested Parallel tasks. The following pipeline do the exact same things except that we create a nest parallel tasks:

  • “osx-agent-1”: Parent stage
    • osx-agent-1 / jdk7: Runs on node “osx-agent-1” and use the tool “jdk7”
      • osx-agent-1 / jdk8: Runs on node “osx-agent-1” and use the tool “jdk8”
  • “osx-agent-2”: Parent stage
    • osx-agent-2 / jdk7: Runs on node “osx-agent-2” and use the tool “jdk7”
      • osx-agent-2 / jdk8: Runs on node “osx-agent-2” and use the tool “jdk8”
def axisNode = ["osx-agent-1","osx-agent-2"]
def axisTool = ["jdk7","jdk8"]
def tasks = [:]

stage("Before") {
    node {
        echo "before"
    }
}

for(int i=0; i< axisNode.size(); i++) {
    def axisNodeValue = axisNode[i]
    def subTasks = [:]
    for(int j=0; j< axisTool.size(); j++) {
        def axisToolValue = axisTool[j]
        subTasks["${axisNodeValue}/${axisToolValue}"] = {
            def javaHome = tool axisToolValue
            println "Node=${env.NODE_NAME}"
            println "Java=${javaHome}"
        }
    }
    tasks["${axisNodeValue}"] = {
        node(axisNodeValue) {
            parallel subTasks
        }
    }
}

stage ("Matrix") {
    parallel tasks
}

stage("After") {
    node {
        echo "after"
    }
}

Visualization in Blue Ocean

Note: Improvement for the visualization of Nested stages is an open issue, see JENKINS-38442

Was this article helpful?
0 out of 1 found this helpful

Have more questions?

2 Comments

Date Votes
  • 0
    Avatar
    Michael Fowler

    An example for declarative would be nice

  • 0
    Avatar
    Allan Burdajewicz

    Michael,

    For declarative, you would need to rely on the script { ... } block at the moment. So the solution is similar:

    def getTasks(axisNodes, axisTools) {
        def tasks = [:]
        for(int i=0; i< axisNodes.size(); i++) {
            def axisNodeValue = axisNodes[i]
            for(int j=0; j< axisTools.size(); j++) {
                def axisToolValue = axisTools[j]
                tasks["${axisNodeValue}/${axisToolValue}"] = {
                    node(axisNodeValue) {
                        def javaHome = tool axisToolValue
                        println "Node=${env.NODE_NAME}"
                        println "Java=${javaHome}"
                    }
                }
            }
        }
        return tasks
    }

    pipeline {
        agent none

        stages {

            stage("Before") {
                agent any
                steps {
                    echo "before"
                }
            }

            stage("Matrix") {
                steps {
                    script {
                        def axisNodes = ["osx-agent-1","osx-agent-2"]
                        def axisTool = ["jdk7","jdk8"]
                        parallel getTasks(axisNodes, axisTool)
                    }
                }
            }

            stage("After") {
                agent any
                steps {
                    echo "after"
                }
            }
        }
    }

    Regards,

Please sign in to leave a comment.