Pipeline - Files manipulation


  • I am trying to port some legacy Jenkins build jobs that used Linux shell scripts (to Jenkins Workflow scripts) to download and stage/deploy to multiple server nodes.
    I need to be able to save and manipulate files to the Workflow workspace (or to somewhere under /tmp) directory structure.
  • When we use groovy library classes (AntBuilder, File) it is working fine on Client-controller but it is not working on Agent machines.



The operations with File class are run on the controller, so only works if the build is run on controller.
In this example, I create a file and check if I can access it on a node using the method exists. The file can’t be found because the new File(file) code is executed on the controller. To double-check this, I did the same for the folder Users, which exists on my controller but not in the node.

stage 'file move wrong way'

//it only works on master
node('agent') {

    def ws = pwd()
    def context  = ws + "/testArtifact"
    def file = ws + '/file'
    sh 'touch ' + file
    sh 'ls ' + ws

    echo 'File on node : ' + new File(file).exists()
    echo 'Users : ' + new File('/Users').exists()

    sh 'mv ' + file + ' ' + context
    sh 'ls ' + ws

To execute file manipulation command we recommend to use native commands.

This is a simple example of operations in shell

stage 'Init'
node {
    stage 'Environment'
    sh 'export'

    stage 'Create file'
    sh 'touch test.txt'

    stage 'List relative Directory'
    sh 'ls ../../../../../'

    stage 'List / dir'
    sh 'ls /'

    stage 'List Home of user'
    sh 'ls $HOME'

    stage 'List Home of agent'
    sh 'ls $JENKINS_HOME'

    stage 'List temp'
    sh 'ls $TMPDIR'

    stage 'List job dir'
    sh '''
        echo $(pwd)
        ls .
        ls $(pwd)

    stage 'download file'
    def out='$(pwd)/download/maven.tgz'
    sh 'mkdir -p ./download'
    sh 'curl -L http://ftp.cixug.es/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz -o ' + out

    stage 'move/rename'
    def newName = 'mvn.tgz'
    sh 'mkdir -p $(pwd)/other'
    sh 'mv ' + out + ' ' + newName
    sh 'cp ' + newName + ' ' + out


Have more questions?


  • 0
    Renato Please Update


    Thanks for the article.


    Is there a list of methods that ONLY run on master? Lost like 10 hours or more on this one :D...


    Thanks and best regards,


  • 0
    Denys Digtiar

    Hi Renato,

    >Except for the steps themselves, all of the Pipeline logic, the Groovy conditionals, loops, etc execute on the master. Whether simple or complex! Even inside a node block!

    from Best Practices for Scalable Pipeline Code 

Please sign in to leave a comment.