Pipeline - Files manipulation

Issue

  • I am trying to port some legacy Jenkins build jobs that used Linux shell scripts (to Jenkins Workflow scripts) to download and stage/deploy to multiple server nodes.
    I need to be able to save and manipulate files to the Workflow workspace (or to somewhere under /tmp) directory structure.
  • When we use groovy library classes (AntBuilder, File) it is working fine on Client-Master but it is not working on Agent machines.

Environment

Resolution

The operations with File class are run on the master, so only works if the build is run on master.
In this example, I create a file and check if I can access it on a node using the method exists. The file can’t be found because the new File(file) code is executed on the master. To double-check this, I did the same for the folder Users, which exists on my master but not in the node.

stage 'file move wrong way'

//it only works on master
node('agent') {

    def ws = pwd()
    def context  = ws + "/testArtifact"
    def file = ws + '/file'
    sh 'touch ' + file
    sh 'ls ' + ws

    echo 'File on node : ' + new File(file).exists()
    echo 'Users : ' + new File('/Users').exists()

    sh 'mv ' + file + ' ' + context
    sh 'ls ' + ws
}

To execute file manipulation command we recommend to use native commands.

This is a simple example of operations in shell

stage 'Init'
node {
    stage 'Environment'
    sh 'export'

    stage 'Create file'
    sh 'touch test.txt'

    stage 'List relative Directory'
    sh 'ls ../../../../../'

    stage 'List / dir'
    sh 'ls /'

    stage 'List Home of user'
    sh 'ls $HOME'

    stage 'List Home of agent'
    sh 'ls $JENKINS_HOME'

    stage 'List temp'
    sh 'ls $TMPDIR'

    stage 'List job dir'
    sh '''
        echo $(pwd)
        ls .
        ls $(pwd)
        '''

    stage 'download file'
    def out='$(pwd)/download/maven.tgz'
    sh 'mkdir -p ./download'
    sh 'curl -L http://ftp.cixug.es/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz -o ' + out

    stage 'move/rename'
    def newName = 'mvn.tgz'
    sh 'mkdir -p $(pwd)/other'
    sh 'mv ' + out + ' ' + newName
    sh 'cp ' + newName + ' ' + out
}

Reference

Have more questions?

2 Comments

  • 0
    Avatar
    Renato Please Update

    Hi,

    Thanks for the article.

     

    Is there a list of methods that ONLY run on master? Lost like 10 hours or more on this one :D...

     

    Thanks and best regards,

    Renato

  • 0
    Avatar
    Denys Digtiar

    Hi Renato,

    >Except for the steps themselves, all of the Pipeline logic, the Groovy conditionals, loops, etc execute on the master. Whether simple or complex! Even inside a node block!

    from Best Practices for Scalable Pipeline Code 

Please sign in to leave a comment.