Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41805

Pipeline Job-- deletedir() delete only current directory but @script and @tmp dir still there in workspace.

    Details

    • Similar Issues:

      Description

      Directories with an ampersand (like @tmp and @script) are not removed when using 'deletedir()' in pipeline stage.

        Attachments

          Issue Links

            Activity

            Hide
            recampbell Ryan Campbell added a comment -

            Please reopen with step by step instructions to reproduce the problem and clear description of what you expect to happen.

            Show
            recampbell Ryan Campbell added a comment - Please reopen with step by step instructions to reproduce the problem and clear description of what you expect to happen.
            Hide
            stefan1509 Stefan Droog added a comment - - edited

            Multibranch pipeline (Jenkinsfile):

             

            stage('Checkout') {
            checkout scm
            }
            
            stage('Build') { 
            def resultDist = sh returnStdout: true, script: './gradlew build'
            println resultDist
            }
            
            stage('Cleanup') {
            deleteDir()
            }
            

            This will create some directories in the Jenkins workspace, e.g: 

            MyBranch-GWI555O7BRZQK2BI4NIP67CEYQUZYH36AKCBALPQ2TWX4CIGNYIZ
            MyBranch-GWI555O7BRZQK2BI4NIP67CEYQUZYH36AKCBALPQ2TWX4CIGNYIZ@tmp

            After the 'Cleanup' stage the directories with an ampersand are still there

            We expect that all directories are removed.

            Show
            stefan1509 Stefan Droog added a comment - - edited Multibranch pipeline (Jenkinsfile):   stage( 'Checkout' ) { checkout scm } stage( 'Build' ) { def resultDist = sh returnStdout: true , script: './gradlew build' println resultDist } stage( 'Cleanup' ) { deleteDir() } This will create some directories in the Jenkins workspace, e.g:  MyBranch-GWI555O7BRZQK2BI4NIP67CEYQUZYH36AKCBALPQ2TWX4CIGNYIZ MyBranch-GWI555O7BRZQK2BI4NIP67CEYQUZYH36AKCBALPQ2TWX4CIGNYIZ@tmp After the 'Cleanup' stage the directories with an ampersand are still there We expect that all directories are removed.
            Hide
            hiten_prajapati Hiten Prajapati added a comment - - edited

            yes Stefan Droog you are right.
            i am still facing this issue. 
            currently i have to delete that directory manual.

            i faced this when my storage is fully occupied, i have many branches so each branch taking lots spaces after completed build. 

            Show
            hiten_prajapati Hiten Prajapati added a comment - - edited yes Stefan Droog you are right. i am still facing this issue.  currently i have to delete that directory manual. i faced this when my storage is fully occupied, i have many branches so each branch taking lots spaces after completed build. 
            Hide
            batmat Baptiste Mathus added a comment -

            I fail to understand how this issue can be set to "Important" and that importance not be explained in details. At first sight it seems like one of Minor or less priority. Not a bug by the way, more an improvement.

            So given this, I would probably not hold my breath that this is fixed anytime soon if people thinking this should be configurable or so contribute this fix.

            Show
            batmat Baptiste Mathus added a comment - I fail to understand how this issue can be set to "Important" and that importance not be explained in details. At first sight it seems like one of Minor or less priority. Not a bug by the way, more an improvement. So given this, I would probably not hold my breath that this is fixed anytime soon if people thinking this should be configurable or so contribute this fix.
            Hide
            alexf Alexandre Feblot added a comment - - edited

            In my case, I'm doing some dockerized stuffs with admin privileges in a specific subdir created for for this purpose, which is supposed to be removed afterwards. so, these additional temp dirs are created as root. Once outside of the container, the next time jenkins starts the job, it obviously fails to delete this root owned directory, so I have to take care of it "manually" from within the container. OK, our docker should maybe be set differently so that a container root user is not mapped to the global root user, but still, a deleteDir() function which does not delete all leftovers is not just a minor annoyance.

            Show
            alexf Alexandre Feblot added a comment - - edited In my case, I'm doing some dockerized stuffs with admin privileges in a specific subdir created for for this purpose, which is supposed to be removed afterwards. so, these additional temp dirs are created as root. Once outside of the container, the next time jenkins starts the job, it obviously fails to delete this root owned directory, so I have to take care of it "manually" from within the container. OK, our docker should maybe be set differently so that a container root user is not mapped to the global root user, but still, a deleteDir() function which does not delete all leftovers is not just a minor annoyance.
            Hide
            spebb23 Sarah LeGault added a comment -

            In our case, we are using multibranch pipeline jobs. We also see this issue occurring. This results in our drive filling up with folders that don't get cleaned up properly. Jenkins is running on a VM in Azure in our environment.

            Show
            spebb23 Sarah LeGault added a comment - In our case, we are using multibranch pipeline jobs. We also see this issue occurring. This results in our drive filling up with folders that don't get cleaned up properly. Jenkins is running on a VM in Azure in our environment.
            Hide
            kamikaze Oleg Korsak added a comment -

            Our Jenkins runs out of space once per day while having 50GB of space. Very intensive building occurs. Please fix cleanup

            Show
            kamikaze Oleg Korsak added a comment - Our Jenkins runs out of space once per day while having 50GB of space. Very intensive building occurs. Please fix cleanup
            Hide
            alex_vasiliev Alexander Vasiliev added a comment -

            i face this issue as well. @libs and @scripts directories contains too much files, so jenkins master runs out of inodes and must be cleaned manually

            Show
            alex_vasiliev Alexander Vasiliev added a comment - i face this issue as well. @libs and @scripts directories contains too much files, so jenkins master runs out of inodes and must be cleaned manually
            Hide
            efmoyano Ernesto Moyano added a comment -

            Hiten Prajapati, I used the Work Space Cleanup plugin, try this:

            ws(pwd() + "@tmp") {
              step([$class: 'WsCleanup'])
            }
            Show
            efmoyano Ernesto Moyano added a comment - Hiten Prajapati , I used the Work Space Cleanup plugin, try this: ws(pwd() + "@tmp" ) {   step([$class: 'WsCleanup' ]) }
            Hide
            jomega Alexander Trauzzi added a comment -

            Experiencing this as well on our master node.  Running on Azure, would rather not have to pay for infinite HDD space

            Show
            jomega Alexander Trauzzi added a comment - Experiencing this as well on our master node.  Running on Azure, would rather not have to pay for infinite HDD space
            Hide
            vanaken Niels van Aken added a comment -

            We're experiencing this as well, but while using `cleanWs()`. Also, our file / folder ownerships are simply `jenkins:jenkins` so no problem there as mentioned by Alexandre Feblot. The biggest problem about this bug is that it can create a massive overhead of inodes in some cases. We constantly experience `No space left on device` errors. We're now about to create a cronjob to periodically remove the @script / @tmp folders. I'd love to hear better workarounds. 

            Show
            vanaken Niels van Aken added a comment - We're experiencing this as well, but while using `cleanWs()`. Also, our file / folder ownerships are simply `jenkins:jenkins` so no problem there as mentioned by Alexandre Feblot . The biggest problem about this bug is that it can create a massive overhead of inodes in some cases. We constantly experience `No space left on device` errors. We're now about to create a cronjob to periodically remove the @script / @tmp folders. I'd love to hear better workarounds. 
            Hide
            vanaken Niels van Aken added a comment -

            I'd like to add, we've just added the following workaround, which seems to work pretty nicely for now.

            post {
              always {
                cleanWs()
                dir("${env.WORKSPACE}@tmp") {
                  deleteDir()
                }
                dir("${env.WORKSPACE}@script") {
                  deleteDir()
                }
                dir("${env.WORKSPACE}@script@tmp") {
                  deleteDir()
                }
              }
            }
            
            Show
            vanaken Niels van Aken added a comment - I'd like to add, we've just added the following workaround, which seems to work pretty nicely for now. post { always {     cleanWs() dir( "${env.WORKSPACE}@tmp" ) { deleteDir() } dir( "${env.WORKSPACE}@script" ) { deleteDir() } dir( "${env.WORKSPACE}@script@tmp" ) { deleteDir() } } }
            Hide
            jomega Alexander Trauzzi added a comment - - edited

            Is there any way to ensure the directories with this format on master get deleted as well?

            /var/lib/jenkins/workspace/application_PR-2375-5BLTOX3EZC6OGXGE7L2TFKXEY6Y3FYMDZ2VZ37EORCWPFIFMUKHA@script
            

            These directories are created when Jenkins goes to read and validate the Jenkinsfile initially. No further work appears to be done. These directories continue to accumulate until my master machine runs out of disk space. A bit of a nuisance.

            Show
            jomega Alexander Trauzzi added a comment - - edited Is there any way to ensure the directories with this format on master get deleted as well? / var /lib/jenkins/workspace/application_PR-2375-5BLTOX3EZC6OGXGE7L2TFKXEY6Y3FYMDZ2VZ37EORCWPFIFMUKHA@script These directories are created when Jenkins goes to read and validate the Jenkinsfile initially. No further work appears to be done. These directories continue to accumulate until my master machine runs out of disk space. A bit of a nuisance.
            Hide
            vanaken Niels van Aken added a comment -

            I'm not entirely sure what you mean? These workspace `@script` folders should be deleted with the snippet I posted.

            Show
            vanaken Niels van Aken added a comment - I'm not entirely sure what you mean? These workspace `@script` folders should be deleted with the snippet I posted.
            Hide
            jomega Alexander Trauzzi added a comment - - edited

            I suspect that only does it on the agent, not on the master. The path on my Jenkins master where files are accumulating is only referenced at the start of a build, and never again throughout.

            Show
            jomega Alexander Trauzzi added a comment - - edited I suspect that only does it on the agent , not on the master . The path on my Jenkins master where files are accumulating is only referenced at the start of a build, and never again throughout.
            Hide
            vanaken Niels van Aken added a comment -

            Ah I see, we only build on the same machine, I'm not sure how a master node would work. I'm not sure how to delete the dirs there. 

            Show
            vanaken Niels van Aken added a comment - Ah I see, we only build on the same machine, I'm not sure how a master node would work. I'm not sure how to delete the dirs there. 
            Hide
            hiten_prajapati Hiten Prajapati added a comment -

            please Help me this is still problem me cos we have many jobs with many branches in project that's make my ssd full. 

            Show
            hiten_prajapati Hiten Prajapati added a comment - please Help me this is still problem me cos we have many jobs with many branches in project that's make my ssd full. 
            Hide
            hyei Henry Yei added a comment -

            I run a simple cleanup bash script inside a job every once in awhile on my nodes.

            #!/usr/bin/env bash

            1. get to main job directory
              cd ..
              ls -al
              cd ..
              ls -al
            2. delete all @tmp files
              find . | grep @tmp$ | xargs -n1 rm -fr

            Of course you need to make sure you don't have any artifacts that you might match.

             

            Show
            hyei Henry Yei added a comment - I run a simple cleanup bash script inside a job every once in awhile on my nodes. #!/usr/bin/env bash get to main job directory cd .. ls -al cd .. ls -al delete all @tmp files find . | grep @tmp$ | xargs -n1 rm -fr Of course you need to make sure you don't have any artifacts that you might match.  
            Hide
            ericnelson Eric Nelson added a comment -

            The standard cleanWs() leaves at least the tmp directories behind on the agents. This is a real big problem for people using secret files , as it leaves them behind and they could be discoverable from another job! 

            I'm going to try the workaround posted by Niels van Aken

            For what its worth Alexander Trauzzi I'm not seeing the orphaned @scipt directories on my jenkins master as of version 2.108 for my multibranch pipeline jobs. 

            Show
            ericnelson Eric Nelson added a comment - The standard cleanWs() leaves at least the tmp directories behind on the agents. This is a real big problem for people using secret files , as it leaves them behind and they could be discoverable from another job!  I'm going to try the workaround posted by Niels van Aken .  For what its worth Alexander Trauzzi I'm not seeing the orphaned @scipt directories on my jenkins master as of version 2.108 for my multibranch pipeline jobs. 
            Hide
            gl1koz3 Edgars Batna added a comment - - edited

            Not sure why these directories would be required. There are system-wide temporary directories that should be used for this. The temporary directories, no matter what implementation detail, should not concern us users.

            Show
            gl1koz3 Edgars Batna added a comment - - edited Not sure why these directories would be required. There are system-wide temporary directories that should be used for this. The temporary directories, no matter what implementation detail, should not concern us users.
            Hide
            zuboram Cesar Ibarra added a comment -

            Im having the same issue, but also a folder named WORKSPACE_cleanup is created.

            Show
            zuboram Cesar Ibarra added a comment - Im having the same issue, but also a folder named WORKSPACE_cleanup is created.
            Hide
            smagness Steve Magness added a comment - - edited

            Has anyone solved this for the 'master - slave' setup? Our master is rapidly running out of disk space due to mercurial performing a full checkout into @script just to get the jenkinsfile (see https://issues.jenkins-ci.org/browse/JENKINS-50490). I'd like a solution that can be implemented within a scripted jenkinsfile eg:

            node ('slave') {
                // do useful build things first

                cleanWs() // clean up workspace on slave
            }
            cleanWs // clean workspace(s) on master (eg @script @libs directories)

            Show
            smagness Steve Magness added a comment - - edited Has anyone solved this for the 'master - slave' setup? Our master is rapidly running out of disk space due to mercurial performing a full checkout into @script just to get the jenkinsfile (see https://issues.jenkins-ci.org/browse/JENKINS-50490 ). I'd like a solution that can be implemented within a scripted jenkinsfile eg: node ('slave') {     // do useful build things first     cleanWs() // clean up workspace on slave } cleanWs // clean workspace(s) on master (eg @script @libs directories)
            Hide
            smagness Steve Magness added a comment -

            To answer my own question, similar to Niels van Aken solution but running on 'master'

            node ('slave') {
                // do useful build things first
                cleanWs() // clean up workspace on slave
            }
            node ('master') {
                dir("${env.WORKSPACE}@libs") {
                    deleteDir()
                }
                dir("${env.WORKSPACE}@script") {
                   deleteDir()
                }
            }

            Although this only cleans the directories when the stages on the slave succeed. You can use try..catch to catch exceptions from the slave stages and perform the cleanup in a finally block if required.

            Show
            smagness Steve Magness added a comment - To answer my own question, similar to Niels van Aken solution but running on 'master' node ('slave') {     // do useful build things first     cleanWs() // clean up workspace on slave } node ('master') {     dir("${env.WORKSPACE}@libs") {         deleteDir()     }     dir("${env.WORKSPACE}@script") {        deleteDir()     } } Although this only cleans the directories when the stages on the slave succeed. You can use try..catch to catch exceptions from the slave stages and perform the cleanup in a finally block if required.
            Hide
            skholod Sergii Kholod added a comment - - edited

            Together with "external workspace" plugin:

            node('master'){
            def extWs = exwsAllocate diskPoolId: 'p2000'
             exws(extWs) {}}
              sh "env | sort >test.txt"
              sleep time: 5, unit: 'MINUTES'
              cleanWs cleanWhenNotBuilt: false, notFailBuild: true
            }
             cleanWs cleanWhenNotBuilt: false, notFailBuild: true
            }

            the result is even worse:

            $ls -la /p2000/test-extws/
            total 0
            drwxr-xr-x 7 jenkins jenkins 62 Jun 25 13:42 .
            drwxr-xr-x 3 jenkins jenkins 23 Jun 25 11:38 ..
            drwxr-xr-x 2 jenkins jenkins 21 Jun 25 11:38 3
            drwxr-xr-x 2 jenkins jenkins 6 Jun 25 11:38 3@tmp
            drwxr-xr-x 2 jenkins jenkins 6 Jun 25 11:45 4@tmp
            drwxr-xr-x 2 jenkins jenkins 6 Jun 25 11:53 5@tmp
            drwxr-xr-x 2 jenkins jenkins 6 Jun 25 13:37 6@tmp

            Temp directories are flooding the workspace parent folder.

             

            Show
            skholod Sergii Kholod added a comment - - edited Together with "external workspace" plugin: node('master'){ def extWs = exwsAllocate diskPoolId: 'p2000'  exws(extWs) {}}   sh "env | sort >test.txt"   sleep time: 5, unit: 'MINUTES'   cleanWs cleanWhenNotBuilt: false, notFailBuild: true }  cleanWs cleanWhenNotBuilt: false, notFailBuild: true } the result is even worse: $ls -la /p2000/test-extws/ total 0 drwxr-xr-x 7 jenkins jenkins 62 Jun 25 13:42 . drwxr-xr-x 3 jenkins jenkins 23 Jun 25 11:38 .. drwxr-xr-x 2 jenkins jenkins 21 Jun 25 11:38 3 drwxr-xr-x 2 jenkins jenkins 6 Jun 25 11:38 3@tmp drwxr-xr-x 2 jenkins jenkins 6 Jun 25 11:45 4@tmp drwxr-xr-x 2 jenkins jenkins 6 Jun 25 11:53 5@tmp drwxr-xr-x 2 jenkins jenkins 6 Jun 25 13:37 6@tmp Temp directories are flooding the workspace parent folder.  
            Hide
            openjenkins J S added a comment - - edited

            Hello Guys,

            i have the following Jenkinsfile :

             

            pipeline {
                options { disableConcurrentBuilds() }
                agent { label 'vhost01' }    
            stages {
            [..AllStages..]
                }    post {
                    always {
                        cleanWs()
                    }
                }
            }
            

            I use multibranch pipeline and have the problem that the folder under my node "vhost01" are not deleted. Can anyone help ?

             

             

            Show
            openjenkins J S added a comment - - edited Hello Guys, i have the following Jenkinsfile :   pipeline { options { disableConcurrentBuilds() } agent { label 'vhost01' } stages { [..AllStages..] } post { always { cleanWs() } } } I use multibranch pipeline and have the problem that the folder under my node "vhost01" are not deleted. Can anyone help ?    
            Hide
            idanadar Idan Adar added a comment -

            +1

            Our agents remain unclean because of this.

            Show
            idanadar Idan Adar added a comment - +1 Our agents remain unclean because of this.
            Hide
            andreimuresianu Andrei Muresianu added a comment -

            +1 still a problem. The tmp directories do not add much value.

            Show
            andreimuresianu Andrei Muresianu added a comment - +1 still a problem. The tmp directories do not add much value.
            Hide
            karlparry Karl Parry added a comment - - edited

            +1 we now have about 200-300 @tmp/@script folders being created every day across several slave servers.

            -EDIITED-

            Will add the script snippet provided above for current jobs for now

            Show
            karlparry Karl Parry added a comment - - edited +1 we now have about 200-300 @tmp/@script folders being created every day across several slave servers. - EDIITED - Will add the  script snippet provided above for current jobs for now
            Hide
            jlpinardon jlpinardon added a comment -

            +1 I have added in a post always block a set of folderDelete operation... As far as I have only one slave, it is sustainable, but it will become ugly when using a label referencing several slaves.

            Show
            jlpinardon jlpinardon added a comment - +1 I have added in a post always block a set of folderDelete operation... As far as I have only one slave, it is sustainable, but it will become ugly when using a label referencing several slaves.
            Hide
            alexander_samoylov Alexander Samoylov added a comment - - edited

            Baptiste Mathus wrote: "Not a bug by the way, more an improvement."
            I strongly disagree. Jenkins creates the @tmp automatically and stores there temporary files. Therefore it should be removed also automatically by Jenkins.
            Each tool that produces temporary data should be responsible for its removal. It is easy as ABC.
            Following your logic, memory leaks are also "not a bugs"...

            +1 for the fix (which must be trivial)

            Update: I confirm that the workaround dir(<dir> + '@tmp')

            { deleteDir() }

            is working. Luckily it does not create the nested @tmp@tmp. Thank you, Niels van Aken.

            Show
            alexander_samoylov Alexander Samoylov added a comment - - edited Baptiste Mathus wrote: "Not a bug by the way, more an improvement." I strongly disagree. Jenkins creates the @tmp automatically and stores there temporary files. Therefore it should be removed also automatically by Jenkins. Each tool that produces temporary data should be responsible for its removal. It is easy as ABC. Following your logic, memory leaks are also "not a bugs"... +1 for the fix (which must be trivial) Update: I confirm that the workaround dir(<dir> + '@tmp') { deleteDir() } is working. Luckily it does not create the nested @tmp@tmp. Thank you, Niels van Aken .

              People

              • Assignee:
                Unassigned
                Reporter:
                hiten_prajapati Hiten Prajapati
              • Votes:
                60 Vote for this issue
                Watchers:
                76 Start watching this issue

                Dates

                • Created:
                  Updated: