Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54225

Using disableDeferredWipeout within pipeline job results in no deleted workspace

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Minor
    • Resolution: Cannot Reproduce
    • Component/s: ws-cleanup-plugin
    • Labels:
      None
    • Environment:
      wsCleanup 0.36 and Jenkins ver. 2.138.2
      Jenkins node is Fedora 28
    • Similar Issues:

      Description

      We are using pipeline and at the end of the pipeline we have a stage as below. We added the disableDeferredWipeout recently to make sure that all disk space of the workspace is reclaimed before the job ends. We now notice that the workspace is not deleted anymore, when we remove the disableDeferredWipeout the workspace directory is gone again. Have tried to find some logs but can't find anything.

      stage('Post cleanup') {

      step([$class: 'WsCleanup', disableDeferredWipeout: true])

      }

      In the log we only get the message below but at the end the workspace is still there

      [WS-CLEANUP] Deleting project workspace...
      [WS-CLEANUP] Deferred wipeout is disabled by the job configuration...
      [WS-CLEANUP] done

       

      As reproducer you can use the pipeline job below, only adapt the node argument. After the pipeline job has finished we would expect to have an empty workspace but with disableDeferredWipeout:true the buildtools directory is still there, without the argument the workspace is deleted

      node('Scarab_fc28_x86_64_01') {
      stage('Pre cleanup')

      { cleanWs() }

      stage('Buildtools checkout') {
      dir ('buildtools')

      { git changelog: false, poll: false, url: 'https://github.com/DOCGroup/MPC', shallow: true }

      }

      stage('Post cleanup')

      { cleanWs disableDeferredWipeout:true // cleanWs() }

      }

       

        Attachments

          Issue Links

            Activity

            jwillemsen Johnny Willemsen created issue -
            jwillemsen Johnny Willemsen made changes -
            Field Original Value New Value
            Environment wsCleanup 0.36 and Jenkins ver. 2.138.2
            Jenkins node is Fedora 28
            jwillemsen Johnny Willemsen made changes -
            Description We are using pipeline and at the end of the pipeline we have a stage as below. We added the disableDeferredWipeout recently to make sure that all disk space of the workspace is reclaimed before the job ends. We now notice that the workspace is not deleted anymore, when we remove the disableDeferredWipeout the workspace directory is gone again. Have tried to find some logs but can't find anything.

            {{stage('Post cleanup') {}}

            {{step([$class: 'WsCleanup', disableDeferredWipeout: true])}}

            {{}}}

            In the log we only get the message below but at the end the workspace is still there

            {{[WS-CLEANUP] Deleting project workspace...}}
            {{[WS-CLEANUP] Deferred wipeout is disabled by the job configuration...}}
            {{[WS-CLEANUP] done}}
            We are using pipeline and at the end of the pipeline we have a stage as below. We added the disableDeferredWipeout recently to make sure that all disk space of the workspace is reclaimed before the job ends. We now notice that the workspace is not deleted anymore, when we remove the disableDeferredWipeout the workspace directory is gone again. Have tried to find some logs but can't find anything.

            {{stage('Post cleanup') {}}

            {{step([$class: 'WsCleanup', disableDeferredWipeout: true])}}

            {{}}}

            In the log we only get the message below but at the end the workspace is still there

            {{[WS-CLEANUP] Deleting project workspace...}}
             {{[WS-CLEANUP] Deferred wipeout is disabled by the job configuration...}}
             {{[WS-CLEANUP] done}}

             

            As reproducer you can use the pipeline job below, only adapt the node argument. After the pipeline job has finished we would expect to have an empty workspace but with disableDeferredWipeout:true the buildtools directory is still there, without the argument the workspace is deleted

            {{node('Scarab_fc28_x86_64_01') {}}
            {{  stage('Pre cleanup') {}}
            {{    cleanWs()}}
            {{  }}}
            {{  stage('Buildtools checkout') {}}
            {{    dir ('buildtools') {}}
            {{      git changelog: false, poll: false, url: 'https://github.com/DOCGroup/MPC', shallow: true}}
            {{    }}}
            {{  }}}
             
            {{ stage('Post cleanup') {}}
            {{ cleanWs disableDeferredWipeout:true}}
            {{// cleanWs()}}
            {{ }}}
            {{}}}
            jwillemsen Johnny Willemsen made changes -
            Description We are using pipeline and at the end of the pipeline we have a stage as below. We added the disableDeferredWipeout recently to make sure that all disk space of the workspace is reclaimed before the job ends. We now notice that the workspace is not deleted anymore, when we remove the disableDeferredWipeout the workspace directory is gone again. Have tried to find some logs but can't find anything.

            {{stage('Post cleanup') {}}

            {{step([$class: 'WsCleanup', disableDeferredWipeout: true])}}

            {{}}}

            In the log we only get the message below but at the end the workspace is still there

            {{[WS-CLEANUP] Deleting project workspace...}}
             {{[WS-CLEANUP] Deferred wipeout is disabled by the job configuration...}}
             {{[WS-CLEANUP] done}}

             

            As reproducer you can use the pipeline job below, only adapt the node argument. After the pipeline job has finished we would expect to have an empty workspace but with disableDeferredWipeout:true the buildtools directory is still there, without the argument the workspace is deleted

            {{node('Scarab_fc28_x86_64_01') {}}
            {{  stage('Pre cleanup') {}}
            {{    cleanWs()}}
            {{  }}}
            {{  stage('Buildtools checkout') {}}
            {{    dir ('buildtools') {}}
            {{      git changelog: false, poll: false, url: 'https://github.com/DOCGroup/MPC', shallow: true}}
            {{    }}}
            {{  }}}
             
            {{ stage('Post cleanup') {}}
            {{ cleanWs disableDeferredWipeout:true}}
            {{// cleanWs()}}
            {{ }}}
            {{}}}
            We are using pipeline and at the end of the pipeline we have a stage as below. We added the disableDeferredWipeout recently to make sure that all disk space of the workspace is reclaimed before the job ends. We now notice that the workspace is not deleted anymore, when we remove the disableDeferredWipeout the workspace directory is gone again. Have tried to find some logs but can't find anything.

            {{stage('Post cleanup') {}}

            {{step([$class: 'WsCleanup', disableDeferredWipeout: true])}}

            {{}}}

            In the log we only get the message below but at the end the workspace is still there

            {{[WS-CLEANUP] Deleting project workspace...}}
             {{[WS-CLEANUP] Deferred wipeout is disabled by the job configuration...}}
             {{[WS-CLEANUP] done}}

             

            As reproducer you can use the pipeline job below, only adapt the node argument. After the pipeline job has finished we would expect to have an empty workspace but with disableDeferredWipeout:true the buildtools directory is still there, without the argument the workspace is deleted

            node('Scarab_fc28_x86_64_01') {
             stage('Pre cleanup') {
             cleanWs()
             }
             stage('Buildtools checkout') {
             dir ('buildtools') {
             git changelog: false, poll: false, url: 'https://github.com/DOCGroup/MPC', shallow: true
             }
             }
             
             stage('Post cleanup') {
             cleanWs disableDeferredWipeout:true
            // cleanWs()
             }
            }

             
            Hide
            broussar Adam Brousseau added a comment - - edited

            Also seeing this issue. Seems to leave behind the workspace (and .git folder) but the Git repository is "corrupt" so the next build will fail.

            Running shell script
            + git clean -ffxd
            fatal: Not a git repository (or any parent up to mount point /home/jenkins)
            Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).

            Show
            broussar Adam Brousseau added a comment - - edited Also seeing this issue. Seems to leave behind the workspace (and .git folder) but the Git repository is "corrupt" so the next build will fail. Running shell script + git clean -ffxd fatal: Not a git repository (or any parent up to mount point /home/jenkins) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
            roufique2 Roufique Hossain made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            olivergondza Oliver Gondža made changes -
            Status In Progress [ 3 ] Open [ 1 ]
            olivergondza Oliver Gondža made changes -
            Assignee Oliver Gondža [ olivergondza ] Pavel Janoušek [ pajasoft ]
            Hide
            olivergondza Oliver Gondža added a comment -

            Pavel, can you please have a look? Thanks!

            Show
            olivergondza Oliver Gondža added a comment - Pavel, can you please have a look? Thanks!
            Hide
            pajasoft Pavel Janoušek added a comment -

            Guys,
            thank you for your report of this issue.
            Unfortunately I've tried your reproducer or similar setup several times but no luck on my side - simply I can't reproduce your issue, everything works like charm with expected result - after job execution, the whole workspace is cleaned-up, only the system WORKSPACE directory (<location defined in node setup - "remoteFS">/workspace) itself remains.
            There is a major difference if deferred wipeout is used or not. If it is used, it's taken an improved version of cleaner made by ogondza, if not the legacy original cleaner is taken. The impl. of both is far different.

            My idea here is some kind of clashing with other plugin (like monitor, statistics etc.) on your instance which is manifested by the fact the directory is processed but erased only partially ("Git repo is corrupted" is my clue), if there remains at least one file, system refuses to delete the directory. The main difference during cleaning is the fact that the legacy cleaner is executed and it finishes before the run is completed but enhanced Wipeout (deferred) usually runs after the run finishes - something else blocks files to be erased in the first case.

            Show
            pajasoft Pavel Janoušek added a comment - Guys, thank you for your report of this issue. Unfortunately I've tried your reproducer or similar setup several times but no luck on my side - simply I can't reproduce your issue, everything works like charm with expected result - after job execution, the whole workspace is cleaned-up, only the system WORKSPACE directory (<location defined in node setup - "remoteFS">/workspace) itself remains. There is a major difference if deferred wipeout is used or not. If it is used, it's taken an improved version of cleaner made by ogondza, if not the legacy original cleaner is taken. The impl. of both is far different. My idea here is some kind of clashing with other plugin (like monitor, statistics etc.) on your instance which is manifested by the fact the directory is processed but erased only partially ("Git repo is corrupted" is my clue), if there remains at least one file, system refuses to delete the directory. The main difference during cleaning is the fact that the legacy cleaner is executed and it finishes before the run is completed but enhanced Wipeout (deferred) usually runs after the run finishes - something else blocks files to be erased in the first case.
            Hide
            pajasoft Pavel Janoušek added a comment - - edited

            There is a similar report in JENKINS-55443, but the issue reported there doesn't have any relation to WS Cleanup plugin, but pipeline, can you try the similar setup on you instance of Jenkins to verify if it is the same case please?

            Update: I can't reproduce either JENKINS-55443 issue on my instance - Jenkins core version 2.121.3, SSH Slave Plugin version 1.28, Pipeline stage step plugin version 2.3 if interested.

            Show
            pajasoft Pavel Janoušek added a comment - - edited There is a similar report in JENKINS-55443 , but the issue reported there doesn't have any relation to WS Cleanup plugin, but pipeline, can you try the similar setup on you instance of Jenkins to verify if it is the same case please? Update: I can't reproduce either JENKINS-55443 issue on my instance - Jenkins core version 2.121.3, SSH Slave Plugin version 1.28, Pipeline stage step plugin version 2.3 if interested.
            pajasoft Pavel Janoušek made changes -
            Link This issue is related to JENKINS-55443 [ JENKINS-55443 ]
            Hide
            pajasoft Pavel Janoušek added a comment -

            Just an update...

            Finally I'm able to reproduce an issue and need to verify from you if my result is the same as yours...

            Simple "cleanWs()" -> the whole project dir content is deleted
            "cleanWs disableDeferredWipeout: true" -> the files in project dir (+ subdirs) are deleted except all directories themself
            "cleanWs disableDeferredWipeout: true, deleteDirs: true" -> the same result as from simple "cleanWs()"

            This behavior is correct because when disableDeferredWipeout=false (the default by the way - align with backward compatibility) then Wipeout impl. is used -> everything is deleted.

            On the other hand when disableDeferredWipeout=true, the old legacy workspace cleaner is used and it has more parameters how to control its behavior. One of them is deleteDirs, which is set to false by default and you have to set it to true if you want to have the same result as with Wipeout.

            My reproducer:

            node('RHEL7') {
                stage('Pre cleanup') {
                    cleanWs() 
            
                    sh "ls -lR"
                }
            
                stage('Buildtools checkout') {
                    dir ('buildtools') { 
                        git changelog: false, poll: false, url: 'https://github.com/DOCGroup/MPC', shallow: true 
                    }
            
                    sh "ls -lR"
                }
            
                stage('Post cleanup') { 
            //        cleanWs() 
            
                    cleanWs disableDeferredWipeout: true, deleteDirs: true
            
                    sh "ls -lR"
                }
            }
            Show
            pajasoft Pavel Janoušek added a comment - Just an update... Finally I'm able to reproduce an issue and need to verify from you if my result is the same as yours... Simple " cleanWs() " -> the whole project dir content is deleted " cleanWs disableDeferredWipeout: true " -> the files in project dir (+ subdirs) are deleted except all directories themself " cleanWs disableDeferredWipeout: true, deleteDirs: true " -> the same result as from simple " cleanWs() " This behavior is correct because when disableDeferredWipeout=false (the default by the way - align with backward compatibility) then Wipeout impl. is used -> everything is deleted. On the other hand when disableDeferredWipeout=true , the old legacy workspace cleaner is used and it has more parameters how to control its behavior. One of them is deleteDirs , which is set to false by default and you have to set it to true if you want to have the same result as with Wipeout. My reproducer: node( 'RHEL7' ) { stage( 'Pre cleanup' ) { cleanWs() sh "ls -lR" } stage( 'Buildtools checkout' ) { dir ( 'buildtools' ) { git changelog: false , poll: false , url: 'https: //github.com/DOCGroup/MPC' , shallow: true } sh "ls -lR" } stage( 'Post cleanup' ) { // cleanWs() cleanWs disableDeferredWipeout: true , deleteDirs: true sh "ls -lR" } }
            Hide
            jwillemsen Johnny Willemsen added a comment -

            Did several builds today with 'deleteDirs' added and it seems to work, this night a lot more builds will run but it looks you found the cause

            Show
            jwillemsen Johnny Willemsen added a comment - Did several builds today with 'deleteDirs' added and it seems to work, this night a lot more builds will run but it looks you found the cause
            Hide
            jwillemsen Johnny Willemsen added a comment -

            I can confirm that adding deleteDirs solved the issue of the partial delete.

            As workaround on Windows we use `rm -rf` from MinGW, it looks that we have some files in the workspace which are still in use in some builds, rm can remove them without problems, with get the error below, is there a forced delete options (... is a full path)

            ERROR: Cannot delete workspace :Unable to delete '...ACE_ETCL_Parserd.dll’. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
            Error when executing always post condition:
            hudson.AbortException: Cannot delete workspace: Unable to delete '...\ACE_ETCL_Parserd.dll’. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
            at hudson.plugins.ws_cleanup.WsCleanup.perform(WsCleanup.java:244)
            at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)

            Show
            jwillemsen Johnny Willemsen added a comment - I can confirm that adding deleteDirs solved the issue of the partial delete. As workaround on Windows we use `rm -rf` from MinGW, it looks that we have some files in the workspace which are still in use in some builds, rm can remove them without problems, with get the error below, is there a forced delete options (... is a full path) ERROR: Cannot delete workspace :Unable to delete '...ACE_ETCL_Parserd.dll’. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. Error when executing always post condition: hudson.AbortException: Cannot delete workspace: Unable to delete '...\ACE_ETCL_Parserd.dll’. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. at hudson.plugins.ws_cleanup.WsCleanup.perform(WsCleanup.java:244) at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
            Hide
            pajasoft Pavel Janoušek added a comment -

            Thank you for your confirmation Johnny Willemsen. If there isn't a complain soon, I'll close this issue without any additional action item.

            Show
            pajasoft Pavel Janoušek added a comment - Thank you for your confirmation Johnny Willemsen . If there isn't a complain soon, I'll close this issue without any additional action item.
            Hide
            jwillemsen Johnny Willemsen added a comment -

            Shouldn't the plugin set deleteDirs to true in case we do a deferred delete? Or at least have some documentation extensions to mention this, this is confusing for people who think they just disable the async delete and get strange errors due to that

            Show
            jwillemsen Johnny Willemsen added a comment - Shouldn't the plugin set deleteDirs to true in case we do a deferred delete? Or at least have some documentation extensions to mention this, this is confusing for people who think they just disable the async delete and get strange errors due to that
            Hide
            pajasoft Pavel Janoušek added a comment -

            Johnny Willemsen Good idea, I've put a note to the wiki.

            Show
            pajasoft Pavel Janoušek added a comment - Johnny Willemsen Good idea, I've put a note to the wiki.
            Hide
            pajasoft Pavel Janoušek added a comment -

            The same can be achieved with the proper plugin setting.

            Show
            pajasoft Pavel Janoušek added a comment - The same can be achieved with the proper plugin setting.
            pajasoft Pavel Janoušek made changes -
            Status Open [ 1 ] Resolved [ 5 ]
            Resolution Cannot Reproduce [ 5 ]

              People

              • Assignee:
                pajasoft Pavel Janoušek
                Reporter:
                jwillemsen Johnny Willemsen
              • Votes:
                1 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: