Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-12783

Delete workspace before build starts ignores Exlude patterns (for directories)

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Component/s: ws-cleanup-plugin
    • Environment:
    • Similar Issues:

      Description

      I need to exclude some directories from being deleted on workspace cleanup. It always deleted the entire workspace, pattern is always ignored.
      I used several combinations: */DIRNAME/., DIRNAME/., DIRNAME, *DIRNAME with or without checking "Apply pattern also on directories"

        Attachments

          Issue Links

            Activity

            bavoiculescu Voiculescu Bogdan Alexandru created issue -
            Hide
            vjuranek vjuranek added a comment -

            Hi,
            I cannot reproduce it (to be more specific, DIRNAME/** keeps DIRNAME and all files in it, DIRNAME keeps only this directory, no matter if "apply to directories" is switched on or off). Do you still have this issue?

            Show
            vjuranek vjuranek added a comment - Hi, I cannot reproduce it (to be more specific, DIRNAME/** keeps DIRNAME and all files in it, DIRNAME keeps only this directory, no matter if "apply to directories" is switched on or off). Do you still have this issue?
            Hide
            rrrrando Rando Õispuu added a comment -

            When "Apply pattern also on directories" everything gets deleted regardless of the Exclude pattern. I use **/DIR/DIR/ it works when "Apply pattern also on directories" is not checked.

            Show
            rrrrando Rando Õispuu added a comment - When "Apply pattern also on directories" everything gets deleted regardless of the Exclude pattern. I use **/DIR/DIR/ it works when "Apply pattern also on directories" is not checked.
            vjuranek vjuranek made changes -
            Field Original Value New Value
            Assignee vjuranek [ vjuranek ] Lucie Votypkova [ lvotypkova ]
            Hide
            jmartenstein Justin Martenstein added a comment -

            I'm having the same issue. I have a workspace with "foo" and "bar" directories. I check the checkbox for "Delete workspace before build starts", and then underneath I add a directory and specify "Exclude" and then "foo". I also check the checkbox for "Apply patterns on directories". Then when I run my build, it cleans out BOTH the "foo" and "bar" directories. I would expect that it would delete "bar", but not "foo". But no matter what I specify for an "exclude", it always gets ignored.

            Show
            jmartenstein Justin Martenstein added a comment - I'm having the same issue. I have a workspace with "foo" and "bar" directories. I check the checkbox for "Delete workspace before build starts", and then underneath I add a directory and specify "Exclude" and then "foo". I also check the checkbox for "Apply patterns on directories". Then when I run my build, it cleans out BOTH the "foo" and "bar" directories. I would expect that it would delete "bar", but not "foo". But no matter what I specify for an "exclude", it always gets ignored.
            Hide
            pacey James Pace added a comment -

            I am also having issues with the exclude pattern. I do not want to delete a directory called "node_modules", or any of its sub directories or files, in the root of the workspace.

            I have added an exclude pattern "*/node_modules//" but it still gets deleted. It doesn't seem to matter if the delete directories checkbox is selected or not.

            Show
            pacey James Pace added a comment - I am also having issues with the exclude pattern. I do not want to delete a directory called "node_modules", or any of its sub directories or files, in the root of the workspace. I have added an exclude pattern "* /node_modules/ / " but it still gets deleted. It doesn't seem to matter if the delete directories checkbox is selected or not.
            Hide
            dackerman David Ackerman added a comment - - edited

            I had the same issue, and I think I figured out what was going wrong (after cloning the repo and adding enough logging statements).

            Excluding (not-deleting) everything under a toplevel directory works fine, i.e "toplevel/**" excludes everything under the folder "$WORKSPACE/toplevel". However, excluding something in a subdirectory does not work - i.e. "toplevel/asubdir/**" ends up deleting everything. This happens when you have "apply pattern also on directories" turned on. The reason is that DirectoryScanner will find directories to delete that might be parents of files that you've excluded, but the plugin deletes those folders recursively (i.e. rm -rf). The way to fix this is to add "toplevel" explicitly as an exclude to prevent it from deleting the parent directory as well. The reason I have the directories checkbox turned on is because I don't want the ".git" directory (or any extra directories) lying around and confusing jenkins when it builds the next time.

            My project looks something like this (in a multi-module maven project, where node_modules/ resides in several subdirectories).

            * folder-1
              - node_modules
            * folder-2
              - node_modules
            * folder-3
            * folder-4
              - node_modules
            

            My Configuration looks like this:

            EXCLUDE  **/node_modules/**
            EXCLUDE folder-1
            EXCLUDE folder-2
            EXCLUDE folder-4
            

            And that seems to work as expected. It seems like the plugin needs to be more careful about the directories it deletes recursively - it needs to avoid deleting a parent that would delete excluded files (or maybe just do "rm" instead of "rm -r" so it won't delete the directory if it has otherwise excluded files in it). I would offer to create a patch, but my Intellij doesn't seem to be resolving dependencies in the project properly (although maven commands work fine), and I am not sure what others would consider the ideal behavior. At the very least, I think more documentation in the plugin help and/or plugin page would save people a lot of time.

            Show
            dackerman David Ackerman added a comment - - edited I had the same issue, and I think I figured out what was going wrong (after cloning the repo and adding enough logging statements). Excluding (not-deleting) everything under a toplevel directory works fine, i.e "toplevel/**" excludes everything under the folder "$WORKSPACE/toplevel". However, excluding something in a subdirectory does not work - i.e. "toplevel/asubdir/**" ends up deleting everything. This happens when you have "apply pattern also on directories" turned on. The reason is that DirectoryScanner will find directories to delete that might be parents of files that you've excluded, but the plugin deletes those folders recursively (i.e. rm -rf). The way to fix this is to add "toplevel" explicitly as an exclude to prevent it from deleting the parent directory as well. The reason I have the directories checkbox turned on is because I don't want the ".git" directory (or any extra directories) lying around and confusing jenkins when it builds the next time. My project looks something like this (in a multi-module maven project, where node_modules/ resides in several subdirectories). * folder-1 - node_modules * folder-2 - node_modules * folder-3 * folder-4 - node_modules My Configuration looks like this: EXCLUDE **/node_modules/** EXCLUDE folder-1 EXCLUDE folder-2 EXCLUDE folder-4 And that seems to work as expected. It seems like the plugin needs to be more careful about the directories it deletes recursively - it needs to avoid deleting a parent that would delete excluded files (or maybe just do "rm" instead of "rm -r" so it won't delete the directory if it has otherwise excluded files in it). I would offer to create a patch, but my Intellij doesn't seem to be resolving dependencies in the project properly (although maven commands work fine), and I am not sure what others would consider the ideal behavior. At the very least, I think more documentation in the plugin help and/or plugin page would save people a lot of time.
            ophyte O. phyTe made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            ophyte O. phyTe made changes -
            Status In Progress [ 3 ] Open [ 1 ]
            Hide
            lsanders Larry Sanders added a comment - - edited

            I spent quite a bit of time on this very issue (ironically, also trying to avoid the node penalty).

            Of course leaving empty directories laying around is problematic, specifically for .repository when using a private maven repository for your build.

            Show
            lsanders Larry Sanders added a comment - - edited I spent quite a bit of time on this very issue (ironically, also trying to avoid the node penalty). Of course leaving empty directories laying around is problematic, specifically for .repository when using a private maven repository for your build.
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 143241 ] JNJira + In-Review [ 175847 ]
            Hide
            spikeynick Nick Romanyshyn added a comment - - edited

            I don't know if this applies to the OP, but I had a very similar issue. I specified patterns to exclude, but it would completely delete everything in the workspace. If I instead specified a pattern to include that would work as expected. In my case the exclude actually was working correctly. I verified this by adding a custom delete command which was just a simple python script that appended the name of the file to be deleted to a text file (so it didn't actually do any deleting). It turned out that I wasn't excluding the .git directory, so that was getting deleted. You can't clone into a non empty directory, so Jenkins had to remove the workspace completely in order to do a new clone. I haven't tried with .svn, but I would expect similar results. One other thing to note is that if that you don't specify an include pattern then the plugin automatically uses the pattern **/* So if you have included anything that doesn't match the .git directory that will also spare it from deletion.

            tl;dr Make sure you also exclude your .git directory

            Show
            spikeynick Nick Romanyshyn added a comment - - edited I don't know if this applies to the OP, but I had a very similar issue. I specified patterns to exclude, but it would completely delete everything in the workspace. If I instead specified a pattern to include that would work as expected. In my case the exclude actually was working correctly. I verified this by adding a custom delete command which was just a simple python script that appended the name of the file to be deleted to a text file (so it didn't actually do any deleting). It turned out that I wasn't excluding the .git directory, so that was getting deleted. You can't clone into a non empty directory, so Jenkins had to remove the workspace completely in order to do a new clone. I haven't tried with .svn, but I would expect similar results. One other thing to note is that if that you don't specify an include pattern then the plugin automatically uses the pattern ** / * So if you have included anything that doesn't match the .git directory that will also spare it from deletion. tl;dr Make sure you also exclude your .git directory
            Hide
            joyeux pierre Salaun added a comment -

            Hi,
            I'm having the same issue, Include work fine but not Exclude.
            I have a maven project with custom workspace and a private maven repository local to workspace.
            I want to keep the ".repository" directory but it is delete by the "Delete workspace before build starts" function.
            I have add a Exclude ".respository/**" but it isn't work.
            Can you help me?

            Show
            joyeux pierre Salaun added a comment - Hi, I'm having the same issue, Include work fine but not Exclude. I have a maven project with custom workspace and a private maven repository local to workspace. I want to keep the ".repository" directory but it is delete by the "Delete workspace before build starts" function. I have add a Exclude ".respository/**" but it isn't work. Can you help me?
            Hide
            grim Grim Reaper added a comment - - edited

            Jenkins Versions: 2.60.3 & 2.73.3 (WAR deployments on WildFly)
            Plugin Version: 0.34
            Platform: Linux
            Plugins:
            Git client plugin 2.6.0
            Git plugin 3.6.4

            Details:
            No issues when the SCM is CVS. However we are migrating from CVS to GIT.
            When I configure a freestyle job to use GIT as the SCM and with "Delete workspace before build starts" checked it ignores any "Advanced" options I use (I'm trying to exclude ' *.build.* ' and a 'reports/**' directory) and just deletes everything in the entire workspace.

            In troubleshooting I also testing the Post Build Action > "Delete workspace when build is done". This half works. I can tell it to "Exclude" specific files, but as soon as the "Apply pattern also on directories" is checked it goes back to the above behavior of deleting everything regardless of excludes.

            Really hoping someone can have a look at this and find the root cause and a fix.

            Thanks in advance!

            Edit:
            I forgot to add that the GIT install is on a windows machine and the repo is located through a UNC share. Not that I think this has any bearing as the repo pull/clone works fine. Just trying to provide as much info as possible.

            Show
            grim Grim Reaper added a comment - - edited Jenkins Versions: 2.60.3 & 2.73.3 (WAR deployments on WildFly) Plugin Version: 0.34 Platform: Linux Plugins: Git client plugin 2.6.0 Git plugin 3.6.4 Details: No issues when the SCM is CVS. However we are migrating from CVS to GIT. When I configure a freestyle job to use GIT as the SCM and with "Delete workspace before build starts" checked it ignores any "Advanced" options I use (I'm trying to exclude ' * .build. * ' and a 'reports/**' directory) and just deletes everything in the entire workspace. In troubleshooting I also testing the Post Build Action > "Delete workspace when build is done". This half works. I can tell it to "Exclude" specific files, but as soon as the "Apply pattern also on directories" is checked it goes back to the above behavior of deleting everything regardless of excludes. Really hoping someone can have a look at this and find the root cause and a fix. Thanks in advance! Edit: I forgot to add that the GIT install is on a windows machine and the repo is located through a UNC share. Not that I think this has any bearing as the repo pull/clone works fine. Just trying to provide as much info as possible.
            grim Grim Reaper made changes -
            Environment Linux, Jenkins ver. 1.451, plugin 0.7, job running on master or slave OP - Linux, Jenkins ver. 1.451, plugin 0.7, job running on master or slave
            Grim -
            grim Grim Reaper made changes -
            Environment OP - Linux, Jenkins ver. 1.451, plugin 0.7, job running on master or slave
            Grim -
            OP - Linux, Jenkins ver. 1.451, plugin 0.7, job running on master or slave
            Grim:
            grim Grim Reaper made changes -
            Environment OP - Linux, Jenkins ver. 1.451, plugin 0.7, job running on master or slave
            Grim:
            OP - Linux, Jenkins ver. 1.451, plugin 0.7, job running on master or slave
            Grim:
               Jenkins Versions: 2.60.3 & 2.73.3 (WAR deployments on WildFly)
               Plugin Version: 0.34
               Platform: Linux
               Plugins:
                  Git client plugin 2.6.0
                  Git plugin 3.6.4
            grim Grim Reaper made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            grim Grim Reaper made changes -
            Labels bug plugin scm workspace
            Hide
            grim Grim Reaper added a comment -

            Assigning to you as the plugins page indicates you are the ws-cleanup maintainer.
            Also for visibility.

            Show
            grim Grim Reaper added a comment - Assigning to you as the plugins page indicates you are the ws-cleanup maintainer. Also for visibility.
            grim Grim Reaper made changes -
            Assignee Lucie Votypkova [ lvotypkova ] Oliver Gondža [ olivergondza ]
            Hide
            grim Grim Reaper added a comment -

            For those looking for a work around, I've use the following plugin:
            https://plugins.jenkins.io/preSCMbuildstep

            With the following Shell script to do the same thing as WS-cleanup.

            echo "Deleting..."
            ls ${WORKSPACE}/ | egrep -v '*.build.*|suppressions.xml|reports' 
            ls ${WORKSPACE}/ | egrep -v '*.build.*|suppressions.xml|reports' | xargs rm -rf
            

             

            The above deletes everything except the items in the "egrep" command (excludes).

            Show
            grim Grim Reaper added a comment - For those looking for a work around, I've use the following plugin: https://plugins.jenkins.io/preSCMbuildstep With the following Shell script to do the same thing as WS-cleanup. echo "Deleting..." ls ${WORKSPACE}/ | egrep -v '*.build.*|suppressions.xml|reports' ls ${WORKSPACE}/ | egrep -v '*.build.*|suppressions.xml|reports' | xargs rm -rf   The above deletes everything except the items in the "egrep" command (excludes).
            Hide
            tpbradt Tim Bradt added a comment -

            This bug is unfortunate.  We're trying to preserve our IntelliJ project files so we don't have to restore project settings every time we build.  This should really be addressed.

            +1 vote

            Show
            tpbradt Tim Bradt added a comment - This bug is unfortunate.  We're trying to preserve our IntelliJ project files so we don't have to restore project settings every time we build.  This should really be addressed. +1 vote
            Hide
            grim Grim Reaper added a comment - - edited

            Updated my test server to Jenkins 2.89.4 and we are now using "Gitea" as our backend SCM.

            Still the same issue.

            Also update on the work around. Found out it wasn't working as intended. So now I shuffle files around to make it work. This is all on Linux.

            Steps to configure the job:

            1. Under "Source Code management" (Git), select "Wipe out repo & force clone" under "Additional behaviours".
            2. Under "Build Environment", note you still need to have the preSCMBuildStep plugin and check "Run buildstep before SCM runs" and add "Execute shell", then use the following to "Exclude files" (You may need to modify for your needs):
              # Move build files to temp location before workspace wipe
              ls -Al "${WORKSPACE}"
              [ ! -d "/tmp/${BUILD_TAG}/" ] && mkdir -p "/tmp/${BUILD_TAG}/"
              mv -v "${WORKSPACE}"/*.build.* "${WORKSPACE}"/suppressions.xml "/tmp/${BUILD_TAG}/"
              ls -Al "/tmp/${BUILD_TAG}/"
              
            1. Under the "Build" section of your job. Add "Execute shell" and move it to the top. In the textbox add the following script commands to move your files back into the workspace:
              # Move build files back to workspace before build runs 
              mv -v "/tmp/${BUILD_TAG}"/* "${WORKSPACE}" 
              ls -Al "${WORKSPACE}" 
              rm -rfv "/tmp/${BUILD_TAG}/"
              

            Off to try the latest and greatest Jenkins (Hopefully it can still build our legacy apps >.<).

             

            Show
            grim Grim Reaper added a comment - - edited Updated my test server to Jenkins 2.89.4 and we are now using "Gitea" as our backend SCM. Still the same issue. Also update on the work around. Found out it wasn't working as intended. So now I shuffle files around to make it work. This is all on Linux. Steps to configure the job: Under "Source Code management" (Git), select "Wipe out repo & force clone" under "Additional behaviours". Under "Build Environment", note you still need to have the preSCMBuildStep plugin and check "Run buildstep before SCM runs" and add "Execute shell", then use the following to "Exclude files" (You may need to modify for your needs): # Move build files to temp location before workspace wipe ls -Al "${WORKSPACE}" [ ! -d "/tmp/${BUILD_TAG}/" ] && mkdir -p "/tmp/${BUILD_TAG}/" mv -v "${WORKSPACE}" /*.build.* "${WORKSPACE}" /suppressions.xml "/tmp/${BUILD_TAG}/" ls -Al "/tmp/${BUILD_TAG}/" Under the "Build" section of your job. Add "Execute shell" and move it to the top. In the textbox add the following script commands to move your files back into the workspace: # Move build files back to workspace before build runs mv -v "/tmp/${BUILD_TAG}" /* "${WORKSPACE}" ls -Al "${WORKSPACE}" rm -rfv "/tmp/${BUILD_TAG}/" Off to try the latest and greatest Jenkins (Hopefully it can still build our legacy apps >.<).  
            grim Grim Reaper made changes -
            Link This issue relates to JENKINS-30179 [ JENKINS-30179 ]

              People

              • Assignee:
                olivergondza Oliver Gondža
                Reporter:
                bavoiculescu Voiculescu Bogdan Alexandru
              • Votes:
                8 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated: