Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-15418

Fails to clean windows workspace with long path

    Details

    • Similar Issues:

      Description

      I see that issue happening on long paths. A workaround was to use drive mapping in order to reduce the length but it doesn't solve the issue and we came to the limit of path shortening possibilities.

      Here's a stacktrace from a Matrix job:
      https://qa.nuxeo.org/jenkins/job/nuxeo-master-fullbuild-part2-distribution-multios/218/Slave=MULTIDB_WINDOWS

      08:59:26 Building remotely on tweedledum in workspace C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS
      08:59:26 Checkout:MULTIDB_WINDOWS / C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS - hudson.remoting.Channel@4a58a509:tweedledum
      08:59:26 Using strategy: Default
      08:59:26 Last Built Revision: Revision 3676821964588c85aa8b71e288c743c24edd00a3 (origin/master)
      08:59:27 Cloning the remote Git repository
      08:59:27 Cloning repository git://github.com/nuxeo/nuxeo-distribution.git
      08:59:27 git --version
      08:59:27 git version 1.7.6.msysgit.0
      08:59:27 ERROR: Failed to clean the workspace
      08:59:27 java.io.IOException: Unable to delete C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS\nuxeo-distribution-jboss\target\nuxeo-cap-5.7-SNAPSHOT-jboss\server\default\deploy\jbossws.sar\jbossws-management.war\META-INF\maven\org.jboss.ws.native\jbossws-native-management - files in dir: [C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS\nuxeo-distribution-jboss\target\nuxeo-cap-5.7-SNAPSHOT-jboss\server\default\deploy\jbossws.sar\jbossws-management.war\META-INF\maven\org.jboss.ws.native\jbossws-native-management\pom.properties, C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS\nuxeo-distribution-jboss\target\nuxeo-cap-5.7-SNAPSHOT-jboss\server\default\deploy\jbossws.sar\jbossws-management.war\META-INF\maven\org.jboss.ws.native\jbossws-native-management\pom.xml]
      08:59:27 	at hudson.Util.deleteFile(Util.java:238)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:289)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.FilePath$11.invoke(FilePath.java:910)
      08:59:27 	at hudson.FilePath$11.invoke(FilePath.java:908)
      08:59:27 	at hudson.FilePath.act(FilePath.java:842)
      08:59:27 	at hudson.FilePath.act(FilePath.java:824)
      08:59:27 	at hudson.FilePath.deleteRecursive(FilePath.java:908)
      08:59:27 	at hudson.plugins.git.GitAPI.clone(GitAPI.java:239)
      08:59:27 	at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:1040)
      08:59:27 	at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:982)
      08:59:27 	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2200)
      08:59:27 	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
      08:59:27 	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      08:59:27 	at hudson.remoting.Request$2.run(Request.java:326)
      08:59:27 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      08:59:27 	at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
      08:59:27 	at java.util.concurrent.FutureTask.run(Unknown Source)
      08:59:27 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
      08:59:27 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      08:59:27 	at hudson.remoting.Engine$1$1.run(Engine.java:60)
      08:59:27 	at java.lang.Thread.run(Unknown Source)
      08:59:27 ERROR: Error cloning remote repo 'origin' : Failed to delete workspace
      08:59:27 hudson.plugins.git.GitException: Failed to delete workspace
      08:59:27 	at hudson.plugins.git.GitAPI.clone(GitAPI.java:242)
      08:59:27 	at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:1040)
      08:59:27 	at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:982)
      08:59:27 	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2200)
      08:59:27 	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
      08:59:27 	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      08:59:27 	at hudson.remoting.Request$2.run(Request.java:326)
      08:59:27 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      08:59:27 	at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
      08:59:27 	at java.util.concurrent.FutureTask.run(Unknown Source)
      08:59:27 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
      08:59:27 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      08:59:27 	at hudson.remoting.Engine$1$1.run(Engine.java:60)
      08:59:27 	at java.lang.Thread.run(Unknown Source)
      08:59:27 Caused by: java.io.IOException: Unable to delete C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS\nuxeo-distribution-jboss\target\nuxeo-cap-5.7-SNAPSHOT-jboss\server\default\deploy\jbossws.sar\jbossws-management.war\META-INF\maven\org.jboss.ws.native\jbossws-native-management - files in dir: [C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS\nuxeo-distribution-jboss\target\nuxeo-cap-5.7-SNAPSHOT-jboss\server\default\deploy\jbossws.sar\jbossws-management.war\META-INF\maven\org.jboss.ws.native\jbossws-native-management\pom.properties, C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS\nuxeo-distribution-jboss\target\nuxeo-cap-5.7-SNAPSHOT-jboss\server\default\deploy\jbossws.sar\jbossws-management.war\META-INF\maven\org.jboss.ws.native\jbossws-native-management\pom.xml]
      08:59:27 	at hudson.Util.deleteFile(Util.java:238)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:289)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.Util.deleteContentsRecursive(Util.java:200)
      08:59:27 	at hudson.Util.deleteRecursive(Util.java:280)
      08:59:27 	at hudson.FilePath$11.invoke(FilePath.java:910)
      08:59:27 	at hudson.FilePath$11.invoke(FilePath.java:908)
      08:59:27 	at hudson.FilePath.act(FilePath.java:842)
      08:59:27 	at hudson.FilePath.act(FilePath.java:824)
      08:59:27 	at hudson.FilePath.deleteRecursive(FilePath.java:908)
      08:59:27 	at hudson.plugins.git.GitAPI.clone(GitAPI.java:239)
      08:59:27 	... 13 more
      08:59:27 Trying next repository
      08:59:27 ERROR: Could not clone repository
      08:59:27 FATAL: Could not clone
      08:59:27 hudson.plugins.git.GitException: Could not clone
      08:59:27 	at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:1052)
      08:59:27 	at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:982)
      08:59:27 	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2200)
      08:59:27 	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
      08:59:27 	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      08:59:27 	at hudson.remoting.Request$2.run(Request.java:326)
      08:59:27 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      08:59:27 	at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
      08:59:27 	at java.util.concurrent.FutureTask.run(Unknown Source)
      08:59:27 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
      08:59:27 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      08:59:27 	at hudson.remoting.Engine$1$1.run(Engine.java:60)
      08:59:27 	at java.lang.Thread.run(Unknown Source)
      

      It is still possible to manually delete the files.

      FYI, we encountered such a seemingly similar issue in a Python script using Python internal delTree implementation, and it has been resolved by using Windows Shell command: "rmdir /s /q path/to/file" instead.

      Using a matrix job is increasing the path length.
      We would also need to use a different workspace path when under Windows.

        Attachments

          Issue Links

            Activity

            Hide
            markewaite Mark Waite added a comment -

            We've seen a general problem on Windows file systems with paths which exceed about 250 characters. The problems were visible in our MSBuild scripts and affected other programs as well. We finally decided that it was not worth fighting what appeared to be Microsoft file system limitations, so we forced ourselves to use shorter paths. The sample path in your output is longer than 250 characters, so you may be encountering the same problem we encountered.

            You might find Jeff Attwood's comments on the topic interesting.

            http://www.codinghorror.com/blog/2006/11/filesystem-paths-how-long-is-too-long.html

            Show
            markewaite Mark Waite added a comment - We've seen a general problem on Windows file systems with paths which exceed about 250 characters. The problems were visible in our MSBuild scripts and affected other programs as well. We finally decided that it was not worth fighting what appeared to be Microsoft file system limitations, so we forced ourselves to use shorter paths. The sample path in your output is longer than 250 characters, so you may be encountering the same problem we encountered. You might find Jeff Attwood's comments on the topic interesting. http://www.codinghorror.com/blog/2006/11/filesystem-paths-how-long-is-too-long.html
            Hide
            jcarsique Julien Carsique added a comment -

            Hi,

            Thanks for that interesting link.
            You're right and we already worked on shortening our paths. We also wrote a few scripts which automatically map the current path on a new drive letter in order to reduce the risk to encounter that ridiculous Windows issue. But anything we do is only delaying the issue. And it came back with that job.

            The point here is:

            • the job is a "matrix" job so the working dir path is longer (C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS). Reducing too much the job name is an issue when you have 400 jobs.
            • in the currently failing path, the only part we could still reduce is not very long: nuxeo-distribution-jboss\target\nuxeo-cap-5.7-SNAPSHOT-jboss.

            The deletion fails in Java File.delete() so there's no quick solution for that issue. But we could imagine some workarounds to reduce the times it's happening:

            • Jenkins could mount the working directory on a new drive letter,
            • Jenkins could propose to use a specific short path when on Windows (ie c:\ws\buildNumber or c:\ws\jobName\buildNumber),
            • when failing to delete a file on Windows, the code could try to fallback on calling a delete windows command (rmdir /s /q path/to/file),
            • ...
              I can't see a "clean" solution but anything that could help to shorten the path when working on windows would help...
            Show
            jcarsique Julien Carsique added a comment - Hi, Thanks for that interesting link. You're right and we already worked on shortening our paths. We also wrote a few scripts which automatically map the current path on a new drive letter in order to reduce the risk to encounter that ridiculous Windows issue. But anything we do is only delaying the issue. And it came back with that job. The point here is: the job is a "matrix" job so the working dir path is longer ( C:\Jenkins\workspace\nuxeo-master-fullbuild-part2-distribution-multios\Slave\MULTIDB_WINDOWS ). Reducing too much the job name is an issue when you have 400 jobs. in the currently failing path, the only part we could still reduce is not very long: nuxeo-distribution-jboss\target\nuxeo-cap-5.7-SNAPSHOT-jboss . The deletion fails in Java File.delete() so there's no quick solution for that issue. But we could imagine some workarounds to reduce the times it's happening: Jenkins could mount the working directory on a new drive letter, Jenkins could propose to use a specific short path when on Windows (ie c:\ws\buildNumber or c:\ws\jobName\buildNumber ), when failing to delete a file on Windows, the code could try to fallback on calling a delete windows command ( rmdir /s /q path/to/file ), ... I can't see a "clean" solution but anything that could help to shorten the path when working on windows would help...
            Hide
            onemanbucket onemanbucket added a comment -

            We're also running into the windows MAX_PATH problem.

            I tried calling File.delete with the unicode prefix (\\?\C:\...) but it didn't work.

            Show
            onemanbucket onemanbucket added a comment - We're also running into the windows MAX_PATH problem. I tried calling File.delete with the unicode prefix (\\?\C:\...) but it didn't work.
            Hide
            onemanbucket onemanbucket added a comment -

            I have a fix for the issue, but it needs review. The two main issues are:
            1) safe guard that the pivot point in the path is in the workspace so we don't delete too much
            2) should the code go in deleteRecursive

            What it does is move the subtree close to the MAX_PATH limit to a temp directory and try to delete from there.

            Commit at: https://github.com/onemanbucket/jenkins/commit/0ab18187eb60dd89cf5759eb231c326cb31c866b

            Show
            onemanbucket onemanbucket added a comment - I have a fix for the issue, but it needs review. The two main issues are: 1) safe guard that the pivot point in the path is in the workspace so we don't delete too much 2) should the code go in deleteRecursive What it does is move the subtree close to the MAX_PATH limit to a temp directory and try to delete from there. Commit at: https://github.com/onemanbucket/jenkins/commit/0ab18187eb60dd89cf5759eb231c326cb31c866b
            Hide
            jcarsique Julien Carsique added a comment -
            Show
            jcarsique Julien Carsique added a comment - Interesting solution. I commented your commit with my tests results: https://github.com/onemanbucket/jenkins/commit/0ab18187eb60dd89cf5759eb231c326cb31c866b#commitcomment-2033773
            Hide
            onemanbucket onemanbucket added a comment -

            Hi, I've done some more research now, and it turns out the jre actually supports using unicode prefixes, and all win api calls are correct. The problem is in hudsons Kernel32Utils that doesn't call GetFileAttributesW correctly (the path should be prefixed as far as I can tell). This makes deleteRecursive() mistakenly think that the first directory after MAX_PATH is a file, and since delete only works on empty directories the subsequent delete call fails.

            I'll post a test case and a patch in a bit.

            Show
            onemanbucket onemanbucket added a comment - Hi, I've done some more research now, and it turns out the jre actually supports using unicode prefixes, and all win api calls are correct. The problem is in hudsons Kernel32Utils that doesn't call GetFileAttributesW correctly (the path should be prefixed as far as I can tell). This makes deleteRecursive() mistakenly think that the first directory after MAX_PATH is a file, and since delete only works on empty directories the subsequent delete call fails. I'll post a test case and a patch in a bit.
            Show
            onemanbucket onemanbucket added a comment - https://github.com/jenkinsci/jenkins/pull/593
            Show
            jglick Jesse Glick added a comment - Now: https://github.com/jenkinsci/jenkins/pull/595
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Erik Lindblad
            Path:
            core/src/main/java/hudson/util/jna/Kernel32Utils.java
            core/src/test/java/hudson/FilePathTest.java
            http://jenkins-ci.org/commit/jenkins/f66e19c072a24afb6f083db9fc0124f58b9742e6
            Log:
            JENKINS-15418 take shares int account, ignore short paths

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Erik Lindblad Path: core/src/main/java/hudson/util/jna/Kernel32Utils.java core/src/test/java/hudson/FilePathTest.java http://jenkins-ci.org/commit/jenkins/f66e19c072a24afb6f083db9fc0124f58b9742e6 Log: JENKINS-15418 take shares int account, ignore short paths
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            core/src/main/java/hudson/util/jna/Kernel32Utils.java
            core/src/test/java/hudson/FilePathTest.java
            http://jenkins-ci.org/commit/jenkins/d7ef52e1bf68c3e7ca0a82596be89650625c1099
            Log:
            Merge pull request #595 from onemanbucket/jenkins-15418

            [FIXED JENKINS-15418] Allow long paths on windows.

            Compare: https://github.com/jenkinsci/jenkins/compare/edbe2aedca03...d7ef52e1bf68

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/hudson/util/jna/Kernel32Utils.java core/src/test/java/hudson/FilePathTest.java http://jenkins-ci.org/commit/jenkins/d7ef52e1bf68c3e7ca0a82596be89650625c1099 Log: Merge pull request #595 from onemanbucket/jenkins-15418 [FIXED JENKINS-15418] Allow long paths on windows. Compare: https://github.com/jenkinsci/jenkins/compare/edbe2aedca03...d7ef52e1bf68
            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: changelog.html http://jenkins-ci.org/commit/jenkins/85019c46d79dd201de8898f4c82e70f761aa2aa5 Log: JENKINS-15418 Noting. Compare: https://github.com/jenkinsci/jenkins/compare/d7ef52e1bf68...85019c46d79d
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #2032
            JENKINS-15418 take shares int account, ignore short paths (Revision f66e19c072a24afb6f083db9fc0124f58b9742e6)
            JENKINS-15418 Noting. (Revision 85019c46d79dd201de8898f4c82e70f761aa2aa5)

            Result = UNSTABLE
            erik.lindblad : f66e19c072a24afb6f083db9fc0124f58b9742e6
            Files :

            • core/src/test/java/hudson/FilePathTest.java
            • core/src/main/java/hudson/util/jna/Kernel32Utils.java

            Jesse Glick : 85019c46d79dd201de8898f4c82e70f761aa2aa5
            Files :

            • changelog.html
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #2032 JENKINS-15418 take shares int account, ignore short paths (Revision f66e19c072a24afb6f083db9fc0124f58b9742e6) JENKINS-15418 Noting. (Revision 85019c46d79dd201de8898f4c82e70f761aa2aa5) Result = UNSTABLE erik.lindblad : f66e19c072a24afb6f083db9fc0124f58b9742e6 Files : core/src/test/java/hudson/FilePathTest.java core/src/main/java/hudson/util/jna/Kernel32Utils.java Jesse Glick : 85019c46d79dd201de8898f4c82e70f761aa2aa5 Files : changelog.html
            Hide
            aliceten ernestina muck added a comment -

            Alternatively, you can use Long Path Tool, it works good for such issues!

            Show
            aliceten ernestina muck added a comment - Alternatively, you can use Long Path Tool, it works good for such issues!
            Hide
            markewaite Mark Waite added a comment -

            I assume you're referring to http://longpathtool.com/, a commercial product which can assist with the fundamental limitations of Windows. I don't know how that would help Jenkins, since Jenkins needs to reliably remove files with long path names, whether or not a commercial product is available. Can you describe further how you envision long path tool would work in a Jenkins environment?

            Show
            markewaite Mark Waite added a comment - I assume you're referring to http://longpathtool.com/ , a commercial product which can assist with the fundamental limitations of Windows. I don't know how that would help Jenkins, since Jenkins needs to reliably remove files with long path names, whether or not a commercial product is available. Can you describe further how you envision long path tool would work in a Jenkins environment?

              People

              • Assignee:
                ndeloof Nicolas De Loof
                Reporter:
                jcarsique Julien Carsique
              • Votes:
                1 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: