Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46298

Jenkins workspace-cleanup (and FilePath API) cannot delete resources with unicode names

    Details

    • Similar Issues:

      Description

      (Apologies if not filing this under the right components.)

       

      I've had a problem on a software project lately, whereby we have a multi-branch pipeline built from a Jenkinsfile and during the automated workspace cleanup (either by requesting it from the scm/git plugin or when deleted branches are removed in the background by the pipeline multi-branch job) it fails to delete the workspaces.

       

      Somewhere in this software project, there are some files and folders with somewhat unusual names, in that they use an extended charset. For instance, one project uses a node module with some test folders as follows:

      $> ls -la
      total 32
      drwxr-xr-x 10 i316748 1694527156 340 Aug 8 14:28 .
      drwxr-xr-x 14 i316748 1694527156 476 Aug 8 14:28 ..
      -rw-r--r-- 1 i316748 1694527156 13 Aug 8 13:58 404.html
      -rw-r--r-- 1 i316748 1694527156 5 Aug 8 13:58 a.txt
      -rw-r--r-- 1 i316748 1694527156 5 Aug 8 13:58 b.txt
      -rw-r--r-- 1 i316748 1694527156 21 Aug 8 13:58 c.js
      drwxr-xr-x 4 i316748 1694527156 136 Aug 8 14:28 compress
      drwxr-xr-x 4 i316748 1694527156 136 Aug 8 14:28 subdir
      drwxr-xr-x 4 i316748 1694527156 136 Aug 8 14:28 subdir_with space
      drwxr-xr-x 3 i316748 1694527156 102 Aug 8 14:28 中文

       

      Jenkins will interrupt the entire recursive cleanup of the workspace as soon as it hits this file, and as a result we have remaining stray workspaces. It's sort of ok when it's triggered by usage from the Git or SCM plugins "force clean checkout" or similar options, as at least we can see it blow up (build is failed, error is reported in the console).

      However, it's a bit sneakier in the case of the multi-branch pipeline scanner, as it fails to delete silently from a user perspective, and you'll only see the issue if you look at the Jenkins log files or when your sysadmin asks why you have hundreds of workspaces eating away at his precious disk space (which we can debate, but... ).

       

      Anyways, I thought this would be easy, and that simply setting proper locales (LC_ALL, LANG, and so forth...) on the machines hosting my master and slaves, and that giving appropriate properties (file.encoding, sun.jnu.encoding, ...) to the Java processes running them would suffice...

      Sadly, that's not the case.

      And I still these in my logs, and stray workspaces remain:

      WARNING: could not clean up workspace directory /home/jenkins/workspace/-page-search-sans-resultats-KHKRZ5WQ5FWRJIC7XKB7CKLDV5U4MILCU4MQPHZW2J3ABWOUCYWQ on jenkins-slave
      java.io.IOException: remote file operation failed: /home/jenkins/workspace/-page-search-sans-resultats-KHKRZ5WQ5FWRJIC7XKB7CKLDV5U4MILCU4MQPHZW2J3ABWOUCYWQ at hudson.remoting.Channel@2404ed4d:jenkins-slave: java.io.IOException: Unable to delete '/home/jenkins/workspace/-page-search-sans-resultats-KHKRZ5WQ5FWRJIC7XKB7CKLDV5U4MILCU4MQPHZW2J3ABWOUCYWQ/hybris/bin/ext-content/npmancillary/resources/npm/node_modules/http-server/node_modules/ecstatic/test/public'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
      at hudson.FilePath.act(FilePath.java:994)
      at hudson.FilePath.act(FilePath.java:976)
      at hudson.FilePath.deleteRecursive(FilePath.java:1178)
      at jenkins.branch.WorkspaceLocatorImpl$Deleter$CleanupTask.run(WorkspaceLocatorImpl.java:211)
      at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.io.IOException: Unable to delete '/home/jenkins/workspace/-page-search-sans-resultats-KHKRZ5WQ5FWRJIC7XKB7CKLDV5U4MILCU4MQPHZW2J3ABWOUCYWQ/hybris/bin/ext-content/npmancillary/resources/npm/node_modules/http-server/node_modules/ecstatic/test/public'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
      at hudson.Util.deleteFile(Util.java:250)
      at hudson.FilePath.deleteRecursive(FilePath.java:1211)
      at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
      at hudson.FilePath.deleteRecursive(FilePath.java:1202)
      at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
      at hudson.FilePath.deleteRecursive(FilePath.java:1202)
      at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
      at hudson.FilePath.deleteRecursive(FilePath.java:1202)
      at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
      at hudson.FilePath.deleteRecursive(FilePath.java:1202)
      at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
      at hudson.FilePath.deleteRecursive(FilePath.java:1202)
      at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
      at hudson.FilePath.deleteRecursive(FilePath.java:1202)
      at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
      at hudson.FilePath.deleteRecursive(FilePath.java:1202)
      at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
      at hudson.FilePath.deleteRecursive(FilePath.java:1202)
      at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
      at hudson.FilePath.deleteRecursive(FilePath.java:1202)
      at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
      at hudson.FilePath.deleteRecursive(FilePath.java:1202)
      at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
      at hudson.FilePath.deleteRecursive(FilePath.java:1202)
      at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
      at hudson.FilePath.deleteRecursive(FilePath.java:1202)
      at hudson.FilePath.access$1000(FilePath.java:197)
      at hudson.FilePath$14.invoke(FilePath.java:1181)
      at hudson.FilePath$14.invoke(FilePath.java:1178)
      at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2749)
      at hudson.remoting.UserRequest.perform(UserRequest.java:153)
      at hudson.remoting.UserRequest.perform(UserRequest.java:50)
      at hudson.remoting.Request$2.run(Request.java:336)
      at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
      at ......remote call to jenkins-slave(Native Method)
      at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1545)
      at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
      at hudson.remoting.Channel.call(Channel.java:830)
      at hudson.FilePath.act(FilePath.java:987)
      ... 9 more
      Caused by: java.nio.file.DirectoryNotEmptyException: /home/jenkins/workspace/-page-search-sans-resultats-KHKRZ5WQ5FWRJIC7XKB7CKLDV5U4MILCU4MQPHZW2J3ABWOUCYWQ/hybris/bin/ext-content/npmancillary/resources/npm/node_modules/http-server/node_modules/ecstatic/test/public
      at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
      at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
      at java.nio.file.Files.deleteIfExists(Files.java:1165)
      at hudson.Util.tryOnceDeleteFile(Util.java:290)
      at hudson.Util.deleteFile(Util.java:245)
      at hudson.FilePath.deleteRecursive(FilePath.java:1211)
      at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
      at hudson.FilePath.deleteRecursive(FilePath.java:1202)
      at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
      at hudson.FilePath.deleteRecursive(FilePath.java:1202)
      at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
      at hudson.FilePath.deleteRecursive(FilePath.java:1202)
      at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
      at hudson.FilePath.deleteRecursive(FilePath.java:1202)
      at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
      at hudson.FilePath.deleteRecursive(FilePath.java:1202)
      at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
      at hudson.FilePath.deleteRecursive(FilePath.java:1202)
      at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
      at hudson.FilePath.deleteRecursive(FilePath.java:1202)
      at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
      at hudson.FilePath.deleteRecursive(FilePath.java:1202)
      at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
      at hudson.FilePath.deleteRecursive(FilePath.java:1202)
      at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
      at hudson.FilePath.deleteRecursive(FilePath.java:1202)
      at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
      at hudson.FilePath.deleteRecursive(FilePath.java:1202)
      at hudson.FilePath.deleteContentsRecursive(FilePath.java:1220)
      at hudson.FilePath.deleteRecursive(FilePath.java:1202)
      at hudson.FilePath.access$1000(FilePath.java:197)
      at hudson.FilePath$14.invoke(FilePath.java:1181)
      at hudson.FilePath$14.invoke(FilePath.java:1178)
      at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2749)
      at hudson.remoting.UserRequest.perform(UserRequest.java:153)
      at hudson.remoting.UserRequest.perform(UserRequest.java:50)
      at hudson.remoting.Request$2.run(Request.java:336)
      at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
      ... 4 more

      Note that in this log you don't actually see why it fails, it just complains about the directly not being empty. I know use these env vars and properties on my build machines:

      LC_ALL=en_US.UTF-8
      LANG=en_US.UTF-8
      -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8

      Originally the machines were plainly not configured, and the error was slightly different but I didn't keep the logs. It was, however, more explicit about the fact that it could not handle a malformed name, which led me to these tickets (same error logs): 

       * https://issues.jenkins-ci.org/browse/JENKINS-33478

       * https://issues.jenkins-ci.org/browse/JENKINS-12610 (might actually be a duplicate of this one)

       

      While I can see a benefit in Jenkins being locale and charset dependent on some things as it will help to identify bugs in the products it builds, here it's a bit of an issue as it happens really in the management of the build jobs themselves, and I'd expect admins and users to not have to worry about this.

      I'd recommend to rewrite some of the filesystem manipulation code to be a bit more aggressive in its removal approach.

       

      In the meantime, workarounds are doable but a bit ugly and unatural. Again, I'd simply expect not to have to worry about it. E.g.:

       * crontabs and scripts to regularly delete workspaces,

       * specific steps to implement in a pipeline to take care of it in a more resilient manner, e.g. something akin to:

      post { always { sh("violent-workspace-cleanup.sh") } }

       

        Attachments

          Issue Links

            Activity

            There are no comments yet on this issue.

              People

              • Assignee:
                olivergondza Oliver Gondža
                Reporter:
                laurent_malvert Laurent Malvert
              • Votes:
                4 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: