Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-17995

Jenkins produces DirectoryNotEmptyException when attempting to delete large directories

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Won't Fix
    • Component/s: core, subversion-plugin
    • Labels:
      None
    • Environment:
      Jenkins version: 1.514
      Master: Debian Wheezy, standard openjdk (6b27-1.12.5-1, java version 1.6.0_27).
      Slave: Windows 7, Java 1.7.0_04
    • Similar Issues:

      Description

      We're getting DirectoryNotEmptyException's in 2 scenario's when dealing with workspaces that have directories that contain a lot of files and are heavily nested (specifically: maven local repositories). Both when applying the "Emulate clean checkout by first deleting unversioned/ignored files, then 'svn update'" SVN checkout strategy, and when deleting a workspace (both on a Windows slave), Jenkins seems to sometimes have trouble deleting the local maven repository (which is contained in the workspace, and yes that's stupid and we changed it, but I thought I log this bug anyway). See the logs below.

      Here's the error I got in the build log:

      Bouwen op afstand op fitnesse-host in workspace C:\jenkins\workspace\Test_Job
      Checking out a fresh workspace because the workspace is not http://svn.local/fitnesse/tags/LAST_STABLE
      Cleaning local Directory .
      hudson.util.IOException2: remote file operation failed: C:\jenkins\workspace\Test_Job at hudson.remoting.Channel@5defe20d:fitnesse-host
        at hudson.FilePath.act(FilePath.java:900)
        at hudson.FilePath.act(FilePath.java:877)
        at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:843)
        at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:781)
        at hudson.model.AbstractProject.checkout(AbstractProject.java:1369)
        at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:676)
        at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:581)
        at hudson.model.Run.execute(Run.java:1575)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
        at hudson.model.ResourceController.execute(ResourceController.java:88)
        at hudson.model.Executor.run(Executor.java:241)
      Caused by: java.io.IOException: Remote call on fitnesse-host failed
        at hudson.remoting.Channel.call(Channel.java:677)
        at hudson.FilePath.act(FilePath.java:893)
        ... 11 more
      Caused by: java.lang.ClassNotFoundException: java.nio.file.DirectoryNotEmptyException
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1680)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:266)
        at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:622)
        at hudson.remoting.MultiClassLoaderSerializer$Input.resolveClass(MultiClassLoaderSerializer.java:116)
        at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1609)
        at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1514)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1768)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
        at hudson.remoting.UserRequest.deserialize(UserRequest.java:182)
        at hudson.remoting.UserResponse.retrieve(UserRequest.java:211)
        at hudson.remoting.Channel.call(Channel.java:673)
        ... 12 more
      Recording test results
      [htmlpublisher] Archiving HTML reports...
      [htmlpublisher] Archiving at BUILD level 
      ......
      Notifying upstream projects of job completion
      Finished: FAILURE
      

      And here's the error I got from the "Wipe out current workspace" button:

      Status Code: 500
      
      Exception: remote file operation failed: C:\jenkins\workspace\Test_Job at hudson.remoting.Channel@536e8e87:fitnesse-host
      Stacktrace:
      hudson.util.IOException2: remote file operation failed: C:\jenkins\workspace\Test_Job at hudson.remoting.Channel@536e8e87:fitnesse-host
      	at hudson.FilePath.act(FilePath.java:900)
      	at hudson.FilePath.act(FilePath.java:877)
      	at hudson.FilePath.deleteRecursive(FilePath.java:1036)
      	at hudson.model.AbstractProject.doDoWipeOutWorkspace(AbstractProject.java:2023)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:616)
      	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:288)
      	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:151)
      	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:90)
      	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:111)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:677)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770)
      	at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:241)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:677)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770)
      	at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:241)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:677)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:583)
      	at org.kohsuke.stapler.Stapler.service(Stapler.java:214)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)
      	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:124)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:64)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
      	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
      	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470)
      	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
      	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
      	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
      	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
      	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
      	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
      	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
      	at java.lang.Thread.run(Thread.java:679)
      Caused by: java.io.IOException: Remote call on fitnesse-host failed
      	at hudson.remoting.Channel.call(Channel.java:677)
      	at hudson.FilePath.act(FilePath.java:893)
      	... 73 more
      Caused by: java.lang.ClassNotFoundException: java.nio.file.DirectoryNotEmptyException
      	at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1680)
      	at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
      	at java.lang.Class.forName0(Native Method)
      	at java.lang.Class.forName(Class.java:266)
      	at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:622)
      	at hudson.remoting.MultiClassLoaderSerializer$Input.resolveClass(MultiClassLoaderSerializer.java:116)
      	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1609)
      	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1514)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1768)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
      	at hudson.remoting.UserRequest.deserialize(UserRequest.java:182)
      	at hudson.remoting.UserResponse.retrieve(UserRequest.java:211)
      	at hudson.remoting.Channel.call(Channel.java:673)
      	... 74 more
      

        Attachments

          Issue Links

            Activity

            Hide
            brian3791 Brian Brooks added a comment - - edited

            This issue seems to be a duplicate of
            JENKINS-15331 Workaround Windows unpredictable
            The heart of this issue is JVM file locking, particularly with Windows 7 / Windows 2008.

            Show
            brian3791 Brian Brooks added a comment - - edited This issue seems to be a duplicate of JENKINS-15331 Workaround Windows unpredictable The heart of this issue is JVM file locking, particularly with Windows 7 / Windows 2008.
            Hide
            raspy Krzysztof Malinowski added a comment -

            Actually it's not DirectoryNotEmptyException itself that causes problems, it's java.lang.ClassNotFoundException: java.nio.file.DirectoryNotEmptyException, meaning the exception class could not be found. Any clues?

            Show
            raspy Krzysztof Malinowski added a comment - Actually it's not DirectoryNotEmptyException itself that causes problems, it's java.lang.ClassNotFoundException: java.nio.file.DirectoryNotEmptyException, meaning the exception class could not be found. Any clues?
            Hide
            raspy Krzysztof Malinowski added a comment -

            OK, reading further that java.nio.file.DirectoryNotEmptyException was introduced in Java 7, I guess the issue is caused by your server running Java 6, while slaves run Java 7. This way server cannot deserialize exception coming from your slave and this issue occurs.

            I think I encountered the same issue, yet I am unable to switch to 1.7 on the server right now. Once I'll check I will update this issue. If it's possible for you to update java to 1.7 on the server, you could check it for yourself.

            Show
            raspy Krzysztof Malinowski added a comment - OK, reading further that java.nio.file.DirectoryNotEmptyException was introduced in Java 7, I guess the issue is caused by your server running Java 6, while slaves run Java 7. This way server cannot deserialize exception coming from your slave and this issue occurs. I think I encountered the same issue, yet I am unable to switch to 1.7 on the server right now. Once I'll check I will update this issue. If it's possible for you to update java to 1.7 on the server, you could check it for yourself.
            Hide
            mdirkse Maarten Dirkse added a comment -

            Krzysztof: our issue was due to a master/server JDK version mismatch as well, sorry for not reporting back.

            Show
            mdirkse Maarten Dirkse added a comment - Krzysztof: our issue was due to a master/server JDK version mismatch as well, sorry for not reporting back.
            Hide
            mdirkse Maarten Dirkse added a comment -

            As said in earlier comments, this issue was due to a mismatch in JDK version between the master and the slave machines. The exception thrown is one that doesn't exists in JDK 6, so deserialization fails. The issue disappeared when we downgraded the slaves to JDK 6.

            Show
            mdirkse Maarten Dirkse added a comment - As said in earlier comments, this issue was due to a mismatch in JDK version between the master and the slave machines. The exception thrown is one that doesn't exists in JDK 6, so deserialization fails. The issue disappeared when we downgraded the slaves to JDK 6.

              People

              • Assignee:
                Unassigned
                Reporter:
                mdirkse Maarten Dirkse
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: