Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-10905

Deleting previous project files regularly fails the build

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Blocker
    • Resolution: Duplicate
    • Component/s: core
    • Labels:
      None
    • Environment:
      Window 7 operating in master/slave environment (the failing machine is the slave)
    • Similar Issues:

      Description

      I have a project that is quite large. I'm not sure if it's coincidental or not, but since upgrading to 1.428, I've been getting a number of intermittent build failures, that actually fail in the cleanup stage deleting files from the previous build. My svn settings are set to 'clean checkout' although I also get this problem if I select 'revert and update'. Is it possible to make this stage more tolerant of locked files (perhaps by increasing the number of retries?)

      09:51:40 Started by user niy
      09:51:40 Building remotely on nwb-r5win32b
      09:51:40 Cleaning workspace C:\jenkins\workspace\R5_Win_Overnight
      09:52:29 hudson.util.IOException2: remote file operation failed: c:\jenkins\workspace\R5_Win_Overnight at hudson.remoting.Channel@470d6c27:nwb-r5win32b
      09:52:29 at hudson.FilePath.act(FilePath.java:754)
      09:52:29 at hudson.FilePath.act(FilePath.java:740)
      09:52:29 at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:731)
      09:52:29 at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:676)
      09:52:29 at hudson.model.AbstractProject.checkout(AbstractProject.java:1193)
      09:52:29 at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:555)
      09:52:29 at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:443)
      09:52:29 at hudson.model.Run.run(Run.java:1376)
      09:52:29 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      09:52:29 at hudson.model.ResourceController.execute(ResourceController.java:88)
      09:52:29 at hudson.model.Executor.run(Executor.java:230)
      09:52:29 Caused by: java.io.IOException: Unable to delete c:\jenkins\workspace\R5_Win_Overnight\viscob\coretech\checker\src\tools\.svn - files in dir: [c:\jenkins\workspace\R5_Win_Overnight\viscob\coretech\checker\src\tools\.svn\tmp]
      09:52:29 at hudson.Util.deleteFile(Util.java:265)
      09:52:29 at hudson.Util.deleteRecursive(Util.java:316)
      09:52:29 at hudson.Util.deleteContentsRecursive(Util.java:227)
      09:52:29 at hudson.Util.deleteRecursive(Util.java:307)
      09:52:29 at hudson.Util.deleteContentsRecursive(Util.java:227)
      09:52:29 at hudson.Util.deleteRecursive(Util.java:307)
      09:52:29 at hudson.Util.deleteContentsRecursive(Util.java:227)
      09:52:29 at hudson.Util.deleteRecursive(Util.java:307)
      09:52:29 at hudson.Util.deleteContentsRecursive(Util.java:227)
      09:52:29 at hudson.Util.deleteRecursive(Util.java:307)
      09:52:29 at hudson.Util.deleteContentsRecursive(Util.java:227)
      09:52:29 at hudson.Util.deleteRecursive(Util.java:307)
      09:52:29 at hudson.Util.deleteContentsRecursive(Util.java:227)
      09:52:29 at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:74)
      09:52:29 at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:136)
      09:52:29 at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:773)
      09:52:29 at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:754)
      09:52:29 at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:738)
      09:52:29 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:1994)
      09:52:29 at hudson.remoting.UserRequest.perform(UserRequest.java:118)
      09:52:29 at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      09:52:29 at hudson.remoting.Request$2.run(Request.java:287)
      09:52:29 at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
      09:52:29 at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
      09:52:29 at java.util.concurrent.FutureTask.run(Unknown Source)
      09:52:29 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
      09:52:29 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      09:52:29 at hudson.remoting.Engine$1$1.run(Engine.java:60)
      09:52:29 at java.lang.Thread.run(Unknown Source)
      09:52:29 Retrying after 10 seconds
      09:52:39 Cleaning workspace C:\jenkins\workspace\R5_Win_Overnight
      09:55:02 hudson.util.IOException2: remote file operation failed: c:\jenkins\workspace\R5_Win_Overnight at hudson.remoting.Channel@470d6c27:nwb-r5win32b
      09:55:02 at hudson.FilePath.act(FilePath.java:754)
      09:55:02 at hudson.FilePath.act(FilePath.java:740)
      09:55:02 at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:731)
      09:55:02 at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:676)
      09:55:02 at hudson.model.AbstractProject.checkout(AbstractProject.java:1193)
      09:55:02 at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:555)
      09:55:02 at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:443)
      09:55:02 at hudson.model.Run.run(Run.java:1376)
      09:55:02 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      09:55:02 at hudson.model.ResourceController.execute(ResourceController.java:88)
      09:55:02 at hudson.model.Executor.run(Executor.java:230)
      09:55:02 Caused by: java.io.IOException: Unable to delete c:\jenkins\workspace\R5_Win_Overnight\viscob\eclipseide\feature.builder\core\build\plugins\com.microfocus.eclipse.core@dot\com\microfocus\eclipse\core - files in dir: [c:\jenkins\workspace\R5_Win_Overnight\viscob\eclipseide\feature.builder\core\build\plugins\com.microfocus.eclipse.core\@dot\com\microfocus\eclipse\core\wizards]
      09:55:02 at hudson.Util.deleteFile(Util.java:265)
      09:55:02 at hudson.Util.deleteRecursive(Util.java:316)
      09:55:02 at hudson.Util.deleteContentsRecursive(Util.java:227)
      09:55:02 at hudson.Util.deleteRecursive(Util.java:307)
      09:55:02 at hudson.Util.deleteContentsRecursive(Util.java:227)
      09:55:02 at hudson.Util.deleteRecursive(Util.java:307)
      09:55:02 at hudson.Util.deleteContentsRecursive(Util.java:227)
      09:55:02 at hudson.Util.deleteRecursive(Util.java:307)
      09:55:02 at hudson.Util.deleteContentsRecursive(Util.java:227)
      09:55:02 at hudson.Util.deleteRecursive(Util.java:307)
      09:55:02 at hudson.Util.deleteContentsRecursive(Util.java:227)
      09:55:02 at hudson.Util.deleteRecursive(Util.java:307)
      09:55:02 at hudson.Util.deleteContentsRecursive(Util.java:227)
      09:55:02 at hudson.Util.deleteRecursive(Util.java:307)
      09:55:02 at hudson.Util.deleteContentsRecursive(Util.java:227)
      09:55:02 at hudson.Util.deleteRecursive(Util.java:307)
      09:55:02 at hudson.Util.deleteContentsRecursive(Util.java:227)
      09:55:02 at hudson.Util.deleteRecursive(Util.java:307)
      09:55:02 at hudson.Util.deleteContentsRecursive(Util.java:227)
      09:55:02 at hudson.Util.deleteRecursive(Util.java:307)
      09:55:02 at hudson.Util.deleteContentsRecursive(Util.java:227)
      09:55:02 at hudson.Util.deleteRecursive(Util.java:307)
      09:55:02 at hudson.Util.deleteContentsRecursive(Util.java:227)
      09:55:02 at hudson.Util.deleteRecursive(Util.java:307)
      09:55:02 at hudson.Util.deleteContentsRecursive(Util.java:227)
      09:55:02 at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:74)
      09:55:02 at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:136)
      09:55:02 at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:773)
      09:55:02 at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:754)
      09:55:02 at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:738)
      09:55:02 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:1994)
      09:55:02 at hudson.remoting.UserRequest.perform(UserRequest.java:118)
      09:55:02 at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      09:55:02 at hudson.remoting.Request$2.run(Request.java:287)
      09:55:02 at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
      09:55:02 at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
      09:55:02 at java.util.concurrent.FutureTask.run(Unknown Source)
      09:55:02 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
      09:55:02 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      09:55:02 at hudson.remoting.Engine$1$1.run(Engine.java:60)
      09:55:02 at java.lang.Thread.run(Unknown Source)
      09:55:02 Retrying after 10 seconds
      09:55:12 Cleaning workspace C:\jenkins\workspace\R5_Win_Overnight
      09:55:13 hudson.util.IOException2: remote file operation failed: c:\jenkins\workspace\R5_Win_Overnight at hudson.remoting.Channel@470d6c27:nwb-r5win32b
      09:55:13 at hudson.FilePath.act(FilePath.java:754)
      09:55:13 at hudson.FilePath.act(FilePath.java:740)
      09:55:13 at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:731)
      09:55:13 at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:676)
      09:55:13 at hudson.model.AbstractProject.checkout(AbstractProject.java:1193)
      09:55:13 at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:555)
      09:55:13 at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:443)
      09:55:13 at hudson.model.Run.run(Run.java:1376)
      09:55:13 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      09:55:13 at hudson.model.ResourceController.execute(ResourceController.java:88)
      09:55:13 at hudson.model.Executor.run(Executor.java:230)
      09:55:13 Caused by: java.io.IOException: Unable to delete c:\jenkins\workspace\R5_Win_Overnight\viscob\eclipseide\feature.builder\core\build\plugins\com.microfocus.eclipse.core\src\com\microfocus\eclipse\core - files in dir: [c:\jenkins\workspace\R5_Win_Overnight\viscob\eclipseide\feature.builder\core\build\plugins\com.microfocus.eclipse.core\src\com\microfocus\eclipse\core\wizards]
      09:55:13 at hudson.Util.deleteFile(Util.java:265)
      09:55:13 at hudson.Util.deleteRecursive(Util.java:316)
      09:55:13 at hudson.Util.deleteContentsRecursive(Util.java:227)
      09:55:13 at hudson.Util.deleteRecursive(Util.java:307)
      09:55:13 at hudson.Util.deleteContentsRecursive(Util.java:227)
      09:55:13 at hudson.Util.deleteRecursive(Util.java:307)
      09:55:13 at hudson.Util.deleteContentsRecursive(Util.java:227)
      09:55:13 at hudson.Util.deleteRecursive(Util.java:307)
      09:55:13 at hudson.Util.deleteContentsRecursive(Util.java:227)
      09:55:13 at hudson.Util.deleteRecursive(Util.java:307)
      09:55:13 at hudson.Util.deleteContentsRecursive(Util.java:227)
      09:55:13 at hudson.Util.deleteRecursive(Util.java:307)
      09:55:13 at hudson.Util.deleteContentsRecursive(Util.java:227)
      09:55:13 at hudson.Util.deleteRecursive(Util.java:307)
      09:55:13 at hudson.Util.deleteContentsRecursive(Util.java:227)
      09:55:13 at hudson.Util.deleteRecursive(Util.java:307)
      09:55:13 at hudson.Util.deleteContentsRecursive(Util.java:227)
      09:55:13 at hudson.Util.deleteRecursive(Util.java:307)
      09:55:13 at hudson.Util.deleteContentsRecursive(Util.java:227)
      09:55:13 at hudson.Util.deleteRecursive(Util.java:307)
      09:55:13 at hudson.Util.deleteContentsRecursive(Util.java:227)
      09:55:13 at hudson.Util.deleteRecursive(Util.java:307)
      09:55:13 at hudson.Util.deleteContentsRecursive(Util.java:227)
      09:55:13 at hudson.Util.deleteRecursive(Util.java:307)
      09:55:13 at hudson.Util.deleteContentsRecursive(Util.java:227)
      09:55:13 at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:74)
      09:55:13 at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:136)
      09:55:13 at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:773)
      09:55:13 at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:754)
      09:55:13 at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:738)
      09:55:13 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:1994)
      09:55:13 at hudson.remoting.UserRequest.perform(UserRequest.java:118)
      09:55:13 at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      09:55:13 at hudson.remoting.Request$2.run(Request.java:287)
      09:55:13 at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
      09:55:13 at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
      09:55:13 at java.util.concurrent.FutureTask.run(Unknown Source)
      09:55:13 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
      09:55:13 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      09:55:13 at hudson.remoting.Engine$1$1.run(Engine.java:60)
      09:55:13 at java.lang.Thread.run(Unknown Source)
      09:55:13 Archiving artifacts
      10:12:03 Recording fingerprints
      10:12:03 Email was triggered for: Failure
      10:12:03 Sending email for trigger: Failure

        Attachments

          Issue Links

            Activity

            Hide
            robsimon robsimon added a comment -
            Show
            robsimon robsimon added a comment - duplicate to https://issues.jenkins-ci.org/browse/JENKINS-10859 ?
            Hide
            nyoung02 nyoung02 added a comment -

            It's possible, but I'm not sure - my errors seem to be during deleting files created in the previous build whereas the other defect report seems to be suggesting it's the connection itself that's failing. In my case the files involved are different each time, and it suggests in the stack trace that a file is locked. Mostly these seem to be very small files so I'm wondering if there's not a great tolerance of the file system not keeping up perhaps (hence why I suggested allowing more retries!)

            Show
            nyoung02 nyoung02 added a comment - It's possible, but I'm not sure - my errors seem to be during deleting files created in the previous build whereas the other defect report seems to be suggesting it's the connection itself that's failing. In my case the files involved are different each time, and it suggests in the stack trace that a file is locked. Mostly these seem to be very small files so I'm wondering if there's not a great tolerance of the file system not keeping up perhaps (hence why I suggested allowing more retries!)
            Hide
            nyoung02 nyoung02 added a comment - - edited

            Changing severity to critical - blocker - I had worked around this issue till now by selecting update and using some other manual svn trickery to revert and update. Because of another bug (this time in svn I think), it seems that externals are not always updated correctly using this method - therefore I've had to switch back to clean checkout - and now I'm seeing this issue again - causing around 1 in every 4 builds to fail.

            Show
            nyoung02 nyoung02 added a comment - - edited Changing severity to critical - blocker - I had worked around this issue till now by selecting update and using some other manual svn trickery to revert and update. Because of another bug (this time in svn I think), it seems that externals are not always updated correctly using this method - therefore I've had to switch back to clean checkout - and now I'm seeing this issue again - causing around 1 in every 4 builds to fail.
            Hide
            pjdarton pjdarton added a comment - - edited

            I'm glad it's not just me - I keep getting these as well.
            In my case, it's from the Accurev plugin calling hudson.Util.deleteContentsRecursive and getting spurious errors from the filesystem.

            I believe that the underlying cause is an operating system bug, either that or a very common misconfiguration, as I (sometimes) get the same problems doing "rd /s /q xxx" from a CMD window, and (sometimes) find that Explorer fails to delete everything when doing a Shift+Delete on directories.
            i.e. I think that Windows sometimes locks files briefly, causing deletes to fail.
            Note: I've configured the Windows Search service to leave well alone, and also the anti-virus, so I don't think it's them - I've had these problems from both of those on previous versions of Windows, but on Windows 7 it doesn't seem to be enough.

            However, regardless of the cause of the problem, the workaround is probably to delete everything that can be deleted, and to retry a number of times until either all files are deleted, or we've exhausted the retries.
            I wouldn't call it a blocker though - just triggering the build again usually makes the problem disappear. i.e. it's as annoying as hell, but it isn't a blocker.

            Note: JENKINS-3053 appears to be reporting the same problem.

            Show
            pjdarton pjdarton added a comment - - edited I'm glad it's not just me - I keep getting these as well. In my case, it's from the Accurev plugin calling hudson.Util.deleteContentsRecursive and getting spurious errors from the filesystem. I believe that the underlying cause is an operating system bug, either that or a very common misconfiguration, as I (sometimes) get the same problems doing "rd /s /q xxx" from a CMD window, and (sometimes) find that Explorer fails to delete everything when doing a Shift+Delete on directories. i.e. I think that Windows sometimes locks files briefly, causing deletes to fail. Note: I've configured the Windows Search service to leave well alone, and also the anti-virus, so I don't think it's them - I've had these problems from both of those on previous versions of Windows, but on Windows 7 it doesn't seem to be enough. However, regardless of the cause of the problem, the workaround is probably to delete everything that can be deleted, and to retry a number of times until either all files are deleted, or we've exhausted the retries. I wouldn't call it a blocker though - just triggering the build again usually makes the problem disappear. i.e. it's as annoying as hell, but it isn't a blocker. Note: JENKINS-3053 appears to be reporting the same problem.
            Hide
            nyoung02 nyoung02 added a comment -

            This issue (raised by me) and JENKINS-3053 are both reported on Windows. However I also see the same issue on some Linux and Unix slaves on larger checkouts (3gb+), so I'm not sure the suggested resolutions such as search service/anti-virus are relevant to this issue - sure, it may help, and will undoubtedly help with overall speed of builds.

            A possible workaround would be to change the checkout strategy to 'update' and do the svn checkout into a directory under %BUILD_ID% or %BUILD_NUMBER% - then each checkout would be definitely in a clean directory. To avoid filling the hard drive up though with old builds, we'd also need to delete the workspace at the end of the build - I think there's a plugin to do that too, so possibly worth a try...

            Show
            nyoung02 nyoung02 added a comment - This issue (raised by me) and JENKINS-3053 are both reported on Windows. However I also see the same issue on some Linux and Unix slaves on larger checkouts (3gb+), so I'm not sure the suggested resolutions such as search service/anti-virus are relevant to this issue - sure, it may help, and will undoubtedly help with overall speed of builds. A possible workaround would be to change the checkout strategy to 'update' and do the svn checkout into a directory under %BUILD_ID% or %BUILD_NUMBER% - then each checkout would be definitely in a clean directory. To avoid filling the hard drive up though with old builds, we'd also need to delete the workspace at the end of the build - I think there's a plugin to do that too, so possibly worth a try...
            Hide
            pespinoza Paul Espinoza added a comment -

            I am also experiencing these occasional checkout failures while trying to delete the files from the last build. The Checkout settings are set to 'Always check out a fresh copy'.

            This is happening on a Windows XP (version 5.1) node connected to a Linux master Jenkins server. Jenkins version 1.466.2.

            I have experienced this in 7 of my last 64 builds (which are scheduled to run nightly), so it's happening roughly 10% of the time for me.

            Triggering the job again works as a temporary work around.

            Show
            pespinoza Paul Espinoza added a comment - I am also experiencing these occasional checkout failures while trying to delete the files from the last build. The Checkout settings are set to 'Always check out a fresh copy'. This is happening on a Windows XP (version 5.1) node connected to a Linux master Jenkins server. Jenkins version 1.466.2. I have experienced this in 7 of my last 64 builds (which are scheduled to run nightly), so it's happening roughly 10% of the time for me. Triggering the job again works as a temporary work around.
            Hide
            pjdarton pjdarton added a comment -

            I have a prospective fix for this in JENKINS-15331.

            However, on Windows XP, I've found that it is possible to workaround the issue by configuring any anti-virus to not scan the workspace, and to ensure that the windows search service is either completely disabled or configured to avoid files in the workspace.
            In my experience, it's only on Vista & Windows 7 onwards that it's impossible to disable this "malware-like" behavior as it's now too deeply embedded in the OS.

            Show
            pjdarton pjdarton added a comment - I have a prospective fix for this in JENKINS-15331 . However, on Windows XP, I've found that it is possible to workaround the issue by configuring any anti-virus to not scan the workspace, and to ensure that the windows search service is either completely disabled or configured to avoid files in the workspace. In my experience, it's only on Vista & Windows 7 onwards that it's impossible to disable this "malware-like" behavior as it's now too deeply embedded in the OS.
            Hide
            danielbeck Daniel Beck added a comment -

            Resolving as duplicate of JENKINS-15331 as these are basically the same issue and the other has more watchers.

            Show
            danielbeck Daniel Beck added a comment - Resolving as duplicate of JENKINS-15331 as these are basically the same issue and the other has more watchers.

              People

              • Assignee:
                Unassigned
                Reporter:
                nyoung02 nyoung02
              • Votes:
                5 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: