Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52404

Job fails randomly when using copyArtifact

    Details

    • Type: Bug
    • Status: Fixed but Unreleased (View Workflow)
    • Priority: Minor
    • Resolution: Won't Fix
    • Component/s: copyartifact-plugin
    • Labels:
      None
    • Environment:
      Jenkins ver. 2.107.3
      Windows 10x64 on master and slave-nodes
      copy-artifact-plugin 1.39.1
    • Similar Issues:

      Description

      As the title states, the job fails randomly. Thus, I could not find out any more precise scenarios where the error occurs.

       

      [Pipeline] [DSWZ10001] }
      [Pipeline] [DSWZ10022] }
      [Pipeline] // parallel
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] stage
      [Pipeline] { (Archive)
      [Pipeline] copyArtifacts
      Copied 5 artifacts from "AutoDevSuite_on_node" build number 65
      [Pipeline] copyArtifacts
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] End of Pipeline
      Also:   java.nio.file.AccessDeniedException: C:\appl\Jenkins\fingerprints\af\90\atomic5419828718498066796tmp -> C:\appl\Jenkins\fingerprints\af\90\f9276921737b82db27ef4b8b8710.xml
      		at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
      		at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
      		at sun.nio.fs.WindowsFileCopy.move(Unknown Source)
      		at sun.nio.fs.WindowsFileSystemProvider.move(Unknown Source)
      		at java.nio.file.Files.move(Unknown Source)
      		at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:191)
      java.nio.file.FileAlreadyExistsException: C:\appl\Jenkins\fingerprints\af\90\atomic5419828718498066796tmp -> C:\appl\Jenkins\fingerprints\af\90\f9276921737b82db27ef4b8b8710.xml
      	at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
      	at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
      	at sun.nio.fs.WindowsFileCopy.move(Unknown Source)
      	at sun.nio.fs.WindowsFileSystemProvider.move(Unknown Source)
      	at java.nio.file.Files.move(Unknown Source)
      	at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:206)
      	at hudson.model.Fingerprint.save(Fingerprint.java:1295)
      	at hudson.model.Fingerprint.save(Fingerprint.java:1245)
      	at hudson.model.Fingerprint.add(Fingerprint.java:1031)
      	at hudson.model.Fingerprint.addFor(Fingerprint.java:1023)
      	at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyOne(FingerprintingCopyMethod.java:113)
      Caused: hudson.util.IOException2: Failed to copy C:\appl\Jenkins\jobs\AutoDevSuite_on_node\builds\66\archive\LibraryInstaller_4022_1.1.2.1_unversioned_20180706_084642.zip to C:\appl\Jenkins\workspace\AutoDevSuite\LibraryInstaller_4022_1.1.2.1_unversioned_20180706_084642.zip
      	at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyOne(FingerprintingCopyMethod.java:118)
      	at hudson.plugins.copyartifact.FingerprintingCopyMethod.copyAll(FingerprintingCopyMethod.java:67)
      	at hudson.plugins.copyartifact.CopyArtifact.perform(CopyArtifact.java:527)
      	at hudson.plugins.copyartifact.CopyArtifact.perform(CopyArtifact.java:461)
      	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
      	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
      	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:49)
      	at hudson.security.ACL.impersonate(ACL.java:290)
      	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:46)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
      	at java.util.concurrent.FutureTask.run(Unknown Source)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      	at java.lang.Thread.run(Unknown Source)
      Finished: FAILURE

        Attachments

          Issue Links

            Activity

            Hide
            ilmarinen Mathias Helminger added a comment - - edited

            In our case, these exceptions appear to relate to kaspersky antivirus running on the jenkins server. The underlying problem is that the antivirus file scans are integrated in windows in an asynchronous way (for performance/design reasons) Therefore, the scans can temporarily lock files or keep them alive for a short time after a delete operation appears to have succeeded.

            For us the fingerprint exception above has become the most frequent cause for random build failures (p~1/200) and we are wondering if this code properly uses the retry infrastructure built elsewhere in jenkins. (JENKINS-15331) (Turning kaspersky off temporarily helped but we do want to keep it running)

            Looking at hudson.model.Fingerprint.save() it appears there is no retry implemented, but doing so on top of AtomicFileWriter.commit() sounds like potential trouble...

            Show
            ilmarinen Mathias Helminger added a comment - - edited In our case, these exceptions appear to relate to kaspersky antivirus running on the jenkins server. The underlying problem is that the antivirus file scans are integrated in windows in an asynchronous way (for performance/design reasons) Therefore, the scans can temporarily lock files or keep them alive for a short time after a delete operation appears to have succeeded. For us the fingerprint exception above has become the most frequent cause for random build failures (p~1/200) and we are wondering if this code properly uses the retry infrastructure built elsewhere in jenkins. ( JENKINS-15331 ) (Turning kaspersky off temporarily helped but we do want to keep it running) Looking at hudson.model.Fingerprint.save() it appears there is no retry implemented, but doing so on top of AtomicFileWriter.commit() sounds like potential trouble...
            Hide
            marcosteffan Marco Steffan added a comment -

            Mathias Helminger We have already excluded these Directories from the AV-scanner which has lowered the number of exceptions by roughly 50%. We are now at a point where 1/15 builds fail due to such errors which is better but still far from good...

            https://issues.jenkins-ci.org/browse/JENKINS-53187?focusedCommentId=347311&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-347311 might be of interest for you.

            Show
            marcosteffan Marco Steffan added a comment - Mathias Helminger We have already excluded these Directories from the AV-scanner which has lowered the number of exceptions by roughly 50%. We are now at a point where 1/15 builds fail due to such errors which is better but still far from good... https://issues.jenkins-ci.org/browse/JENKINS-53187?focusedCommentId=347311&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-347311  might be of interest for you.
            Hide
            ikedam ikedam added a comment -

            There’s no fundamental resolution for this issue as it comes from the OS level issue.
            And I don’t plan to add retry feature to copyartifact as it cause codes complicated and hard to maintain.

            Please consider workarounds like excluding scan directory, remove files in the target directory before copying, and so on.

            Show
            ikedam ikedam added a comment - There’s no fundamental resolution for this issue as it comes from the OS level issue. And I don’t plan to add retry feature to copyartifact as it cause codes complicated and hard to maintain. Please consider workarounds like excluding scan directory, remove files in the target directory before copying, and so on.

              People

              • Assignee:
                marcosteffan Marco Steffan
                Reporter:
                marcosteffan Marco Steffan
              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: