Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-20759

On windows slaves, occasional crash after successful build: Failed to write winp.x64.dll

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Duplicate
    • Component/s: core
    • Labels:
      None
    • Environment:
      Windows Server 2012 VM (amd64)
      Build via Maven 3.0.4 / 3.1.0
      JRE 7.0_25-b17
    • Similar Issues:

      Description

      Occasionally, remote windows slaves die with the following trace spit to the screen on the remote slave (I am running via command line, not as a service):

       
      Nov 20, 2013 4:16:10 PM org.jvnet.winp.Native load
      WARNING: Failed to write winp.x64.dll
      java.io.FileNotFoundException: C:\Users\<redacted>\.jenkins\cache\jars\01\winp.x64.dll (The process can not access the file because it is being used by another process)
              at java.io.FileOutputStream.open(Native Method)
              at java.io.FileOutputStream.<init>(FileOutputStream.java:212)
              at java.io.FileOutputStream.<init>(FileOutputStream.java:165)
              at org.jvnet.winp.Native.load(Native.java:81)
              at org.jvnet.winp.Native.<clinit>(Native.java:52)
              at org.jvnet.winp.WinProcess.enableDebugPrivilege(WinProcess.java:200)
              at hudson.util.ProcessTree$Windows.<clinit>(ProcessTree.java:477)
              at hudson.util.ProcessTree.get(ProcessTree.java:336)
              at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:899)
              at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:890)
              at hudson.remoting.UserRequest.perform(UserRequest.java:118)
              at hudson.remoting.UserRequest.perform(UserRequest.java:48)
              at hudson.remoting.Request$2.run(Request.java:326)
              at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
              at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
              at java.util.concurrent.FutureTask.run(FutureTask.java:166)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at hudson.remoting.Engine$1$1.run(Engine.java:63)
              at java.lang.Thread.run(Thread.java:724)
      
      #
      # A fatal error has been detected by the Java Runtime Environment:
      #
      #  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x000000000c8b3a90, pid=7136, tid=2024
      #
      # JRE version: 7.0_25-b17
      # Java VM: Java HotSpot(TM) 64-Bit Server VM (23.25-b01 mixed mode windows-amd64 compressed oops)
      # Problematic frame:
      # C  0x000000000c8b3a90
      #
      # Core dump written. Default location: C:\\hs_err_pid7136.mdmp
      #
      # An error report file with more information is saved as:
      # C:\\hs_err_pid7136.log
      

      When this occurs, here's the trailing end of the build console for this build:

       
      ...
      ...
      ------------------------------------------------------------------------
      [INFO] BUILD SUCCESS
      [INFO] ------------------------------------------------------------------------
      [INFO] Total time: 1:33.420s
      [INFO] Finished at: Wed Nov 20 16:16:09 EST 2013
      [INFO] Final Memory: 42M/272M
      [INFO] ------------------------------------------------------------------------
      [JENKINS] Archiving C:\jenkins\workspace\<redacted>\pom.xml to <redacted>SNAPSHOT.pom
      [JENKINS] Archiving C:\jenkins\workspace\<redacted>-SNAPSHOT.jar to <redacted>jar
      channel stopped
      FATAL: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
      hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
      	at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41)
      	at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34)
      	at hudson.remoting.Request.call(Request.java:174)
      	at hudson.remoting.Channel.call(Channel.java:714)
      	at hudson.Launcher$RemoteLauncher.kill(Launcher.java:887)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:585)
      	at hudson.model.Run.execute(Run.java:1679)
      	at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:509)
      	at hudson.model.ResourceController.execute(ResourceController.java:88)
      	at hudson.model.Executor.run(Executor.java:230)
      Caused by: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
      	at hudson.remoting.Request.abort(Request.java:299)
      	at hudson.remoting.Channel.terminate(Channel.java:774)
      	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
      Caused by: java.net.SocketException: Connection reset
      	at java.net.SocketInputStream.read(Unknown Source)
      	at java.net.SocketInputStream.read(Unknown Source)
      	at java.io.BufferedInputStream.fill(Unknown Source)
      	at java.io.BufferedInputStream.read(Unknown Source)
      	at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
      	at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
      	at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
      	at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
      	at java.io.ObjectInputStream.readObject0(Unknown Source)
      	at java.io.ObjectInputStream.readObject(Unknown Source)
      	at hudson.remoting.Command.readFrom(Command.java:92)
      	at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
      	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
      

      The post build action that should have occurred at this point was the "Editable Email Notification" plugin. Not sure if it got that far or if the crash is immediately prior to that.

      This happens about 4-5 times out of every 100 builds. All of my build configs (for different projects, all maven builds, all executing the email-ext plugin afterwards...) do this.

      Occasionally, it will show the same stacktrace on the remote slave..

      WARNING: Failed to write winp.x64.dll
      java.io.FileNotFoundException: C:\Users\<redacted>\.jenkins\cache\jars\01\winp.x64.dll (The process can not access the file because it is being used by another process)

      .. but not crash.

      According to my google searches, a couple of people have been hitting this, though I couldn't find a JIRA issue logged against it. Not sure if anyone has determined root cause yet. I'll attach what information I can, though my company will frown upon me attaching the core dump, so I might not be able to do that.

        Attachments

          Issue Links

            Activity

            borisivan boris ivan created issue -
            borisivan boris ivan made changes -
            Field Original Value New Value
            Description Occasionally, remote windows slaves die with the following trace spit to the screen on the remote slave (I am running via command line, not as a service):

            Nov 20, 2013 4:16:10 PM org.jvnet.winp.Native load
            WARNING: Failed to write winp.x64.dll
            java.io.FileNotFoundException: C:\Users\<redacted>\.jenkins\cache\jars\01\winp.x64.dll (The process can not access the file because it is being used by another process)
                    at java.io.FileOutputStream.open(Native Method)
                    at java.io.FileOutputStream.<init>(FileOutputStream.java:212)
                    at java.io.FileOutputStream.<init>(FileOutputStream.java:165)
                    at org.jvnet.winp.Native.load(Native.java:81)
                    at org.jvnet.winp.Native.<clinit>(Native.java:52)
                    at org.jvnet.winp.WinProcess.enableDebugPrivilege(WinProcess.java:200)
                    at hudson.util.ProcessTree$Windows.<clinit>(ProcessTree.java:477)
                    at hudson.util.ProcessTree.get(ProcessTree.java:336)
                    at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:899)
                    at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:890)
                    at hudson.remoting.UserRequest.perform(UserRequest.java:118)
                    at hudson.remoting.UserRequest.perform(UserRequest.java:48)
                    at hudson.remoting.Request$2.run(Request.java:326)
                    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
                    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                    at hudson.remoting.Engine$1$1.run(Engine.java:63)
                    at java.lang.Thread.run(Thread.java:724)

            #
            # A fatal error has been detected by the Java Runtime Environment:
            #
            # EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x000000000c8b3a90, pid=7136, tid=2024
            #
            # JRE version: 7.0_25-b17
            # Java VM: Java HotSpot(TM) 64-Bit Server VM (23.25-b01 mixed mode windows-amd64 compressed oops)
            # Problematic frame:
            # C 0x000000000c8b3a90
            #
            # Core dump written. Default location: C:\\hs_err_pid7136.mdmp
            #
            # An error report file with more information is saved as:
            # C:\\hs_err_pid7136.log

            When this occurs, here's the trailing end of the build console for this build:

            ...
            ...
            ------------------------------------------------------------------------
            [INFO] BUILD SUCCESS
            [INFO] ------------------------------------------------------------------------
            [INFO] Total time: 1:33.420s
            [INFO] Finished at: Wed Nov 20 16:16:09 EST 2013
            [INFO] Final Memory: 42M/272M
            [INFO] ------------------------------------------------------------------------
            [JENKINS] Archiving C:\jenkins\workspace\<redacted>\pom.xml to <redacted>SNAPSHOT.pom
            [JENKINS] Archiving C:\jenkins\workspace\<redacted>-SNAPSHOT.jar to <redacted>jar
            channel stopped
            FATAL: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
            hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
            at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41)
            at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34)
            at hudson.remoting.Request.call(Request.java:174)
            at hudson.remoting.Channel.call(Channel.java:714)
            at hudson.Launcher$RemoteLauncher.kill(Launcher.java:887)
            at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:585)
            at hudson.model.Run.execute(Run.java:1679)
            at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:509)
            at hudson.model.ResourceController.execute(ResourceController.java:88)
            at hudson.model.Executor.run(Executor.java:230)
            Caused by: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
            at hudson.remoting.Request.abort(Request.java:299)
            at hudson.remoting.Channel.terminate(Channel.java:774)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
            Caused by: java.net.SocketException: Connection reset
            at java.net.SocketInputStream.read(Unknown Source)
            at java.net.SocketInputStream.read(Unknown Source)
            at java.io.BufferedInputStream.fill(Unknown Source)
            at java.io.BufferedInputStream.read(Unknown Source)
            at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
            at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
            at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
            at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
            at java.io.ObjectInputStream.readObject0(Unknown Source)
            at java.io.ObjectInputStream.readObject(Unknown Source)
            at hudson.remoting.Command.readFrom(Command.java:92)
            at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

            The post build action that should have occurred at this point was the "Editable Email Notification" plugin. Not sure if it got that far or if the crash is immediately prior to that.

            This happens about 4-5 times out of every 100 builds. All of my build configs (for different projects, all maven builds, all executing the email-ext plugin afterwards...) do this.

            Occasionally, it will show the same stacktrace..

            WARNING: Failed to write winp.x64.dll
            java.io.FileNotFoundException: C:\Users\<redacted>\.jenkins\cache\jars\01\winp.x64.dll (The process can not access the file because it is being used by another process)

            .. but not crash.

            According to my google searches, a couple of people have been hitting this. Not sure if anyone has determined root cause yet. I'll attach what information I can, though my company will frown upon me attaching the core dump.


            Occasionally, remote windows slaves die with the following trace spit to the screen on the remote slave (I am running via command line, not as a service):

            {noformat}
            Nov 20, 2013 4:16:10 PM org.jvnet.winp.Native load
            WARNING: Failed to write winp.x64.dll
            java.io.FileNotFoundException: C:\Users\<redacted>\.jenkins\cache\jars\01\winp.x64.dll (The process can not access the file because it is being used by another process)
                    at java.io.FileOutputStream.open(Native Method)
                    at java.io.FileOutputStream.<init>(FileOutputStream.java:212)
                    at java.io.FileOutputStream.<init>(FileOutputStream.java:165)
                    at org.jvnet.winp.Native.load(Native.java:81)
                    at org.jvnet.winp.Native.<clinit>(Native.java:52)
                    at org.jvnet.winp.WinProcess.enableDebugPrivilege(WinProcess.java:200)
                    at hudson.util.ProcessTree$Windows.<clinit>(ProcessTree.java:477)
                    at hudson.util.ProcessTree.get(ProcessTree.java:336)
                    at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:899)
                    at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:890)
                    at hudson.remoting.UserRequest.perform(UserRequest.java:118)
                    at hudson.remoting.UserRequest.perform(UserRequest.java:48)
                    at hudson.remoting.Request$2.run(Request.java:326)
                    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
                    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                    at hudson.remoting.Engine$1$1.run(Engine.java:63)
                    at java.lang.Thread.run(Thread.java:724)

            #
            # A fatal error has been detected by the Java Runtime Environment:
            #
            # EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x000000000c8b3a90, pid=7136, tid=2024
            #
            # JRE version: 7.0_25-b17
            # Java VM: Java HotSpot(TM) 64-Bit Server VM (23.25-b01 mixed mode windows-amd64 compressed oops)
            # Problematic frame:
            # C 0x000000000c8b3a90
            #
            # Core dump written. Default location: C:\\hs_err_pid7136.mdmp
            #
            # An error report file with more information is saved as:
            # C:\\hs_err_pid7136.log
            {noformat}

            When this occurs, here's the trailing end of the build console for this build:

            {noformat}
            ...
            ...
            ------------------------------------------------------------------------
            [INFO] BUILD SUCCESS
            [INFO] ------------------------------------------------------------------------
            [INFO] Total time: 1:33.420s
            [INFO] Finished at: Wed Nov 20 16:16:09 EST 2013
            [INFO] Final Memory: 42M/272M
            [INFO] ------------------------------------------------------------------------
            [JENKINS] Archiving C:\jenkins\workspace\<redacted>\pom.xml to <redacted>SNAPSHOT.pom
            [JENKINS] Archiving C:\jenkins\workspace\<redacted>-SNAPSHOT.jar to <redacted>jar
            channel stopped
            FATAL: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
            hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
            at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41)
            at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34)
            at hudson.remoting.Request.call(Request.java:174)
            at hudson.remoting.Channel.call(Channel.java:714)
            at hudson.Launcher$RemoteLauncher.kill(Launcher.java:887)
            at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:585)
            at hudson.model.Run.execute(Run.java:1679)
            at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:509)
            at hudson.model.ResourceController.execute(ResourceController.java:88)
            at hudson.model.Executor.run(Executor.java:230)
            Caused by: hudson.remoting.RequestAbortedException: java.net.SocketException: Connection reset
            at hudson.remoting.Request.abort(Request.java:299)
            at hudson.remoting.Channel.terminate(Channel.java:774)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
            Caused by: java.net.SocketException: Connection reset
            at java.net.SocketInputStream.read(Unknown Source)
            at java.net.SocketInputStream.read(Unknown Source)
            at java.io.BufferedInputStream.fill(Unknown Source)
            at java.io.BufferedInputStream.read(Unknown Source)
            at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:77)
            at java.io.ObjectInputStream$PeekInputStream.peek(Unknown Source)
            at java.io.ObjectInputStream$BlockDataInputStream.peek(Unknown Source)
            at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Unknown Source)
            at java.io.ObjectInputStream.readObject0(Unknown Source)
            at java.io.ObjectInputStream.readObject(Unknown Source)
            at hudson.remoting.Command.readFrom(Command.java:92)
            at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
            {noformat}

            The post build action that should have occurred at this point was the "Editable Email Notification" plugin. Not sure if it got that far or if the crash is immediately prior to that.

            This happens about 4-5 times out of every 100 builds. All of my build configs (for different projects, all maven builds, all executing the email-ext plugin afterwards...) do this.

            Occasionally, it will show the same stacktrace on the remote slave..

            WARNING: Failed to write winp.x64.dll
            java.io.FileNotFoundException: C:\Users\<redacted>\.jenkins\cache\jars\01\winp.x64.dll (The process can not access the file because it is being used by another process)

            .. but not crash.

            According to my google searches, a couple of people have been hitting this, though I couldn't find a JIRA issue logged against it. Not sure if anyone has determined root cause yet. I'll attach what information I can, though my company will frown upon me attaching the core dump, so I might not be able to do that.


            borisivan boris ivan made changes -
            Attachment hs_err_pid7136.txt [ 24735 ]
            wolfs Stefan Wolf made changes -
            Assignee wolfs [ wolfs ]
            Component/s core [ 15593 ]
            Component/s all-changes [ 15911 ]
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue is related to JENKINS-20913 [ JENKINS-20913 ]
            danielbeck Daniel Beck made changes -
            Status Open [ 1 ] Resolved [ 5 ]
            Resolution Duplicate [ 3 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 152568 ] JNJira + In-Review [ 194265 ]

              People

              • Assignee:
                Unassigned
                Reporter:
                borisivan boris ivan
              • Votes:
                7 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: