Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-20913

[jar caching] - winp.x64.dll access conflicts

    Details

    • Similar Issues:

      Description

      Seems that JAR caching also takes DLL files on Windows.
      Thanks to Windows architecture (), it leads to access conflicts, because DDLs may be locked by other process.

      The issue is being heavily reproduced during the process. After several hours, any process termination fails with the FileNotFoundException exception (see the log below).

      Probably, the high frequency of the issue is somehow related to "Cygwin Process Killer" plugin. On my installation it calls additional remote call on the node before any process termination (it is an interim workaround for JENKINS-19156 and JENKINS-20289 in the custom 1.509.4 core build).

      Dec 07, 2013 7:02:28 AM org.jvnet.winp.Native load
      WARNING: Failed to write winp.x64.dll
      java.io.FileNotFoundException: C:\Users\XXX\.jenkins\cache\jars\01\winp.x64.dll (The process cannot access the file because it is being used by another process)
      at java.io.FileOutputStream.open(Native Method)
      at java.io.FileOutputStream.<init>(Unknown Source)
      at java.io.FileOutputStream.<init>(Unknown Source)
      at org.jvnet.winp.Native.load(Native.java:81)
      at org.jvnet.winp.Native.<clinit>(Native.java:52)
      at org.jvnet.winp.WinProcess.enableDebugPrivilege(WinProcess.java:200)
      at hudson.util.ProcessTree$Windows.<clinit>(ProcessTree.java:470)
      at hudson.util.ProcessTree.get(ProcessTree.java:335)
      at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:899)
      at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:890)
      at hudson.remoting.UserRequest.perform(UserRequest.java:118)
      at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      at hudson.remoting.Request$2.run(Request.java:326)
      at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      at java.util.concurrent.FutureTask.run(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at hudson.remoting.Engine$1$1.run(Engine.java:63)
      at java.lang.Thread.run(Unknown Source)

      Dec 07, 2013 7:02:35 AM hudson.util.ProcessTree get
      WARNING: Failed to load winp. Reverting to the default
      java.lang.UnsatisfiedLinkError: Native Library C:\Users\XXX\.jenkins\cache\jars\01\winp.x64.dll already loaded in another classloader
      at java.lang.ClassLoader.loadLibrary1(Unknown Source)
      at java.lang.ClassLoader.loadLibrary0(Unknown Source)
      at java.lang.ClassLoader.loadLibrary(Unknown Source)
      at java.lang.Runtime.load0(Unknown Source)
      at java.lang.System.load(Unknown Source)
      at org.jvnet.winp.Native.loadDll(Native.java:158)
      at org.jvnet.winp.Native.load(Native.java:90)
      at org.jvnet.winp.Native.<clinit>(Native.java:52)
      at org.jvnet.winp.WinProcess.enableDebugPrivilege(WinProcess.java:200)
      at hudson.util.ProcessTree$Windows.<clinit>(ProcessTree.java:470)
      at hudson.util.ProcessTree.get(ProcessTree.java:335)
      at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:899)
      at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:890)
      at hudson.remoting.UserRequest.perform(UserRequest.java:118)
      at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      at hudson.remoting.Request$2.run(Request.java:326)
      at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      at java.util.concurrent.FutureTask.run(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at hudson.remoting.Engine$1$1.run(Engine.java:63)
      at java.lang.Thread.run(Unknown Source)

        Attachments

          Issue Links

            Activity

            Hide
            borisivan boris ivan added a comment -

            Yeah, this one is killing me. This is an exact duplicate of the bug I logged previously: 20759. Hoping to see some activity on this soon, it's extraordinarily painful. Curious, do your remote build agents run as a process or from the command line (cmd/powershell/etc)?

            Show
            borisivan boris ivan added a comment - Yeah, this one is killing me. This is an exact duplicate of the bug I logged previously: 20759. Hoping to see some activity on this soon, it's extraordinarily painful. Curious, do your remote build agents run as a process or from the command line (cmd/powershell/etc)?
            Hide
            borisivan boris ivan added a comment -

            One additional note: with other CI servers (TC) with the same JRE running the same Maven goal (integration-test) with the same version of Maven on a build agent running the same version of windows, etc... this has never occurred. Therefore I don't think it's something that is contained within the Maven build itself. It's either plugin related or something else Jenkins specific that interacts with this that causes the scenario to occur. But on each of my 6 remote Jenkins build slaves running various different Maven projects, this always occurs 1-2 times a week.

            Show
            borisivan boris ivan added a comment - One additional note: with other CI servers (TC) with the same JRE running the same Maven goal (integration-test) with the same version of Maven on a build agent running the same version of windows, etc... this has never occurred. Therefore I don't think it's something that is contained within the Maven build itself. It's either plugin related or something else Jenkins specific that interacts with this that causes the scenario to occur. But on each of my 6 remote Jenkins build slaves running various different Maven projects, this always occurs 1-2 times a week.
            Hide
            kishorerp kishorerp added a comment - - edited

            We are seeing this on win8/8.1 slaves once in a week!!
            slave.jar crashes with the above error
            we are on Jenkins LTS 1.532.1 running on CentOs6.5
            also, at the end of each job, the above issue occurs everytime!!

            Show
            kishorerp kishorerp added a comment - - edited We are seeing this on win8/8.1 slaves once in a week!! slave.jar crashes with the above error we are on Jenkins LTS 1.532.1 running on CentOs6.5 also, at the end of each job, the above issue occurs everytime!!
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            @Kishore

            If I understand toy correctly, you see the following issue:

            Unable to render embedded object: File (..can this be fixed please) not found.

            If yes, I suppose that this is an another issue. Please create ticket for it

            Show
            oleg_nenashev Oleg Nenashev added a comment - @Kishore If I understand toy correctly, you see the following issue: Unable to render embedded object: File (..can this be fixed please) not found. If yes, I suppose that this is an another issue. Please create ticket for it
            Hide
            kishorerp kishorerp added a comment -

            @Oleg

            Not sure how that embedded object got added.
            We are seeing the same [jar caching] - winp.x64.dll access conflicts mentioned above

            Show
            kishorerp kishorerp added a comment - @Oleg Not sure how that embedded object got added. We are seeing the same [jar caching] - winp.x64.dll access conflicts mentioned above
            Hide
            rdmurali Murali Devakumar added a comment -

            This issue is killing the whole continuous integration process. Do we have any workaround or any time line to get this fixed?

            Show
            rdmurali Murali Devakumar added a comment - This issue is killing the whole continuous integration process. Do we have any workaround or any time line to get this fixed?
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            @Murali
            Currently there is no solutions or ETAs.

            As a workaround, you can disable Jar caching by calling a "hudson.remoting.Engine.current().setJarCache(null)" to disable the cache (e.g. call a System Groovy script on the node).

            // Probably, it is possible to pass null via "-jar-cache" CLI parameter.

            Show
            oleg_nenashev Oleg Nenashev added a comment - @Murali Currently there is no solutions or ETAs. As a workaround, you can disable Jar caching by calling a "hudson.remoting.Engine.current().setJarCache(null)" to disable the cache (e.g. call a System Groovy script on the node). // Probably, it is possible to pass null via "-jar-cache" CLI parameter.
            Hide
            rdmurali Murali Devakumar added a comment -

            @Oleg,
            Thanks for quick response.

            Can you please be more elaborate on either of options and I am running slave as windows service, which limits me with CLI parameters.

            Show
            rdmurali Murali Devakumar added a comment - @Oleg, Thanks for quick response. Can you please be more elaborate on either of options and I am running slave as windows service, which limits me with CLI parameters.
            Hide
            kishorerp kishorerp added a comment -

            @Oleg

            Thanks for your quick response, could you please mention the groovy script that you have told above, this is very predominant in our environment and our ci is going for a toss!

            Show
            kishorerp kishorerp added a comment - @Oleg Thanks for your quick response, could you please mention the groovy script that you have told above, this is very predominant in our environment and our ci is going for a toss!
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Disclaimer: This approach works on my installation (1.509.4), but it actually corrupts the normal flow and may lead to serious classloading issues.

            You will need a Groovy plugin...

            A simple manual approach:
            1) Go ${JENKINS_URL}/computer/${NODE_NAME}/script
            2) Call "http://arcjenkinsdev/computer/ru20-hw-farm23/script"

            After that, the Jar caching will be disabled till the node's reconnection. Please note that it may affect the performance of node.

            In order to setup an automatic cache disabling, just use Startup Trigger together with a System Groovy Script build step.
            https://wiki.jenkins-ci.org/display/JENKINS/Startup+Trigger

            Show
            oleg_nenashev Oleg Nenashev added a comment - Disclaimer: This approach works on my installation (1.509.4), but it actually corrupts the normal flow and may lead to serious classloading issues. You will need a Groovy plugin... A simple manual approach: 1) Go ${JENKINS_URL}/computer/${NODE_NAME}/script 2) Call "http://arcjenkinsdev/computer/ru20-hw-farm23/script" After that, the Jar caching will be disabled till the node's reconnection. Please note that it may affect the performance of node. In order to setup an automatic cache disabling, just use Startup Trigger together with a System Groovy Script build step. https://wiki.jenkins-ci.org/display/JENKINS/Startup+Trigger
            Hide
            kishorerp kishorerp added a comment - - edited

            @Oleg
            thanks for your response
            not sure i can access the URL of the script mentioned, can you please check.
            when you say this affects performance of a node, could you be a little specific in terms what would be affected, (network bandwidth or cpu cycles?)
            Meanwhile i tried
            to pass null via "-jar-cache" CLI parameter
            It is not possible to do so, the slave doesn't launch via jnlp

            Show
            kishorerp kishorerp added a comment - - edited @Oleg thanks for your response not sure i can access the URL of the script mentioned, can you please check. when you say this affects performance of a node, could you be a little specific in terms what would be affected, (network bandwidth or cpu cycles?) Meanwhile i tried to pass null via "-jar-cache" CLI parameter It is not possible to do so, the slave doesn't launch via jnlp
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            @Kishore
            > not sure i can access the URL of the script mentioned
            There should be a "Script Console" action on the slave's main page

            > when you say this affects performance of a node, could you be a little specific in terms what would be affected, (network bandwidth or cpu cycles?)
            The JAR caching will be disabled, so your slave will have do download all classed from the master node. So the first run of jobs/etc may take more time, but all other runs won't be affected. Actually, it affects distributed systems only.

            Show
            oleg_nenashev Oleg Nenashev added a comment - @Kishore > not sure i can access the URL of the script mentioned There should be a "Script Console" action on the slave's main page > when you say this affects performance of a node, could you be a little specific in terms what would be affected, (network bandwidth or cpu cycles?) The JAR caching will be disabled, so your slave will have do download all classed from the master node. So the first run of jobs/etc may take more time, but all other runs won't be affected. Actually, it affects distributed systems only.
            Hide
            kishorerp kishorerp added a comment -

            Thanks Oleg, have run it once, will observe if the issue occurs again

            Show
            kishorerp kishorerp added a comment - Thanks Oleg, have run it once, will observe if the issue occurs again
            Hide
            treaz Horia Constantin added a comment -

            @Oleg
            I'm following your recommendation:
            1) Added a freestyle job triggered via the Startup trigger plugin.
            2) In the build section, Execute system Groovy script -> Groovy command
            3) Just a single line in that field: hudson.remoting.Engine.current().setJarCache(null)

            Running the job throws me a "FATAL: Cannot invoke method setJarCache() on null object".
            Doing a print of hudson.remoting.Engine.current() in the same build job shows it as null.

            If I do this through "There should be a "Script Console" action on the slave's main page" everything runs fine (and I see that hudson.remoting.Engine.current() is not null).

            Would you have any idea why this is?

            Show
            treaz Horia Constantin added a comment - @Oleg I'm following your recommendation: 1) Added a freestyle job triggered via the Startup trigger plugin. 2) In the build section, Execute system Groovy script -> Groovy command 3) Just a single line in that field: hudson.remoting.Engine.current().setJarCache(null) Running the job throws me a "FATAL: Cannot invoke method setJarCache() on null object". Doing a print of hudson.remoting.Engine.current() in the same build job shows it as null. If I do this through "There should be a "Script Console" action on the slave's main page" everything runs fine (and I see that hudson.remoting.Engine.current() is not null). Would you have any idea why this is?
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            @Horia Constantin
            It was a my mistake.
            System Groovy script always executes on the master instead of the target slave.

            You can use the Scriptler plugin:
            1) Create a Scriptler script

            • Write down the groovy call into the script window
            • Allow execution by users having "Run Script Permission"
            • Check that "Run always on master" is disabled
              2) Call this script as a build step inside your job
            Show
            oleg_nenashev Oleg Nenashev added a comment - @Horia Constantin It was a my mistake. System Groovy script always executes on the master instead of the target slave. You can use the Scriptler plugin: 1) Create a Scriptler script Write down the groovy call into the script window Allow execution by users having "Run Script Permission" Check that "Run always on master" is disabled 2) Call this script as a build step inside your job
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Added the lts-candidate label.
            The issue makes all Windows process termination flows unreliable

            Show
            oleg_nenashev Oleg Nenashev added a comment - Added the lts-candidate label. The issue makes all Windows process termination flows unreliable
            Hide
            treaz Horia Constantin added a comment -

            @Oleg, your latest suggestion worked.

            The job that runs great for me: using the Startup trigger plugin, triggered for a specific label. But, due to some issues in the Startup trigger plugin, with some tweaking:
            install manually version 2.4
            configure the job "This build is parameterized", with a label parameter (the same label as before) and with "Run on all nodes matching the label" checked.

            Show
            treaz Horia Constantin added a comment - @Oleg, your latest suggestion worked. The job that runs great for me: using the Startup trigger plugin, triggered for a specific label. But, due to some issues in the Startup trigger plugin, with some tweaking: install manually version 2.4 configure the job "This build is parameterized", with a label parameter (the same label as before) and with "Run on all nodes matching the label" checked.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Created a PR, which allows to disable JAR caching via the CLI option
            https://github.com/jenkinsci/remoting/pull/21

            BTW, it is just a workaround. The complete resolution requires some tweaks in the caching procedures.

            Show
            oleg_nenashev Oleg Nenashev added a comment - Created a PR, which allows to disable JAR caching via the CLI option https://github.com/jenkinsci/remoting/pull/21 BTW, it is just a workaround. The complete resolution requires some tweaks in the caching procedures.
            Hide
            kohsuke Kohsuke Kawaguchi added a comment -

            I've made the change in winp 1.19 to be more intelligent about overwriting the file.

            Historically winp uses timestamp based up-to-date check, but because remoting uses timestamp of jar files for LRU checks, this overwrite check was broken. In 1.19 I've switched to checksum-based up-to-date check, which resolves the problem.

            But as I was doing that, I noticed a deeper issue here. "Native Library ...\winp.x64.dll already loaded in another classloader " is likely because JNLP slave is retrying after it lost a connection with the master. During the previous connection, the slave has loaded one copy of winp.dll, and after reconnection it's trying to load one more, which is obviously failing.

            To properly solve this requires a change in the client (and possibly winsw), so that after losing a connection the slave will have winsw relaunch a new service, so that JVM will start fresh.

            This has been a long standing TODO, but it's a good change as it would prevent memory leak and static state clutter over time.

            Show
            kohsuke Kohsuke Kawaguchi added a comment - I've made the change in winp 1.19 to be more intelligent about overwriting the file. Historically winp uses timestamp based up-to-date check, but because remoting uses timestamp of jar files for LRU checks, this overwrite check was broken. In 1.19 I've switched to checksum-based up-to-date check, which resolves the problem. But as I was doing that, I noticed a deeper issue here. "Native Library ...\winp.x64.dll already loaded in another classloader " is likely because JNLP slave is retrying after it lost a connection with the master. During the previous connection, the slave has loaded one copy of winp.dll, and after reconnection it's trying to load one more, which is obviously failing. To properly solve this requires a change in the client (and possibly winsw), so that after losing a connection the slave will have winsw relaunch a new service, so that JVM will start fresh. This has been a long standing TODO, but it's a good change as it would prevent memory leak and static state clutter over time.
            Hide
            kohsuke Kohsuke Kawaguchi added a comment -

            I suppose another simple workaround fix would be to copy winp.dll into another file and load it. It'd work around "already loaded in another classloader" problem.

            Show
            kohsuke Kohsuke Kawaguchi added a comment - I suppose another simple workaround fix would be to copy winp.dll into another file and load it. It'd work around "already loaded in another classloader" problem.
            Hide
            kishorerp kishorerp added a comment -

            @Kohsuke, Do we have a rough idea on which LTS release would have this fix. We are using LTS 1.532 on CentOS for our CI which is expected to be online 24x7x365, and we see this issue that seems to pop up on around 30 of our slaves every week or so.
            @Oleg, we would not want the workaround mentioned since all our slaves are not expected to have any overhead whatsoever due to any other running process.

            Show
            kishorerp kishorerp added a comment - @Kohsuke, Do we have a rough idea on which LTS release would have this fix. We are using LTS 1.532 on CentOS for our CI which is expected to be online 24x7x365, and we see this issue that seems to pop up on around 30 of our slaves every week or so. @Oleg, we would not want the workaround mentioned since all our slaves are not expected to have any overhead whatsoever due to any other running process.
            Hide
            oleg_nenashev Oleg Nenashev added a comment - - edited

            @Koshuke
            I've also noticed that I've selected an improper way.

            > To properly solve this requires a change in the client (and possibly winsw), so that after losing a connection the slave will have winsw relaunch a new service, so that JVM will start fresh.
            I suppose it slightly conflicts with "autonomous slaves" approach (slaves may continue operations on temporary disconnects), which usually appears during the infrastructure reliability discussions.

            What about checking if the DLL has been already loaded? http://stackoverflow.com/questions/1007861/how-do-i-get-a-list-of-jni-libraries-which-are-loaded/
            I suppose that the slave may just log a warning in such case

            @Kishore
            I'd prefer to recall the proposed Groovy script above, because it may lead to NPEs on uncached resource loads after restarts or plugin updates.
            Such solution should be used only on long-run slaves with "warm up" jobs and without class unloading.

            Show
            oleg_nenashev Oleg Nenashev added a comment - - edited @Koshuke I've also noticed that I've selected an improper way. > To properly solve this requires a change in the client (and possibly winsw), so that after losing a connection the slave will have winsw relaunch a new service, so that JVM will start fresh. I suppose it slightly conflicts with "autonomous slaves" approach (slaves may continue operations on temporary disconnects), which usually appears during the infrastructure reliability discussions. What about checking if the DLL has been already loaded? http://stackoverflow.com/questions/1007861/how-do-i-get-a-list-of-jni-libraries-which-are-loaded/ I suppose that the slave may just log a warning in such case @Kishore I'd prefer to recall the proposed Groovy script above, because it may lead to NPEs on uncached resource loads after restarts or plugin updates. Such solution should be used only on long-run slaves with "warm up" jobs and without class unloading.
            Hide
            kohsuke Kohsuke Kawaguchi added a comment -

            This bug is not even marked as fixed yet, so it's way too early to predict LTS backporting plan. We'd first have to fix this in the trunk.

            Knowing whether the library is already loaded or not doesn't really help either, because if it's loaded through another classloader, we won't be able to reuse currently loaded copy.

            Show
            kohsuke Kohsuke Kawaguchi added a comment - This bug is not even marked as fixed yet, so it's way too early to predict LTS backporting plan. We'd first have to fix this in the trunk. Knowing whether the library is already loaded or not doesn't really help either, because if it's loaded through another classloader, we won't be able to reuse currently loaded copy.
            Hide
            kohsuke Kohsuke Kawaguchi added a comment -

            I'm now marking this issue fixed with winp 1.19.

            The other part of the issue (of not reusing the same JVM for multiple reconnections to the master) will be tracked in JENKINS-19055,

            Show
            kohsuke Kohsuke Kawaguchi added a comment - I'm now marking this issue fixed with winp 1.19. The other part of the issue (of not reusing the same JVM for multiple reconnections to the master) will be tracked in JENKINS-19055 ,
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Kohsuke Kawaguchi
            Path:
            core/pom.xml
            http://jenkins-ci.org/commit/jenkins/4d969004c121bcef07189dfe89fe3347646e741a
            Log:
            JENKINS-20913 Use winp 1.19

            Updated to a newer version

            (cherry picked from commit 6b350af44dee8e1e6cad4cce67941b3ebf88ac52)

            Conflicts:
            core/pom.xml

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/pom.xml http://jenkins-ci.org/commit/jenkins/4d969004c121bcef07189dfe89fe3347646e741a Log: JENKINS-20913 Use winp 1.19 Updated to a newer version (cherry picked from commit 6b350af44dee8e1e6cad4cce67941b3ebf88ac52) Conflicts: core/pom.xml
            Hide
            ariebell Arie Belenky added a comment -

            Hi,
            This issue started to reproduce during the last week on our AWS Windows Server 2012 R2 base machines.
            Jenkins ver. 1.640
            Traceback:
            SEVERE: I/O error in channel channel
            java.net.SocketException: Connection reset
            at java.net.SocketInputStream.read(Unknown Source)
            at java.net.SocketInputStream.read(Unknown Source)
            at java.io.BufferedInputStream.fill(Unknown Source)
            at java.io.BufferedInputStream.read(Unknown Source)
            at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:82)
            at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:72)
            at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:103)
            at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
            at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

            Dec 31, 2015 5:17:38 PM hudson.remoting.Request$2 run
            SEVERE: Failed to send back a reply
            java.net.SocketException: Connection reset by peer: socket write error
            at java.net.SocketOutputStream.socketWrite0(Native Method)
            at java.net.SocketOutputStream.socketWrite(Unknown Source)
            at java.net.SocketOutputStream.write(Unknown Source)
            at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
            at java.io.BufferedOutputStream.write(Unknown Source)
            at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:94)
            at hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:89)
            at hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:58)
            at java.io.OutputStream.write(Unknown Source)
            at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45)
            at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:45)
            at hudson.remoting.Channel.send(Channel.java:582)
            at hudson.remoting.Request$2.run(Request.java:340)
            at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
            at java.util.concurrent.FutureTask.run(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at hudson.remoting.Engine$1$1.run(Engine.java:62)
            at java.lang.Thread.run(Unknown Source)

            Dec 31, 2015 5:17:38 PM hudson.remoting.jnlp.Main$CuiListener status
            INFO: Terminated
            Dec 31, 2015 5:17:48 PM hudson.remoting.jnlp.Main$CuiListener status
            INFO: Locating server among https://jenkins.interjunk.com/
            Dec 31, 2015 5:17:48 PM hudson.remoting.jnlp.Main$CuiListener status
            INFO: Handshaking
            Dec 31, 2015 5:17:48 PM hudson.remoting.jnlp.Main$CuiListener status
            INFO: Connecting to jenkins.interjunk.com:15400
            Dec 31, 2015 5:17:48 PM hudson.remoting.jnlp.Main$CuiListener status
            INFO: Trying protocol: JNLP2-connect
            Dec 31, 2015 5:17:48 PM hudson.remoting.jnlp.Main$CuiListener status
            INFO: Connected
            Dec 31, 2015 5:19:00 PM hudson.util.ProcessTree get
            WARNING: Failed to load winp. Reverting to the default
            java.lang.UnsatisfiedLinkError: Native Library C:\Users\automation\.jenkins\cache\jars\4A\winp.x64.22D9AB310A3FA2D96B6E03A836A47724.dll
            already loaded in another classloader
            at java.lang.ClassLoader.loadLibrary1(Unknown Source)
            at java.lang.ClassLoader.loadLibrary0(Unknown Source)
            at java.lang.ClassLoader.loadLibrary(Unknown Source)
            at java.lang.Runtime.load0(Unknown Source)
            at java.lang.System.load(Unknown Source)
            at org.jvnet.winp.Native.loadDll(Native.java:190)
            at org.jvnet.winp.Native.load(Native.java:122)
            at org.jvnet.winp.Native.<clinit>(Native.java:56)
            at org.jvnet.winp.WinProcess.enableDebugPrivilege(WinProcess.java:212)
            at hudson.util.ProcessTree$Windows.<clinit>(ProcessTree.java:494)
            at hudson.util.ProcessTree.get(ProcessTree.java:345)
            at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:965)
            at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:956)
            at hudson.remoting.UserRequest.perform(UserRequest.java:120)
            at hudson.remoting.UserRequest.perform(UserRequest.java:48)
            at hudson.remoting.Request$2.run(Request.java:326)
            at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
            at java.util.concurrent.FutureTask.run(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at hudson.remoting.Engine$1$1.run(Engine.java:62)
            at java.lang.Thread.run(Unknown Source)

            Show
            ariebell Arie Belenky added a comment - Hi, This issue started to reproduce during the last week on our AWS Windows Server 2012 R2 base machines. Jenkins ver. 1.640 Traceback: SEVERE: I/O error in channel channel java.net.SocketException: Connection reset at java.net.SocketInputStream.read(Unknown Source) at java.net.SocketInputStream.read(Unknown Source) at java.io.BufferedInputStream.fill(Unknown Source) at java.io.BufferedInputStream.read(Unknown Source) at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:82) at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:72) at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:103) at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) Dec 31, 2015 5:17:38 PM hudson.remoting.Request$2 run SEVERE: Failed to send back a reply java.net.SocketException: Connection reset by peer: socket write error at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(Unknown Source) at java.net.SocketOutputStream.write(Unknown Source) at java.io.BufferedOutputStream.flushBuffer(Unknown Source) at java.io.BufferedOutputStream.write(Unknown Source) at hudson.remoting.ChunkedOutputStream.sendFrame(ChunkedOutputStream.java:94) at hudson.remoting.ChunkedOutputStream.drain(ChunkedOutputStream.java:89) at hudson.remoting.ChunkedOutputStream.write(ChunkedOutputStream.java:58) at java.io.OutputStream.write(Unknown Source) at hudson.remoting.ChunkedCommandTransport.writeBlock(ChunkedCommandTransport.java:45) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.write(AbstractSynchronousByteArrayCommandTransport.java:45) at hudson.remoting.Channel.send(Channel.java:582) at hudson.remoting.Request$2.run(Request.java:340) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:62) at java.lang.Thread.run(Unknown Source) Dec 31, 2015 5:17:38 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Terminated Dec 31, 2015 5:17:48 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among https://jenkins.interjunk.com/ Dec 31, 2015 5:17:48 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Handshaking Dec 31, 2015 5:17:48 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to jenkins.interjunk.com:15400 Dec 31, 2015 5:17:48 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Trying protocol: JNLP2-connect Dec 31, 2015 5:17:48 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connected Dec 31, 2015 5:19:00 PM hudson.util.ProcessTree get WARNING: Failed to load winp. Reverting to the default java.lang.UnsatisfiedLinkError: Native Library C:\Users\automation\.jenkins\cache\jars\4A\winp.x64.22D9AB310A3FA2D96B6E03A836A47724.dll already loaded in another classloader at java.lang.ClassLoader.loadLibrary1(Unknown Source) at java.lang.ClassLoader.loadLibrary0(Unknown Source) at java.lang.ClassLoader.loadLibrary(Unknown Source) at java.lang.Runtime.load0(Unknown Source) at java.lang.System.load(Unknown Source) at org.jvnet.winp.Native.loadDll(Native.java:190) at org.jvnet.winp.Native.load(Native.java:122) at org.jvnet.winp.Native.<clinit>(Native.java:56) at org.jvnet.winp.WinProcess.enableDebugPrivilege(WinProcess.java:212) at hudson.util.ProcessTree$Windows.<clinit>(ProcessTree.java:494) at hudson.util.ProcessTree.get(ProcessTree.java:345) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:965) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:956) at hudson.remoting.UserRequest.perform(UserRequest.java:120) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:62) at java.lang.Thread.run(Unknown Source)
            Hide
            pcazes Pierre-Henri Cazes added a comment -

            Hi, this issue is reproduced on our servers.

            Jenkins ver. 1.625.1 :
            Master on CentOS 6
            Node in Windows 7, slave running as service, working !! (ie. NO java.lang.UnsatisfiedLinkError raised with winp dll)
            Node in Windows 7, slave running via CLA, NOT working (ie. java.lang.UnsatisfiedLinkError raised with winp dll)
            Node in Windows 8.1, not working in both cases ( as service, or via CLI)

            Show
            pcazes Pierre-Henri Cazes added a comment - Hi, this issue is reproduced on our servers. Jenkins ver. 1.625.1 : Master on CentOS 6 Node in Windows 7, slave running as service, working !! (ie. NO java.lang.UnsatisfiedLinkError raised with winp dll) Node in Windows 7, slave running via CLA, NOT working (ie. java.lang.UnsatisfiedLinkError raised with winp dll) Node in Windows 8.1, not working in both cases ( as service, or via CLI)
            Hide
            gmerkin Grigory Merkin added a comment -

            Kohsuke Kawaguchi, this bug seems to be caused by winp library issue https://github.com/kohsuke/winp/issues/26. Could you please promote a new release of fixed winp and update Jenkins to it in next LTS?

            Show
            gmerkin Grigory Merkin added a comment - Kohsuke Kawaguchi , this bug seems to be caused by winp library issue https://github.com/kohsuke/winp/issues/26 . Could you please promote a new release of fixed winp and update Jenkins to it in next LTS?
            Hide
            benh57 Ben Hines added a comment -

            We are also seeing this error pretty often.

            Show
            benh57 Ben Hines added a comment - We are also seeing this error pretty often.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Additional fix has been integrated towards 2.34

            Show
            oleg_nenashev Oleg Nenashev added a comment - Additional fix has been integrated towards 2.34
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            core/pom.xml
            http://jenkins-ci.org/commit/jenkins/63c2f6c5d7d154a3a0f58c54f04f9b1a25ea5385
            Log:
            Update winp to 1.24. In particular, it addresses issues like JENKINS-20913(https://issues.jenkins-ci.org/browse/JENKINS-20913) (#2619)

                1. Changes to be picked
                1. 1.24

            Release date: Nov 2, 2016

                1. 1.23

            Release date: Fev 16, 2015

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: core/pom.xml http://jenkins-ci.org/commit/jenkins/63c2f6c5d7d154a3a0f58c54f04f9b1a25ea5385 Log: Update winp to 1.24. In particular, it addresses issues like JENKINS-20913 ( https://issues.jenkins-ci.org/browse/JENKINS-20913 ) (#2619) Changes to be picked 1.24 Release date: Nov 2, 2016 Issue #22 ( https://github.com/kohsuke/winp/issues/22 ) - Winp sometimes kills random processes when using killRecursive. ( PR #23 ( https://github.com/kohsuke/winp/pull/23 )) [WINP-10] ( https://java.net/jira/browse/WINP-10 ) - Fix for `getCmdLineAndEnvVars()` which fails on x64 versions of Windows. ( PR #20 ( https://github.com/kohsuke/winp/pull/20 )) Issue #24 ( https://github.com/kohsuke/winp/issues/24 ) - Wrong folder when using the `winp.folder.preferred` system property (parent instead of the actual folder). ( PR #25 ( https://github.com/kohsuke/winp/pull/25 )) Issue #26 ( https://github.com/kohsuke/winp/issues/26 ), JENKINS-20913 ( https://issues.jenkins-ci.org/browse/JENKINS-20913 ) - Native class now tries loading DLLs via the temp location. ( PR #27 ( https://github.com/kohsuke/winp/pull/27 )) 1.23 Release date: Fev 16, 2015 Migrate native components to Visual Studio Community 2013. ( PR #14 ( https://github.com/kohsuke/winp/pull/14 )) Provide a `winp.unpack.dll.to.parent.dir` property, which disables DLL unpacking to the parent dir. ( PR #14 ( https://github.com/kohsuke/winp/pull/12 ))
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            core/pom.xml
            http://jenkins-ci.org/commit/jenkins/e66bfbafa99fc092c6f9fe51f8bf5be267340557
            Log:
            Update winp to 1.24. In particular, it addresses issues like JENKINS-20913(https://issues.jenkins-ci.org/browse/JENKINS-20913) (#2619)

                1. Changes to be picked
                1. 1.24

            Release date: Nov 2, 2016

                1. 1.23

            Release date: Fev 16, 2015

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: core/pom.xml http://jenkins-ci.org/commit/jenkins/e66bfbafa99fc092c6f9fe51f8bf5be267340557 Log: Update winp to 1.24. In particular, it addresses issues like JENKINS-20913 ( https://issues.jenkins-ci.org/browse/JENKINS-20913 ) (#2619) Changes to be picked 1.24 Release date: Nov 2, 2016 Issue #22 ( https://github.com/kohsuke/winp/issues/22 ) - Winp sometimes kills random processes when using killRecursive. ( PR #23 ( https://github.com/kohsuke/winp/pull/23 )) [WINP-10] ( https://java.net/jira/browse/WINP-10 ) - Fix for `getCmdLineAndEnvVars()` which fails on x64 versions of Windows. ( PR #20 ( https://github.com/kohsuke/winp/pull/20 )) Issue #24 ( https://github.com/kohsuke/winp/issues/24 ) - Wrong folder when using the `winp.folder.preferred` system property (parent instead of the actual folder). ( PR #25 ( https://github.com/kohsuke/winp/pull/25 )) Issue #26 ( https://github.com/kohsuke/winp/issues/26 ), JENKINS-20913 ( https://issues.jenkins-ci.org/browse/JENKINS-20913 ) - Native class now tries loading DLLs via the temp location. ( PR #27 ( https://github.com/kohsuke/winp/pull/27 )) 1.23 Release date: Fev 16, 2015 Migrate native components to Visual Studio Community 2013. ( PR #14 ( https://github.com/kohsuke/winp/pull/14 )) Provide a `winp.unpack.dll.to.parent.dir` property, which disables DLL unpacking to the parent dir. ( PR #14 ( https://github.com/kohsuke/winp/pull/12 )) (cherry picked from commit 63c2f6c5d7d154a3a0f58c54f04f9b1a25ea5385)

              People

              • Assignee:
                oleg_nenashev Oleg Nenashev
                Reporter:
                oleg_nenashev Oleg Nenashev
              • Votes:
                14 Vote for this issue
                Watchers:
                19 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: