Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-24319

Windows slave dying during build - unsatisfied link error

    Details

    • Similar Issues:

      Description

      Windows slaved builds fail with the following error on the master:

      11:56:20 FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Failed to abort
      11:56:20 hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Failed to abort
      11:56:20 at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41)
      11:56:20 at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34)
      11:56:20 at hudson.remoting.Request.call(Request.java:174)
      11:56:20 at hudson.remoting.Channel.call(Channel.java:739)
      11:56:20 at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:168)
      11:56:20 at com.sun.proxy.$Proxy58.join(Unknown Source)
      11:56:20 at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:956)
      11:56:20 at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:137)
      11:56:20 at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:97)
      11:56:20 at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
      11:56:20 at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
      11:56:20 at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:772)
      11:56:20 at hudson.model.Build$BuildExecution.build(Build.java:199)
      11:56:20 at hudson.model.Build$BuildExecution.doRun(Build.java:160)
      11:56:20 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:535)
      11:56:20 at hudson.model.Run.execute(Run.java:1740)
      11:56:20 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      11:56:20 at hudson.model.ResourceController.execute(ResourceController.java:88)
      11:56:20 at hudson.model.Executor.run(Executor.java:234)
      11:56:20 Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Failed to abort
      11:56:20 at hudson.remoting.Request.abort(Request.java:299)
      11:56:20 at hudson.remoting.Channel.terminate(Channel.java:802)
      11:56:20 at hudson.remoting.Channel$2.terminate(Channel.java:483)
      11:56:20 at hudson.remoting.AbstractByteArrayCommandTransport$1.terminate(AbstractByteArrayCommandTransport.java:72)
      11:56:20 at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.abort(NioChannelHub.java:195)
      11:56:20 at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:581)
      11:56:20 at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      11:56:20 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
      11:56:20 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      11:56:20 at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      11:56:20 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
      11:56:20 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
      11:56:20 at java.lang.Thread.run(Thread.java:695)
      11:56:20 Caused by: java.io.IOException: Failed to abort
      11:56:20 ... 9 more
      11:56:20 Caused by: java.io.IOException: Connection reset by peer
      11:56:20 at sun.nio.ch.FileDispatcher.read0(Native Method)
      11:56:20 at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
      11:56:20 at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
      11:56:20 at sun.nio.ch.IOUtil.read(IOUtil.java:171)
      11:56:20 at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245)
      11:56:20 at org.jenkinsci.remoting.nio.FifoBuffer$Pointer.receive(FifoBuffer.java:136)
      11:56:20 at org.jenkinsci.remoting.nio.FifoBuffer.receive(FifoBuffer.java:306)
      11:56:20 at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:514)
      11:56:20 ... 7 more

      And the following warning on the slave agent:

      Aug 19, 2014 11:51:09 AM hudson.util.ProcessTree get
      WARNING: Failed to load winp. Reverting to the default
      java.lang.UnsatisfiedLinkError: Native Library C:\Users\BuildWin\.jenkins\cache\ jars\12\winp.x64.FEF9CB80B43534DCA303AC36686258E8.dll already loaded in another classloader
      at java.lang.ClassLoader.loadLibrary1(ClassLoader.java:1931)
      at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1890)
      at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1851)
      at java.lang.Runtime.load0(Runtime.java:795)
      at java.lang.System.load(System.java:1062)
      at org.jvnet.winp.Native.loadDll(Native.java:189)
      at org.jvnet.winp.Native.load(Native.java:121)
      at org.jvnet.winp.Native.<clinit>(Native.java:55)
      at org.jvnet.winp.WinProcess.enableDebugPrivilege(WinProcess.java:200)
      at hudson.util.ProcessTree$Windows.<clinit>(ProcessTree.java:469)
      at hudson.util.ProcessTree.get(ProcessTree.java:328)
      at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:930)
      at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:921)
      at hudson.remoting.UserRequest.perform(UserRequest.java:118)
      at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      at hudson.remoting.Request$2.run(Request.java:328)
      at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecut orService.java:72)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:615)
      at hudson.remoting.Engine$1$1.run(Engine.java:63)
      at java.lang.Thread.run(Thread.java:745)

      Currently restarting the slave agent fixes the issue for a few builds but after about an hour the builds start failing again. It doesn't seem to be a memory issue as I've set the Xmx2048m parameter and the monitoring plugin seems fairly normal in it's reports.

        Attachments

          Issue Links

            Activity

            Hide
            tzedar Tony Zedar added a comment -

            We saw the same issue last night...
            The slave eventually restarted itself but we lost a test in the process.
            Only difference is that our master is also a windows machine.

            Show
            tzedar Tony Zedar added a comment - We saw the same issue last night... The slave eventually restarted itself but we lost a test in the process. Only difference is that our master is also a windows machine.
            Hide
            cnagaraju Chinmayee Nagaraju added a comment -

            any update on this ? seeing the same issue on a Windows VM slave connecting to windows master

            Show
            cnagaraju Chinmayee Nagaraju added a comment - any update on this ? seeing the same issue on a Windows VM slave connecting to windows master
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Closing it as a duplicate of JENKINS-20913, because it is likely a root cause of this failure. If the issue still happens after the upgrade to 2.32.1, please reopen it

            Show
            oleg_nenashev Oleg Nenashev added a comment - Closing it as a duplicate of JENKINS-20913 , because it is likely a root cause of this failure. If the issue still happens after the upgrade to 2.32.1, please reopen it
            Hide
            remik P4 added a comment - - edited

            Recently we have added second Windows agent to our Windows master and immediately started to observe this issue. No such problems on the first Windows agent that is running here for months. Both systems are Microsoft Windows [Version 10.0.18363.720].

            This is sanitized log fragment from the agent:

            mar 31, 2020 10:00:57 PM org.jvnet.winp.Native loadByUrl
             WARNING: Failed to load DLL from static location
             java.lang.UnsatisfiedLinkError: Native Library J:\Jenkins\remoting\jarCache\44\winp.x64.BFFE30B3B50581290B1866EF8D48C609.dll already loaded in another classloader
             at java.lang.ClassLoader.loadLibrary0(Unknown Source)
             at java.lang.ClassLoader.loadLibrary(Unknown Source)
             at java.lang.Runtime.load0(Unknown Source)
             at java.lang.System.load(Unknown Source)
             at org.jvnet.winp.Native.loadDll(Native.java:242)
             at org.jvnet.winp.Native.loadByUrl(Native.java:162)
             at org.jvnet.winp.Native.load(Native.java:124)
             at org.jvnet.winp.Native.<clinit>(Native.java:93)
             at org.jvnet.winp.WinProcess.enableDebugPrivilege(WinProcess.java:245)
             at hudson.util.ProcessTree$Windows.<clinit>(ProcessTree.java:701)
             at hudson.util.ProcessTree.get(ProcessTree.java:447)
             at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1100)
             at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1091)
             at hudson.remoting.UserRequest.perform(UserRequest.java:211)
             at hudson.remoting.UserRequest.perform(UserRequest.java:54)
             at hudson.remoting.Request$2.run(Request.java:369)
             at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
             at java.util.concurrent.FutureTask.run(Unknown Source)
             at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
             at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
             at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:97)
             at java.lang.Thread.run(Unknown Source)
            
            kwi 01, 2020 12:25:05 AM hudson.remoting.jnlp.Main$CuiListener status
             INFO: Terminated
             kwi 01, 2020 12:25:05 AM hudson.Launcher$RemoteLaunchCallable$1 join
             INFO: Failed to synchronize IO streams on the channel hudson.remoting.Channel@1aa3bc97:JNLP4-connect connection to jenkins.<mycompany.domain>/<master IPaddress>:6666
             hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@1aa3bc97:JNLP4-connect connection to jenkins.<mycompany.domain>/<master IPaddress>:6666": Remote call on JNLP4-connect connection to jenkins.<mycompany.domain>/<master IPaddress>:6666 failed. The channel is closing down or has closed down
             at hudson.remoting.Channel.call(Channel.java:948)
             at hudson.remoting.Channel.syncIO(Channel.java:1683)
             at hudson.Launcher$RemoteLaunchCallable$1.join(Launcher.java:1328)
             at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
             at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
             at java.lang.reflect.Method.invoke(Unknown Source)
             at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929)
             at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)
             at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)
             at hudson.remoting.UserRequest.perform(UserRequest.java:211)
             at hudson.remoting.UserRequest.perform(UserRequest.java:54)
             at hudson.remoting.Request$2.run(Request.java:369)
             at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
             at java.util.concurrent.FutureTask.run(Unknown Source)
             at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
             at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
             at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:97)
             at java.lang.Thread.run(Unknown Source)
             Caused by: java.nio.channels.ClosedChannelException
             at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
             at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1800(BIONetworkLayer.java:48)
             at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:264)
             ... 4 more
            
            kwi 01, 2020 12:25:05 AM hudson.remoting.UserRequest perform
             WARNING: LinkageError while performing UserRequest:UserRPCRequest:hudson.Launcher$RemoteProcess.join[](26)
             java.lang.NoClassDefFoundError: hudson/util/ProcessTree$OSProcess$CheckVetoes
             at hudson.util.ProcessTree$OSProcess.getVeto(ProcessTree.java:293)
             at hudson.util.ProcessTree$WindowsOSProcess.killRecursively(ProcessTree.java:539)
             at hudson.util.ProcessTree.killAll(ProcessTree.java:182)
             at hudson.Proc$LocalProc.destroy(Proc.java:387)
             at hudson.Proc$LocalProc.join(Proc.java:360)
             at hudson.Launcher$RemoteLaunchCallable$1.join(Launcher.java:1321)
             at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
             at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
             at java.lang.reflect.Method.invoke(Unknown Source)
             at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929)
             at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)
             at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)
             at hudson.remoting.UserRequest.perform(UserRequest.java:211)
             at hudson.remoting.UserRequest.perform(UserRequest.java:54)
             at hudson.remoting.Request$2.run(Request.java:369)
             at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
             at java.util.concurrent.FutureTask.run(Unknown Source)
             at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
             at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
             at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:97)
             at java.lang.Thread.run(Unknown Source)
             Caused by: java.lang.ClassNotFoundException: hudson.util.ProcessTree$OSProcess$CheckVetoes
             at java.net.URLClassLoader.findClass(Unknown Source)
             at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:171)
             at java.lang.ClassLoader.loadClass(Unknown Source)
             at java.lang.ClassLoader.loadClass(Unknown Source)
             ... 21 more
            
            kwi 01, 2020 12:25:05 AM hudson.remoting.Request$2 run
             INFO: Failed to send back a reply to the request hudson.remoting.Request$2@2808c082: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@1aa3bc97:JNLP4-connect connection to jenkins.<mycompany.domain>/<master IPaddress>:6666": channel is already closed
             kwi 01, 2020 12:25:15 AM hudson.remoting.jnlp.Main$CuiListener status
             INFO: Performing onReconnect operation.
             kwi 01, 2020 12:25:15 AM hudson.remoting.jnlp.Main$CuiListener status
             INFO: onReconnect operation completed.
             kwi 01, 2020 12:25:15 AM hudson.remoting.jnlp.Main$CuiListener status
             INFO: Locating server among [http://jenkins.<mycompany.domain>:8080/]
             kwi 01, 2020 12:25:19 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
             INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
             kwi 01, 2020 12:25:19 AM hudson.remoting.jnlp.Main$CuiListener status
             INFO: Agent discovery successful
             Agent address: jenkins.<mycompany.domain>
             Agent port: 6666
             Identity: d4:13:d5:<cut>:31:96:c6
             kwi 01, 2020 12:25:19 AM hudson.remoting.jnlp.Main$CuiListener status
             INFO: Handshaking
             kwi 01, 2020 12:25:19 AM hudson.remoting.jnlp.Main$CuiListener status
             INFO: Connecting to jenkins.<mycompany.domain>:6666
             kwi 01, 2020 12:25:19 AM hudson.remoting.jnlp.Main$CuiListener status
             INFO: Trying protocol: JNLP4-connect
             kwi 01, 2020 12:25:19 AM hudson.remoting.jnlp.Main$CuiListener status
             INFO: Remote identity confirmed: d4:13:d5:<cut>:31:96:c6
             kwi 01, 2020 12:25:19 AM hudson.remoting.jnlp.Main$CuiListener status
             INFO: Connected
             ...
             kwi 01, 2020 8:03:28 AM org.jvnet.winp.Native loadByUrl
             WARNING: Failed to load DLL from static location
             java.lang.UnsatisfiedLinkError: Native Library J:\Jenkins\remoting\jarCache\44\winp.x64.BFFE30B3B50581290B1866EF8D48C609.dll already loaded in another classloader
             at java.lang.ClassLoader.loadLibrary0(Unknown Source)
             at java.lang.ClassLoader.loadLibrary(Unknown Source)
             at java.lang.Runtime.load0(Unknown Source)
             at java.lang.System.load(Unknown Source)
             at org.jvnet.winp.Native.loadDll(Native.java:242)
             at org.jvnet.winp.Native.loadByUrl(Native.java:162)
             at org.jvnet.winp.Native.load(Native.java:124)
             at org.jvnet.winp.Native.<clinit>(Native.java:93)
             at org.jvnet.winp.WinProcess.enableDebugPrivilege(WinProcess.java:245)
             at hudson.util.ProcessTree$Windows.<clinit>(ProcessTree.java:701)
             at hudson.util.ProcessTree.get(ProcessTree.java:447)
             at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1100)
             at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1091)
             at hudson.remoting.UserRequest.perform(UserRequest.java:211)
             at hudson.remoting.UserRequest.perform(UserRequest.java:54)
             at hudson.remoting.Request$2.run(Request.java:369)
             at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
             at java.util.concurrent.FutureTask.run(Unknown Source)
             at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
             at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
             at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:97)
             at java.lang.Thread.run(Unknown Source)
            
            kwi 01, 2020 9:38:14 AM hudson.remoting.jnlp.Main$CuiListener status
             INFO: Terminated
             kwi 01, 2020 9:38:14 AM hudson.Launcher$RemoteLaunchCallable$1 join
             INFO: Failed to synchronize IO streams on the channel hudson.remoting.Channel@b76eafc:JNLP4-connect connection to jenkins.<mycompany.domain>/<master IPaddress>:6666
             hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@b76eafc:JNLP4-connect connection to jenkins.<mycompany.domain>/<master IPaddress>:6666": Remote call on JNLP4-connect connection to jenkins.<mycompany.domain>/<master IPaddress>:6666 failed. The channel is closing down or has closed down
             at hudson.remoting.Channel.call(Channel.java:948)
             at hudson.remoting.Channel.syncIO(Channel.java:1683)
             at hudson.Launcher$RemoteLaunchCallable$1.join(Launcher.java:1328)
             at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
             at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
             at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
             at java.lang.reflect.Method.invoke(Unknown Source)
             at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929)
             at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)
             at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)
             at hudson.remoting.UserRequest.perform(UserRequest.java:211)
             at hudson.remoting.UserRequest.perform(UserRequest.java:54)
             at hudson.remoting.Request$2.run(Request.java:369)
             at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
             at java.util.concurrent.FutureTask.run(Unknown Source)
             at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
             at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
             at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:97)
             at java.lang.Thread.run(Unknown Source)
             Caused by: java.nio.channels.ClosedChannelException
             at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
             at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1800(BIONetworkLayer.java:48)
             at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:264)
             ... 4 more
            
            kwi 01, 2020 9:38:14 AM hudson.remoting.UserRequest perform
             WARNING: LinkageError while performing UserRequest:UserRPCRequest:hudson.Launcher$RemoteProcess.join[](18)
             java.lang.NoClassDefFoundError: hudson/util/ProcessTree$OSProcess$CheckVetoes
             at hudson.util.ProcessTree$OSProcess.getVeto(ProcessTree.java:293)
             at hudson.util.ProcessTree$WindowsOSProcess.killRecursively(ProcessTree.java:539)
             at hudson.util.ProcessTree.killAll(ProcessTree.java:182)
             at hudson.Proc$LocalProc.destroy(Proc.java:387)
             at hudson.Proc$LocalProc.join(Proc.java:360)
             at hudson.Launcher$RemoteLaunchCallable$1.join(Launcher.java:1321)
             at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
             at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
             at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
             at java.lang.reflect.Method.invoke(Unknown Source)
             at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929)
             at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)
             at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)
             at hudson.remoting.UserRequest.perform(UserRequest.java:211)
             at hudson.remoting.UserRequest.perform(UserRequest.java:54)
             at hudson.remoting.Request$2.run(Request.java:369)
             at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
             at java.util.concurrent.FutureTask.run(Unknown Source)
             at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
             at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
             at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:97)
             at java.lang.Thread.run(Unknown Source)
             Caused by: java.lang.ClassNotFoundException: hudson.util.ProcessTree$OSProcess$CheckVetoes
             at java.net.URLClassLoader.findClass(Unknown Source)
             at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:171)
             at java.lang.ClassLoader.loadClass(Unknown Source)
             at java.lang.ClassLoader.loadClass(Unknown Source)
             ... 22 more
            
            kwi 01, 2020 9:38:14 AM hudson.remoting.Request$2 run
             INFO: Failed to send back a reply to the request hudson.remoting.Request$2@13fbdfb5: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@b76eafc:JNLP4-connect connection to jenkins.<mycompany.domain>/<master IPaddress>:6666": channel is already closed
             kwi 01, 2020 9:38:24 AM hudson.remoting.jnlp.Main$CuiListener status
             INFO: Performing onReconnect operation.
             kwi 01, 2020 9:38:24 AM hudson.remoting.jnlp.Main$CuiListener status
             INFO: onReconnect operation completed.
             kwi 01, 2020 9:38:24 AM hudson.remoting.jnlp.Main$CuiListener status
             INFO: Locating server among [http://jenkins.<mycompany.domain>:8080/]
             kwi 01, 2020 9:38:24 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
             INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
             kwi 01, 2020 9:38:24 AM hudson.remoting.jnlp.Main$CuiListener status
             INFO: Agent discovery successful
             Agent address: jenkins.<mycompany.domain>
             Agent port: 6666
             Identity: d4:13:d5:<cut>:31:96:c6
             kwi 01, 2020 9:38:24 AM hudson.remoting.jnlp.Main$CuiListener status
             INFO: Handshaking
             kwi 01, 2020 9:38:24 AM hudson.remoting.jnlp.Main$CuiListener status
             INFO: Connecting to jenkins.<mycompany.domain>:6666
             kwi 01, 2020 9:38:24 AM hudson.remoting.jnlp.Main$CuiListener status
             INFO: Trying protocol: JNLP4-connect
             kwi 01, 2020 9:38:25 AM hudson.remoting.jnlp.Main$CuiListener status
             INFO: Remote identity confirmed: d4:13:d5:<cut>:31:96:c6
             kwi 01, 2020 9:38:25 AM hudson.remoting.jnlp.Main$CuiListener status
             INFO: Connected
             kwi 01, 2020 9:52:25 AM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
             WARNING: Attempt to (de-)serialize anonymous class hudson.FilePath$2; see: [https://jenkins.io/redirect/serialization-of-anonymous-classes/]
             Failed to load native POSIX impl; falling back on Java impl. Unsupported OS.
             kwi 01, 2020 10:00:41 AM hudson.plugins.tmpcleaner.TmpCleanTask delete
             INFO: Deletion failed: C:\Users\jenkins\AppData\Local\Temp\tmpclean6250731123151792916.tmp
             kwi 01, 2020 10:00:41 AM hudson.plugins.tmpcleaner.TmpCleanTask call
             INFO: Temporary directory cleanup fried 0 B disk space, available 145,52 GB

            First it happened when the agent was running as a Windows Service so without inspecting the logs I thought it must be some incopatibility issues since I auto-upgraded the master to Jenkins ver. 2.222.1 in the meantime (however no such issues on the other Windows Agent). I started the agent by a command-line and it was connected fine for several days doing the builds. Then we have fixed failing builds (our bad) and now the problem of disconnections returned, or just got more urgent since now we build more often and the agent reconnects after a failure. Maybe it's worth to note that our single build takes from 3 hours up to 7 days since we are working on a huge Unreal Engine project.

            I have review related issues (JENKINS-20913 and JENKINS-18578) but this seems to not be the case.

            Not sure how to reopen this issue but I think it should be.

            Show
            remik P4 added a comment - - edited Recently we have added second Windows agent to our Windows master and immediately started to observe this issue. No such problems on the first Windows agent that is running here for months. Both systems are Microsoft Windows [Version 10.0.18363.720] . This is sanitized log fragment from the agent: mar 31, 2020 10:00:57 PM org.jvnet.winp.Native loadByUrl WARNING: Failed to load DLL from static location java.lang.UnsatisfiedLinkError: Native Library J:\Jenkins\remoting\jarCache\44\winp.x64.BFFE30B3B50581290B1866EF8D48C609.dll already loaded in another classloader at java.lang. ClassLoader .loadLibrary0(Unknown Source) at java.lang. ClassLoader .loadLibrary(Unknown Source) at java.lang. Runtime .load0(Unknown Source) at java.lang. System .load(Unknown Source) at org.jvnet.winp.Native.loadDll(Native.java:242) at org.jvnet.winp.Native.loadByUrl(Native.java:162) at org.jvnet.winp.Native.load(Native.java:124) at org.jvnet.winp.Native.<clinit>(Native.java:93) at org.jvnet.winp.WinProcess.enableDebugPrivilege(WinProcess.java:245) at hudson.util.ProcessTree$Windows.<clinit>(ProcessTree.java:701) at hudson.util.ProcessTree.get(ProcessTree.java:447) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1100) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1091) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:97) at java.lang. Thread .run(Unknown Source) kwi 01, 2020 12:25:05 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Terminated kwi 01, 2020 12:25:05 AM hudson.Launcher$RemoteLaunchCallable$1 join INFO: Failed to synchronize IO streams on the channel hudson.remoting.Channel@1aa3bc97:JNLP4-connect connection to jenkins.<mycompany.domain>/<master IPaddress>:6666 hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@1aa3bc97:JNLP4-connect connection to jenkins.<mycompany.domain>/<master IPaddress>:6666" : Remote call on JNLP4-connect connection to jenkins.<mycompany.domain>/<master IPaddress>:6666 failed. The channel is closing down or has closed down at hudson.remoting.Channel.call(Channel.java:948) at hudson.remoting.Channel.syncIO(Channel.java:1683) at hudson.Launcher$RemoteLaunchCallable$1.join(Launcher.java:1328) at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:97) at java.lang. Thread .run(Unknown Source) Caused by: java.nio.channels.ClosedChannelException at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154) at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1800(BIONetworkLayer.java:48) at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:264) ... 4 more kwi 01, 2020 12:25:05 AM hudson.remoting.UserRequest perform WARNING: LinkageError while performing UserRequest:UserRPCRequest:hudson.Launcher$RemoteProcess.join[](26) java.lang.NoClassDefFoundError: hudson/util/ProcessTree$OSProcess$CheckVetoes at hudson.util.ProcessTree$OSProcess.getVeto(ProcessTree.java:293) at hudson.util.ProcessTree$WindowsOSProcess.killRecursively(ProcessTree.java:539) at hudson.util.ProcessTree.killAll(ProcessTree.java:182) at hudson.Proc$LocalProc.destroy(Proc.java:387) at hudson.Proc$LocalProc.join(Proc.java:360) at hudson.Launcher$RemoteLaunchCallable$1.join(Launcher.java:1321) at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:97) at java.lang. Thread .run(Unknown Source) Caused by: java.lang.ClassNotFoundException: hudson.util.ProcessTree$OSProcess$CheckVetoes at java.net.URLClassLoader.findClass(Unknown Source) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:171) at java.lang. ClassLoader .loadClass(Unknown Source) at java.lang. ClassLoader .loadClass(Unknown Source) ... 21 more kwi 01, 2020 12:25:05 AM hudson.remoting.Request$2 run INFO: Failed to send back a reply to the request hudson.remoting.Request$2@2808c082: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@1aa3bc97:JNLP4-connect connection to jenkins.<mycompany.domain>/<master IPaddress>:6666" : channel is already closed kwi 01, 2020 12:25:15 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Performing onReconnect operation. kwi 01, 2020 12:25:15 AM hudson.remoting.jnlp.Main$CuiListener status INFO: onReconnect operation completed. kwi 01, 2020 12:25:15 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among [http: //jenkins.<mycompany.domain>:8080/] kwi 01, 2020 12:25:19 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping] kwi 01, 2020 12:25:19 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Agent discovery successful Agent address: jenkins.<mycompany.domain> Agent port: 6666 Identity: d4:13:d5:<cut>:31:96:c6 kwi 01, 2020 12:25:19 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Handshaking kwi 01, 2020 12:25:19 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to jenkins.<mycompany.domain>:6666 kwi 01, 2020 12:25:19 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Trying protocol: JNLP4-connect kwi 01, 2020 12:25:19 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Remote identity confirmed: d4:13:d5:<cut>:31:96:c6 kwi 01, 2020 12:25:19 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Connected ... kwi 01, 2020 8:03:28 AM org.jvnet.winp.Native loadByUrl WARNING: Failed to load DLL from static location java.lang.UnsatisfiedLinkError: Native Library J:\Jenkins\remoting\jarCache\44\winp.x64.BFFE30B3B50581290B1866EF8D48C609.dll already loaded in another classloader at java.lang. ClassLoader .loadLibrary0(Unknown Source) at java.lang. ClassLoader .loadLibrary(Unknown Source) at java.lang. Runtime .load0(Unknown Source) at java.lang. System .load(Unknown Source) at org.jvnet.winp.Native.loadDll(Native.java:242) at org.jvnet.winp.Native.loadByUrl(Native.java:162) at org.jvnet.winp.Native.load(Native.java:124) at org.jvnet.winp.Native.<clinit>(Native.java:93) at org.jvnet.winp.WinProcess.enableDebugPrivilege(WinProcess.java:245) at hudson.util.ProcessTree$Windows.<clinit>(ProcessTree.java:701) at hudson.util.ProcessTree.get(ProcessTree.java:447) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1100) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1091) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:97) at java.lang. Thread .run(Unknown Source) kwi 01, 2020 9:38:14 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Terminated kwi 01, 2020 9:38:14 AM hudson.Launcher$RemoteLaunchCallable$1 join INFO: Failed to synchronize IO streams on the channel hudson.remoting.Channel@b76eafc:JNLP4-connect connection to jenkins.<mycompany.domain>/<master IPaddress>:6666 hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@b76eafc:JNLP4-connect connection to jenkins.<mycompany.domain>/<master IPaddress>:6666" : Remote call on JNLP4-connect connection to jenkins.<mycompany.domain>/<master IPaddress>:6666 failed. The channel is closing down or has closed down at hudson.remoting.Channel.call(Channel.java:948) at hudson.remoting.Channel.syncIO(Channel.java:1683) at hudson.Launcher$RemoteLaunchCallable$1.join(Launcher.java:1328) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:97) at java.lang. Thread .run(Unknown Source) Caused by: java.nio.channels.ClosedChannelException at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154) at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1800(BIONetworkLayer.java:48) at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:264) ... 4 more kwi 01, 2020 9:38:14 AM hudson.remoting.UserRequest perform WARNING: LinkageError while performing UserRequest:UserRPCRequest:hudson.Launcher$RemoteProcess.join[](18) java.lang.NoClassDefFoundError: hudson/util/ProcessTree$OSProcess$CheckVetoes at hudson.util.ProcessTree$OSProcess.getVeto(ProcessTree.java:293) at hudson.util.ProcessTree$WindowsOSProcess.killRecursively(ProcessTree.java:539) at hudson.util.ProcessTree.killAll(ProcessTree.java:182) at hudson.Proc$LocalProc.destroy(Proc.java:387) at hudson.Proc$LocalProc.join(Proc.java:360) at hudson.Launcher$RemoteLaunchCallable$1.join(Launcher.java:1321) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:929) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:97) at java.lang. Thread .run(Unknown Source) Caused by: java.lang.ClassNotFoundException: hudson.util.ProcessTree$OSProcess$CheckVetoes at java.net.URLClassLoader.findClass(Unknown Source) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:171) at java.lang. ClassLoader .loadClass(Unknown Source) at java.lang. ClassLoader .loadClass(Unknown Source) ... 22 more kwi 01, 2020 9:38:14 AM hudson.remoting.Request$2 run INFO: Failed to send back a reply to the request hudson.remoting.Request$2@13fbdfb5: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@b76eafc:JNLP4-connect connection to jenkins.<mycompany.domain>/<master IPaddress>:6666" : channel is already closed kwi 01, 2020 9:38:24 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Performing onReconnect operation. kwi 01, 2020 9:38:24 AM hudson.remoting.jnlp.Main$CuiListener status INFO: onReconnect operation completed. kwi 01, 2020 9:38:24 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among [http: //jenkins.<mycompany.domain>:8080/] kwi 01, 2020 9:38:24 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping] kwi 01, 2020 9:38:24 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Agent discovery successful Agent address: jenkins.<mycompany.domain> Agent port: 6666 Identity: d4:13:d5:<cut>:31:96:c6 kwi 01, 2020 9:38:24 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Handshaking kwi 01, 2020 9:38:24 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to jenkins.<mycompany.domain>:6666 kwi 01, 2020 9:38:24 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Trying protocol: JNLP4-connect kwi 01, 2020 9:38:25 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Remote identity confirmed: d4:13:d5:<cut>:31:96:c6 kwi 01, 2020 9:38:25 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Connected kwi 01, 2020 9:52:25 AM org.jenkinsci.remoting.util.AnonymousClassWarnings warn WARNING: Attempt to (de-)serialize anonymous class hudson.FilePath$2; see: [https: //jenkins.io/redirect/serialization-of-anonymous-classes/] Failed to load native POSIX impl; falling back on Java impl. Unsupported OS. kwi 01, 2020 10:00:41 AM hudson.plugins.tmpcleaner.TmpCleanTask delete INFO: Deletion failed: C:\Users\jenkins\AppData\Local\Temp\tmpclean6250731123151792916.tmp kwi 01, 2020 10:00:41 AM hudson.plugins.tmpcleaner.TmpCleanTask call INFO: Temporary directory cleanup fried 0 B disk space, available 145,52 GB First it happened when the agent was running as a Windows Service so without inspecting the logs I thought it must be some incopatibility issues since I auto-upgraded the master to Jenkins ver. 2.222.1 in the meantime (however no such issues on the other Windows Agent). I started the agent by a command-line and it was connected fine for several days doing the builds. Then we have fixed failing builds (our bad) and now the problem of disconnections returned, or just got more urgent since now we build more often and the agent reconnects after a failure. Maybe it's worth to note that our single build takes from 3 hours up to 7 days since we are working on a huge Unreal Engine project. I have review related issues ( JENKINS-20913 and JENKINS-18578 ) but this seems to not be the case. Not sure how to reopen this issue but I think it should be.

              People

              • Assignee:
                oleg_nenashev Oleg Nenashev
                Reporter:
                andywalker Andy Walker
              • Votes:
                4 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: