Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-20947

Failed to monitor for Free Swap Space

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Component/s: remoting
    • Labels:
    • Environment:
      Jenkins Master : Win 2008 R2 x 64
      Jenkins Slave : Win 2003 x86
      Tomcat web container , connecting to slave using JNLP
    • Similar Issues:

      Description

      Builds running on the slave hang at the initial pre checkout step
      10:42:10 Started by upstream project "Trunk_Master" build number 135
      10:42:10 originally caused by:
      10:42:10 Started by user Build Team
      10:42:10 Starting limited count build: 1
      10:42:10 [EnvInject] - Loading node environment variables.
      10:42:10 Building remotely on SlaveX in workspace C:\Hudson\workspace\Trunk_TS

      Jenkins logs show this error on the slave at the time it is hung.

      WARNING hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor

      Failed to monitor SlaveX for Free Swap Space
      java.util.concurrent.TimeoutException
      at hudson.remoting.Request$1.get(Request.java:275)
      at hudson.remoting.Request$1.get(Request.java:210)
      at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
      at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:97)
      at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:282)

      Let me know if you need more details

      Thanks
      Shobha

        Attachments

          Issue Links

            Activity

            Hide
            stephenconnolly Stephen Connolly added a comment -

            isn't the -D supposed to be before the -jar but in any case, that looks like you have it the correct side of the jar file argument, so I would start looking for networking issues

            Show
            stephenconnolly Stephen Connolly added a comment - isn't the -D supposed to be before the -jar but in any case, that looks like you have it the correct side of the jar file argument, so I would start looking for networking issues
            Hide
            magnayn magnayn added a comment -

            I have just come across this on two types of host.

            First on Illumos(Smartos)/LX zones, it occasionally reports gigantic free memory (probably a bug), after which the buld log stops updating

            {{Failed to monitor Triton-455640a8f44f for Free Swap Space
            java.util.concurrent.ExecutionException: java.io.IOException: Failed to parse: '18446744073709099244' out of 'MemFree: 18446744073709099244 kB'
            at hudson.remoting.Channel$2.adapt(Channel.java:813)
            at hudson.remoting.Channel$2.adapt(Channel.java:808)
            at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
            at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96)
            at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
            Caused by: java.io.IOException: Failed to parse: '18446744073709099244' out of 'MemFree: 18446744073709099244 kB'
            at org.jvnet.hudson.ProcMemInfo.monitor(ProcMemInfo.java:56)
            at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:113)
            at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:103)
            at hudson.remoting.UserRequest.perform(UserRequest.java:120)
            at hudson.remoting.UserRequest.perform(UserRequest.java:48)
            at hudson.remoting.Request$2.run(Request.java:326)
            at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
            at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            at java.lang.Thread.run(Thread.java:745)
            at ......remote call to Triton-455640a8f44f(Native Method)
            at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
            at hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
            at hudson.remoting.Channel$2.adapt(Channel.java:811)
            ... 4 more}}

            I also now get this on Jenkins 2.x on windows slaves:

            Failed to monitor winbuild for Free Swap Space
            java.util.concurrent.ExecutionException: java.lang.UnsatisfiedLinkError: C:\Users\magnayn\AppData\Local\Temp\jna-829177819\jna2657881109746216710.dll: A dynamic link library (DLL) initialization routine failed
            at hudson.remoting.Channel$2.adapt(Channel.java:813)
            at hudson.remoting.Channel$2.adapt(Channel.java:808)
            at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
            at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96)
            at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
            Caused by: java.lang.UnsatisfiedLinkError: C:\Users\magnayn\AppData\Local\Temp\jna-829177819\jna2657881109746216710.dll: A dynamic link library (DLL) initialization routine failed
            at java.lang.ClassLoader$NativeLibrary.load(Native Method)
            at java.lang.ClassLoader.loadLibrary0(Unknown Source)
            at java.lang.ClassLoader.loadLibrary(Unknown Source)
            at java.lang.Runtime.load0(Unknown Source)
            at java.lang.System.load(Unknown Source)
            at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:851)
            at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:826)
            at com.sun.jna.Native.<clinit>(Native.java:140)
            at com.sun.jna.Pointer.<clinit>(Pointer.java:41)
            at com.sun.jna.Structure.<clinit>(Structure.java:2078)
            at org.jvnet.hudson.Windows.monitor(Windows.java:42)
            at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:113)
            at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:103)
            at hudson.remoting.UserRequest.perform(UserRequest.java:120)
            at hudson.remoting.UserRequest.perform(UserRequest.java:48)
            at hudson.remoting.Request$2.run(Request.java:332)
            at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
            at java.util.concurrent.FutureTask.run(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at hudson.remoting.Engine$1$1.run(Engine.java:85)
            at java.lang.Thread.run(Unknown Source)
            at ......remote call to winbuild(Native Method)
            at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
            at hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
            at hudson.remoting.Channel$2.adapt(Channel.java:811)
            ... 4 more

            Failure seems to kill the channel

            Ping failed. Terminating the channel winbuild.
            java.util.concurrent.TimeoutException: Ping started at 1462866895203 hasn't completed by 1462867135208
            at hudson.remoting.PingThread.ping(PingThread.java:126)
            at hudson.remoting.PingThread.run(PingThread.java:85)

            I don't know why the space monitor re-throws the error as an exception - it might be an idea to swallow it and log.

            Show
            magnayn magnayn added a comment - I have just come across this on two types of host. First on Illumos(Smartos)/LX zones, it occasionally reports gigantic free memory (probably a bug), after which the buld log stops updating {{Failed to monitor Triton-455640a8f44f for Free Swap Space java.util.concurrent.ExecutionException: java.io.IOException: Failed to parse: '18446744073709099244' out of 'MemFree: 18446744073709099244 kB' at hudson.remoting.Channel$2.adapt(Channel.java:813) at hudson.remoting.Channel$2.adapt(Channel.java:808) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) Caused by: java.io.IOException: Failed to parse: '18446744073709099244' out of 'MemFree: 18446744073709099244 kB' at org.jvnet.hudson.ProcMemInfo.monitor(ProcMemInfo.java:56) at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:113) at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:103) at hudson.remoting.UserRequest.perform(UserRequest.java:120) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) at ......remote call to Triton-455640a8f44f(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416) at hudson.remoting.UserResponse.retrieve(UserRequest.java:220) at hudson.remoting.Channel$2.adapt(Channel.java:811) ... 4 more}} I also now get this on Jenkins 2.x on windows slaves: Failed to monitor winbuild for Free Swap Space java.util.concurrent.ExecutionException: java.lang.UnsatisfiedLinkError: C:\Users\magnayn\AppData\Local\Temp\jna-829177819\jna2657881109746216710.dll: A dynamic link library (DLL) initialization routine failed at hudson.remoting.Channel$2.adapt(Channel.java:813) at hudson.remoting.Channel$2.adapt(Channel.java:808) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) Caused by: java.lang.UnsatisfiedLinkError: C:\Users\magnayn\AppData\Local\Temp\jna-829177819\jna2657881109746216710.dll: A dynamic link library (DLL) initialization routine failed at java.lang.ClassLoader$NativeLibrary.load(Native Method) at java.lang.ClassLoader.loadLibrary0(Unknown Source) at java.lang.ClassLoader.loadLibrary(Unknown Source) at java.lang.Runtime.load0(Unknown Source) at java.lang.System.load(Unknown Source) at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:851) at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:826) at com.sun.jna.Native.<clinit>(Native.java:140) at com.sun.jna.Pointer.<clinit>(Pointer.java:41) at com.sun.jna.Structure.<clinit>(Structure.java:2078) at org.jvnet.hudson.Windows.monitor(Windows.java:42) at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:113) at hudson.node_monitors.SwapSpaceMonitor$MonitorTask.call(SwapSpaceMonitor.java:103) at hudson.remoting.UserRequest.perform(UserRequest.java:120) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:332) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:85) at java.lang.Thread.run(Unknown Source) at ......remote call to winbuild(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416) at hudson.remoting.UserResponse.retrieve(UserRequest.java:220) at hudson.remoting.Channel$2.adapt(Channel.java:811) ... 4 more Failure seems to kill the channel Ping failed. Terminating the channel winbuild. java.util.concurrent.TimeoutException: Ping started at 1462866895203 hasn't completed by 1462867135208 at hudson.remoting.PingThread.ping(PingThread.java:126) at hudson.remoting.PingThread.run(PingThread.java:85) I don't know why the space monitor re-throws the error as an exception - it might be an idea to swallow it and log.
            Hide
            mtaylor Matt Taylor added a comment -

            Experiencing this same issue with a slave that is used on demand.

            jenkins.err.zip

            Show
            mtaylor Matt Taylor added a comment - Experiencing this same issue with a slave that is used on demand. jenkins.err.zip
            Hide
            xtaylor21x Matt Taylor added a comment -

            Please someone help still experiencing this.

            Show
            xtaylor21x Matt Taylor added a comment - Please someone help still experiencing this.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            JENKINS-48309 may be a culprit. TimeoutException may happen before the timeout time really passes, and the stacktraces from Matt Taylor and the issue reporter point to the affected code.

            Nov 02, 2016 2:51:12 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor
            WARNING: Failed to monitor PRW8-32-A for Free Swap Space
            java.util.concurrent.TimeoutException
            	at hudson.remoting.Request$1.get(Request.java:279)
            	at hudson.remoting.Request$1.get(Request.java:207)
            	at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
            	at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96)
            	at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
            
            Show
            oleg_nenashev Oleg Nenashev added a comment - JENKINS-48309 may be a culprit. TimeoutException may happen before the timeout time really passes, and the stacktraces from Matt Taylor and the issue reporter point to the affected code. Nov 02, 2016 2:51:12 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor WARNING: Failed to monitor PRW8-32-A for Free Swap Space java.util.concurrent.TimeoutException at hudson.remoting.Request$1.get(Request.java:279) at hudson.remoting.Request$1.get(Request.java:207) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)

              People

              • Assignee:
                Unassigned
                Reporter:
                shobhad Shobha Dashottar
              • Votes:
                7 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated: