Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46123

Restart fails on Windows master with slaves connected. "SEVERE: Restarting VM as requested by username" (Windows service not running as local system account)

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Minor
    • Resolution: Fixed
    • Component/s: core
    • Labels:
    • Environment:
    • Similar Issues:

      Description

      I see this issue at every single restart, safeRestart or not. It seems to be related to the connected slaves:

      aug. 10, 2017 1:00:08 PM jenkins.model.Jenkins$26 run
      INFO: Restart in 10 seconds
      aug. 10, 2017 1:00:18 PM jenkins.model.Jenkins$26 run
      SEVERE: Restarting VM as requested by nkj
      aug. 10, 2017 1:00:18 PM jenkins.model.Jenkins cleanUp
      INFO: Stopping Jenkins
      aug. 10, 2017 1:00:18 PM jenkins.model.Jenkins$21 onAttained
      INFO: Started termination
      aug. 10, 2017 1:00:18 PM jenkins.model.Jenkins$21 onAttained
      INFO: Completed termination
      aug. 10, 2017 1:00:18 PM jenkins.model.Jenkins _cleanUpDisconnectComputers
      INFO: Starting node disconnection
      aug. 10, 2017 1:00:18 PM jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
      WARNING: safe-restart thread for EAC3 Win10 (dtdkcphpw200044) terminated
      java.nio.channels.ClosedChannelException

      Please NOTE: The Jenkins service runs as an Active Directory user, not a local system account (using "Log on as").

      The result is that Jenkins stops, but never restarts:

      INFO: Waiting for node disconnection completion
      aug. 10, 2017 1:00:20 PM jenkins.model.Jenkins cleanUp
      INFO: Jenkins stopped

      If I look at the service state at this point (after INFO: Jenkins stopped) using "Services", Jenkins appear to be running. A "Restart the service" brings it back up without giving any passwords (password for the user is stored).

        Attachments

          Activity

          Hide
          nkjensen Niels Kristian Jensen added a comment - - edited

          I've just updated to 2.77 and I hoped that the change about JENKINS_HOME had fixed it (https://issues.jenkins-ci.org/browse/JENKINS-13153) - but no. /SafeRestart still hangs.

          Show
          nkjensen Niels Kristian Jensen added a comment - - edited I've just updated to 2.77 and I hoped that the change about JENKINS_HOME had fixed it ( https://issues.jenkins-ci.org/browse/JENKINS-13153 ) - but no. /SafeRestart still hangs.
          Hide
          danielbeck Daniel Beck added a comment - - edited

          A comment to another similar issue indicated that "Windows Management Instrumentation" (WMI) needs to become a "dependency" (? ) of the Jenkins service. Not a Windows user, but perhaps this means something to you Niels Kristian Jensen

          https://issues.jenkins-ci.org/browse/JENKINS-2566?focusedCommentId=147780&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-147780

           

          Show
          danielbeck Daniel Beck added a comment - - edited A comment to another similar issue indicated that "Windows Management Instrumentation" (WMI) needs to become a "dependency" (? ) of the Jenkins service. Not a Windows user, but perhaps this means something to you Niels Kristian Jensen https://issues.jenkins-ci.org/browse/JENKINS-2566?focusedCommentId=147780&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-147780  
          Hide
          nkjensen Niels Kristian Jensen added a comment -

          The problem was resolved "outside" Jenkins. The user running the Jenkins service (Log-in As) had more rights:

          I created a job, which simply reboots the server weekly. The user running the Jenkins service was not allowed to "shutdown" the server. IT dept granted the "shutdown" right to the user running the Jenkins service - and somehow this also "fixed" the problem of restarting the Jenkins service after a SW or Plugin upgrade.

          I'll set this issue to "Fixed"

          Show
          nkjensen Niels Kristian Jensen added a comment - The problem was resolved "outside" Jenkins. The user running the Jenkins service (Log-in As) had more rights: I created a job, which simply reboots the server weekly. The user running the Jenkins service was not allowed to "shutdown" the server. IT dept granted the "shutdown" right to the user running the Jenkins service - and somehow this also "fixed" the problem of restarting the Jenkins service after a SW or Plugin upgrade. I'll set this issue to "Fixed"
          Hide
          nkjensen Niels Kristian Jensen added a comment -

          The issue is a user-rights issue. The user running the server needs to be granted the right to "shutdown" system.

          Show
          nkjensen Niels Kristian Jensen added a comment - The issue is a user-rights issue. The user running the server needs to be granted the right to "shutdown" system.
          Hide
          chethan_m Chethan Mahabaleswara added a comment -

          Hi,

          I have a similar issue. Can someone guide me about possible fix ?

           

          OS - windows 2012 R2 (both Master and Slave)

          Java - 1.8.0_211 (both Mater and Salve)

          Jenkins Version - 2.176.2Slave

          Launch Method - Java Web start, JNLP(Launch by connecting it to the Master)

           

          Issue - I have a project in Jenkins which fails mainly because the slave terminates after 'Starting ServiceWrapper in the CLI mode' during execution.Finds a child process conhost.exe and failed due to Killing as fallback.

          This is happening only for a project/ pipeline, where I have python script which pings a internal URL (https://URL/ping), invokes the network emulator tool installed, adjust the network latency and also write to a file/ push to ELK. I have a internet proxy set up using .pac file. 

          It completes all the steps in project, at the end the slave terminates, marking the build as failure 

          Note - We also have another old version of jenkins 2.76 installed where the same project/ pipeline works with out any issue. Now we want to upgrade to latest jenkins but facing issue with slave getting terminated. I now have to close the network emulator tool every time and restart the jenkins service to bring it back online. 

          Error as obsreved in Jenkins-slave.wrapper.log:-

          2019-07-24 03:51:59,751 INFO  - Stopping jenkinsslave-e__jenkins
          2019-07-24 03:51:59,751 DEBUG - ProcessKill 6872
          2019-07-24 03:51:59,860 INFO  - Found child process: 6408 Name: conhost.exe
          2019-07-24 03:51:59,892 INFO  - Stopping process 6408
          2019-07-24 03:51:59,907 INFO  - Send SIGINT 6408
          2019-07-24 03:51:59,907 WARN  - SIGINT to 6408 failed - Killing as fallback
          2019-07-24 03:51:59,907 INFO  - Stopping process 6872
          2019-07-24 03:51:59,907 INFO  - Send SIGINT 6872
          2019-07-24 03:51:59,907 WARN  - SIGINT to 6872 failed - Killing as fallback
          2019-07-24 03:51:59,907 INFO  - Finished jenkinsslave-e__jenkins
          2019-07-24 03:51:59,907 DEBUG - Completed. Exit code is 0

           

          Error as obsreved in jenkins-slave.error.log:-
          Jul 24, 2019 3:57:23 AM hudson.remoting.jnlp.Main$CuiListener statusJul
          24, 2019 3:57:23 AM hudson.remoting.jnlp.Main$CuiListener statusINFO: Terminated
          Jul 24, 2019 3:57:23 AM hudson.util.ProcessTree getWARNING: Error while determining if vetoers existhudson.remoting.RequestAbortedException: java.nio.channels.ClosedChannelException
          at hudson.remoting.Request.abort(Request.java:340) at hudson.remoting.Channel.terminate(Channel.java:1040)
          at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:209)
          at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222)
          at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816)
          at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
          at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:172)
          at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816)
          at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
          at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1800(BIONetworkLayer.java:48)
          at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:264)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
          at java.lang.Thread.run(Unknown Source) Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection to jenkinsmater/IP address:33131
          at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
          at hudson.remoting.Request.call(Request.java:202) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286)
          at com.sun.proxy.$Proxy6.fetch3(Unknown Source)
          at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:209)
          at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source)
          at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(Unknown Source)
          at java.lang.ClassLoader.defineClass(Unknown Source) at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:383) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:307)
          at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source)
          at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(Unknown Source)
          at java.lang.ClassLoader.defineClass(Unknown Source) at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:383) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:307)
          at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source)
          at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(Unknown Source)
          at java.lang.ClassLoader.defineClass(Unknown Source) at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:383) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:307)
          at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source)
          at hudson.util.ProcessTree.get(ProcessTree.java:415)
          at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1103) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1094)
          at hudson.remoting.UserRequest.perform(UserRequest.java:212)
          at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          at java.util.concurrent.FutureTask.run(Unknown Source) ... 4 moreCaused by: java.nio.channels.ClosedChannelException ... 7 more
          Jul 24, 2019 3:57:23 AM hudson.util.ProcessTree getWARNING: Failed to load winp. Reverting to the defaultjava.lang.NoClassDefFoundError: hudson/util/ProcessTree$WindowsOSProcessException
          at hudson.util.ProcessTree.get(ProcessTree.java:430)
          at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1103) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1094)
          at hudson.remoting.UserRequest.perform(UserRequest.java:212)
          at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
          at java.lang.Thread.run(Unknown Source)Caused by: java.lang.ClassNotFoundException: hudson.util.ProcessTree$WindowsOSProcessException at java.net.URLClassLoader.findClass(Unknown Source)
          at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:171) at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) ... 12 more
          Jul 24, 2019 3:57:23 AM hudson.remoting.Request$2 runINFO: Failed to send back a reply to the request hudson.remoting.Request$2@626d990c: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@4d834d1e:JNLP4-connect connection to jenkinsmaster/IP Address:33131": channel is already closed

          Jul 24, 2019 3:57:33 AM hudson.remoting.jnlp.Main$CuiListener statusINFO: Performing onReconnect operation.

           

          Can someone please help with this issue ?

           

          Thanks,

          Chethan

          Show
          chethan_m Chethan Mahabaleswara added a comment - Hi, I have a similar issue. Can someone guide me about possible fix ?   OS - windows 2012 R2 (both Master and Slave) Java - 1.8.0_211 (both Mater and Salve) Jenkins Version - 2.176.2Slave Launch Method - Java Web start, JNLP(Launch by connecting it to the Master)   Issue - I have a project in Jenkins which fails mainly because the slave terminates after 'Starting ServiceWrapper in the CLI mode' during execution.Finds a child process conhost.exe and failed due to Killing as fallback. This is happening only for a project/ pipeline, where I have python script which pings a internal URL ( https://URL/ping ), invokes the network emulator tool installed, adjust the network latency and also write to a file/ push to ELK. I have a internet proxy set up using .pac file.  It completes all the steps in project, at the end the slave terminates, marking the build as failure  Note - We also have another old version of jenkins 2.76 installed where the same project/ pipeline works with out any issue. Now we want to upgrade to latest jenkins but facing issue with slave getting terminated. I now have to close the network emulator tool every time and restart the jenkins service to bring it back online.  Error as obsreved in Jenkins-slave.wrapper.log:- 2019-07-24 03:51:59,751 INFO  - Stopping jenkinsslave-e__jenkins 2019-07-24 03:51:59,751 DEBUG - ProcessKill 6872 2019-07-24 03:51:59,860 INFO  - Found child process: 6408 Name: conhost.exe 2019-07-24 03:51:59,892 INFO  - Stopping process 6408 2019-07-24 03:51:59,907 INFO  - Send SIGINT 6408 2019-07-24 03:51:59,907 WARN  - SIGINT to 6408 failed - Killing as fallback 2019-07-24 03:51:59,907 INFO  - Stopping process 6872 2019-07-24 03:51:59,907 INFO  - Send SIGINT 6872 2019-07-24 03:51:59,907 WARN  - SIGINT to 6872 failed - Killing as fallback 2019-07-24 03:51:59,907 INFO  - Finished jenkinsslave-e__jenkins 2019-07-24 03:51:59,907 DEBUG - Completed. Exit code is 0   Error as obsreved in jenkins-slave.error.log:- Jul 24, 2019 3:57:23 AM hudson.remoting.jnlp.Main$CuiListener statusJul 24, 2019 3:57:23 AM hudson.remoting.jnlp.Main$CuiListener statusINFO: Terminated Jul 24, 2019 3:57:23 AM hudson.util.ProcessTree getWARNING: Error while determining if vetoers existhudson.remoting.RequestAbortedException: java.nio.channels.ClosedChannelException at hudson.remoting.Request.abort(Request.java:340) at hudson.remoting.Channel.terminate(Channel.java:1040) at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:209) at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816) at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:172) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816) at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154) at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1800(BIONetworkLayer.java:48) at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:264) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93) at java.lang.Thread.run(Unknown Source) Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection to jenkinsmater/IP address:33131 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743) at hudson.remoting.Request.call(Request.java:202) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286) at com.sun.proxy.$Proxy6.fetch3(Unknown Source) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:209) at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(Unknown Source) at java.lang.ClassLoader.defineClass(Unknown Source) at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:383) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:307) at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(Unknown Source) at java.lang.ClassLoader.defineClass(Unknown Source) at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:383) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:307) at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(Unknown Source) at java.lang.ClassLoader.defineClass(Unknown Source) at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:383) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:307) at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at hudson.util.ProcessTree.get(ProcessTree.java:415) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1103) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1094) at hudson.remoting.UserRequest.perform(UserRequest.java:212) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(Unknown Source) ... 4 moreCaused by: java.nio.channels.ClosedChannelException ... 7 more Jul 24, 2019 3:57:23 AM hudson.util.ProcessTree getWARNING: Failed to load winp. Reverting to the defaultjava.lang.NoClassDefFoundError: hudson/util/ProcessTree$WindowsOSProcessException at hudson.util.ProcessTree.get(ProcessTree.java:430) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1103) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1094) at hudson.remoting.UserRequest.perform(UserRequest.java:212) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93) at java.lang.Thread.run(Unknown Source)Caused by: java.lang.ClassNotFoundException: hudson.util.ProcessTree$WindowsOSProcessException at java.net.URLClassLoader.findClass(Unknown Source) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:171) at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) ... 12 more Jul 24, 2019 3:57:23 AM hudson.remoting.Request$2 runINFO: Failed to send back a reply to the request hudson.remoting.Request$2@626d990c: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@4d834d1e:JNLP4-connect connection to jenkinsmaster/IP Address:33131": channel is already closed Jul 24, 2019 3:57:33 AM hudson.remoting.jnlp.Main$CuiListener statusINFO: Performing onReconnect operation.   Can someone please help with this issue ?   Thanks, Chethan

            People

            • Assignee:
              Unassigned
              Reporter:
              nkjensen Niels Kristian Jensen
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: