Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-22685

Jenkins cannot restart Windows service

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      When running as a Windows service, Jenkins is unable to restart itself after installing/upgrading plugins or running /restart. The tail of jenkins.err.log reads:

      """
      WARNING: Failed to restart Hudson
      java.io.IOException: Restarting a service
      [Jenkins] $ "C:\Program Files (x86)\Jenkins\jenkins.exe" restart

      at hudson.lifecycle.WindowsServiceLifecycle.restart(WindowsServiceLifecycle.java:135)
      at jenkins.model.Jenkins$23.run(Jenkins.java:3288)
      """

      Additionally, jenkins.wrapper.log shows the following error:

      """
      2014-04-16 16:54:57 - Starting C:\Program Files (x86)\Jenkins\jre\bin\java -Xrs -Xmx256m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -jar "C:\Program Files (x86)\Jenkins\jenkins.war" --httpPort=8080
      2014-04-16 16:54:57 - Started 1796
      2014-04-17 18:14:19 - Stopping jenkins
      2014-04-17 18:14:19 - ProcessKill 1796
      2014-04-17 18:14:19 - Send SIGINT 1796
      2014-04-17 18:14:19 - SIGINT to1796 successful
      2014-04-17 18:14:19 - Send SIGINT 1796
      2014-04-17 18:14:19 - SIGINT to 1796 failed - Killing as fallback
      2014-04-17 18:14:19 - Finished jenkins
      """

      On previous versions of Jenkins, none of the lines with "SIGINT" are present, and restart works as expected.

      Seeing this issue on two different Windows machines. Can be tested with a stock installation of jenkins-1.559.msi. After Jenkins is installed, go to localhost:8080/restart and restart. The Jenkins service will stop but not restart, and the above error can be seen in the err log.

      Went back and tested 1.549, 1.554, 1.557 and 1.558. None of these exhibited this problem. Reinstalled 1.559, and saw the problem immediately.

      Work-around is to run Windows Services manager and manually restart Jenkins service.

        Attachments

          Issue Links

            Activity

            Hide
            sdcgm JCK added a comment -

            Bumped priority from minor to major because, depending on admin rights, the listed work-around is not necessarily possible.

            Show
            sdcgm JCK added a comment - Bumped priority from minor to major because, depending on admin rights, the listed work-around is not necessarily possible.
            Hide
            sdcgm JCK added a comment -

            Added text from jenkins.wrapper.log.

            Show
            sdcgm JCK added a comment - Added text from jenkins.wrapper.log.
            Hide
            sdcgm JCK added a comment - - edited

            Here's what's happening:

            1. Restarting Jenkins (after plugin-update or by calling /restart) spawns a new process to execute "jenkins.exe restart" (i.e., restarting via the Windows Service wrapper). The Parent Process ID of this new process equals the Process ID that is running jenkins.war.

            2. Jenkins.exe's "restart" command is a three-part process: A) stop the jenkins.war PID and all children PID; B) Wait for jenkins.war PID to stop; C) launch jenkins.war in a new process.

            3. Since this execution of "jenkins.exe restart" is itself a child process of jenkins.war, the Jenkins service is killing it as part of 2A. This results in:

            3A. Killing "jenkins.exe restart" (via SIGINT) causes that process to return w/ errorlevel=1. The WindowsServiceLifecycle restart method is checking the errorlevel from running "jenkins.exe restart" and seeing that it is not 0, and therefore throws an IOException. This is the error message seen in jenkins.err.log.

            3B. Since the process running "jenkins.exe restart" gets killed as part of Step 2A, it does not get a change to complete the restart process, so Jenkins remains stopped.

            Solution:

            From looking at the WinSW project, this issue was expected. A "restart!" command (vs. the standard "restart") was added that launches a new process to handle restart. WindowsServiceLifecycle should use the new command.

            Show
            sdcgm JCK added a comment - - edited Here's what's happening: 1. Restarting Jenkins (after plugin-update or by calling /restart) spawns a new process to execute "jenkins.exe restart" (i.e., restarting via the Windows Service wrapper). The Parent Process ID of this new process equals the Process ID that is running jenkins.war. 2. Jenkins.exe's "restart" command is a three-part process: A) stop the jenkins.war PID and all children PID; B) Wait for jenkins.war PID to stop; C) launch jenkins.war in a new process. 3. Since this execution of "jenkins.exe restart" is itself a child process of jenkins.war, the Jenkins service is killing it as part of 2A. This results in: 3A. Killing "jenkins.exe restart" (via SIGINT) causes that process to return w/ errorlevel=1. The WindowsServiceLifecycle restart method is checking the errorlevel from running "jenkins.exe restart" and seeing that it is not 0, and therefore throws an IOException. This is the error message seen in jenkins.err.log. 3B. Since the process running "jenkins.exe restart" gets killed as part of Step 2A, it does not get a change to complete the restart process, so Jenkins remains stopped. Solution: From looking at the WinSW project, this issue was expected. A "restart!" command (vs. the standard "restart") was added that launches a new process to handle restart. WindowsServiceLifecycle should use the new command.
            Hide
            sdcgm JCK added a comment - - edited
            Show
            sdcgm JCK added a comment - - edited Pull request: https://github.com/jenkinsci/jenkins/pull/1238
            Hide
            danielbeck Daniel Beck added a comment -

            Would this PR also resolve JENKINS-22024?

            Show
            danielbeck Daniel Beck added a comment - Would this PR also resolve JENKINS-22024 ?
            Hide
            sdcgm JCK added a comment -

            I suspect not. I'm not seeing a scenario where the "jenkins.exe restart" process being killed by the "Jenkins service stop" command could result in the behavior observed in JENKINS-22024.

            Show
            sdcgm JCK added a comment - I suspect not. I'm not seeing a scenario where the "jenkins.exe restart" process being killed by the "Jenkins service stop" command could result in the behavior observed in JENKINS-22024 .
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Marked the issue as lts-candidate

            Show
            oleg_nenashev Oleg Nenashev added a comment - Marked the issue as lts-candidate
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: gmjck
            Path:
            core/src/main/java/hudson/lifecycle/WindowsServiceLifecycle.java
            http://jenkins-ci.org/commit/jenkins/c6671107635a1d0ba0dba4a63b00bda37c3ef428
            Log:
            [FIXED JENKINS-22685] - Update WindowsLifeCycle.java to call restart! to properly restart the service.

            Using "restart!" instead of "restart" performs the restart in a separate process, so the service wrapper doesn't kill itself.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: gmjck Path: core/src/main/java/hudson/lifecycle/WindowsServiceLifecycle.java http://jenkins-ci.org/commit/jenkins/c6671107635a1d0ba0dba4a63b00bda37c3ef428 Log: [FIXED JENKINS-22685] - Update WindowsLifeCycle.java to call restart! to properly restart the service. Using "restart!" instead of "restart" performs the restart in a separate process, so the service wrapper doesn't kill itself.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Daniel Beck
            Path:
            core/src/main/java/hudson/lifecycle/WindowsServiceLifecycle.java
            http://jenkins-ci.org/commit/jenkins/f434c22f58ca156af314f806a39e58ffa619dc1d
            Log:
            Merge pull request #1238 from gmjck/master

            [FIX JENKINS-22685] Jenkins cannot restart Windows service

            Compare: https://github.com/jenkinsci/jenkins/compare/bef077b98c72...f434c22f58ca

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: core/src/main/java/hudson/lifecycle/WindowsServiceLifecycle.java http://jenkins-ci.org/commit/jenkins/f434c22f58ca156af314f806a39e58ffa619dc1d Log: Merge pull request #1238 from gmjck/master [FIX JENKINS-22685] Jenkins cannot restart Windows service Compare: https://github.com/jenkinsci/jenkins/compare/bef077b98c72...f434c22f58ca
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #3402
            [FIXED JENKINS-22685] - Update WindowsLifeCycle.java to call restart! to properly restart the service. (Revision c6671107635a1d0ba0dba4a63b00bda37c3ef428)

            Result = SUCCESS
            jenk : c6671107635a1d0ba0dba4a63b00bda37c3ef428
            Files :

            • core/src/main/java/hudson/lifecycle/WindowsServiceLifecycle.java
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #3402 [FIXED JENKINS-22685] - Update WindowsLifeCycle.java to call restart! to properly restart the service. (Revision c6671107635a1d0ba0dba4a63b00bda37c3ef428) Result = SUCCESS jenk : c6671107635a1d0ba0dba4a63b00bda37c3ef428 Files : core/src/main/java/hudson/lifecycle/WindowsServiceLifecycle.java
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: gmjck
            Path:
            core/src/main/java/hudson/lifecycle/WindowsServiceLifecycle.java
            http://jenkins-ci.org/commit/jenkins/6ac5d15d596cc4790c3a2aedd4f164fac4c096b3
            Log:
            [FIXED JENKINS-22685] - Update WindowsLifeCycle.java to call restart! to properly restart the service.

            Using "restart!" instead of "restart" performs the restart in a separate process, so the service wrapper doesn't kill itself.

            (cherry picked from commit c6671107635a1d0ba0dba4a63b00bda37c3ef428)

            Compare: https://github.com/jenkinsci/jenkins/compare/7f2995405c32...6ac5d15d596c

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: gmjck Path: core/src/main/java/hudson/lifecycle/WindowsServiceLifecycle.java http://jenkins-ci.org/commit/jenkins/6ac5d15d596cc4790c3a2aedd4f164fac4c096b3 Log: [FIXED JENKINS-22685] - Update WindowsLifeCycle.java to call restart! to properly restart the service. Using "restart!" instead of "restart" performs the restart in a separate process, so the service wrapper doesn't kill itself. (cherry picked from commit c6671107635a1d0ba0dba4a63b00bda37c3ef428) Compare: https://github.com/jenkinsci/jenkins/compare/7f2995405c32...6ac5d15d596c
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            BTW, the similar issue fix in WinSW has not been released:
            https://github.com/kohsuke/winsw/commit/d0d4266af2a6155050a44f208722eb2b95cba2d0

            Show
            oleg_nenashev Oleg Nenashev added a comment - BTW, the similar issue fix in WinSW has not been released: https://github.com/kohsuke/winsw/commit/d0d4266af2a6155050a44f208722eb2b95cba2d0
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: gmjck
            Path:
            core/src/main/java/hudson/lifecycle/WindowsServiceLifecycle.java
            http://jenkins-ci.org/commit/jenkins/0d2b82b3b33f920e96c9ff854c776d78102b6399
            Log:
            [FIXED JENKINS-22685] - Update WindowsLifeCycle.java to call restart! to properly restart the service.

            Using "restart!" instead of "restart" performs the restart in a separate process, so the service wrapper doesn't kill itself.

            (cherry picked from commit c6671107635a1d0ba0dba4a63b00bda37c3ef428)

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: gmjck Path: core/src/main/java/hudson/lifecycle/WindowsServiceLifecycle.java http://jenkins-ci.org/commit/jenkins/0d2b82b3b33f920e96c9ff854c776d78102b6399 Log: [FIXED JENKINS-22685] - Update WindowsLifeCycle.java to call restart! to properly restart the service. Using "restart!" instead of "restart" performs the restart in a separate process, so the service wrapper doesn't kill itself. (cherry picked from commit c6671107635a1d0ba0dba4a63b00bda37c3ef428)
            Hide
            sschuberth Sebastian Schuberth added a comment -

            I still see Jenkins 1.554.3 LTS not coming up again after a restart. Is this because of the pending fix in WinSW mentioned by Oleg?

            Show
            sschuberth Sebastian Schuberth added a comment - I still see Jenkins 1.554.3 LTS not coming up again after a restart. Is this because of the pending fix in WinSW mentioned by Oleg?
            Hide
            peejay Paul Adams added a comment - - edited

            I have a similar problem with the 1.554.3 LTS release using the Windows Service, whereby Jenkins does not restart. It just sits there with the message "Please wait while Jenkins is restarting...". I have to manually restart using the Windows Services Manager.

            Note that I am using Windows 2008 R2 Server O/S.

            Show
            peejay Paul Adams added a comment - - edited I have a similar problem with the 1.554.3 LTS release using the Windows Service, whereby Jenkins does not restart. It just sits there with the message "Please wait while Jenkins is restarting...". I have to manually restart using the Windows Services Manager. Note that I am using Windows 2008 R2 Server O/S.
            Hide
            johsnon Johnson Birdman added a comment -

            I still have the same service restarting problem in 1.565.1 (Java Version 7 Update 67), so I can't consider it fixed. As for now I'm staying on 1.554.2 without this problem.

            Show
            johsnon Johnson Birdman added a comment - I still have the same service restarting problem in 1.565.1 (Java Version 7 Update 67), so I can't consider it fixed. As for now I'm staying on 1.554.2 without this problem.
            Hide
            mglauche Michael Glauche added a comment -

            Having the same problem with 1.565.1, running as windows service. Also stuck at "Please wait while Jenkins is restarting ..."

            Show
            mglauche Michael Glauche added a comment - Having the same problem with 1.565.1, running as windows service. Also stuck at "Please wait while Jenkins is restarting ..."
            Hide
            danielbeck Daniel Beck added a comment -

            Are you running Jenkins as a non-default user? Maybe one who's not allowed to restart services? Try to give that user permission to use WMI:

            http://technet.microsoft.com/en-us/library/cc787533%28v=WS.10%29.aspx

            I'm asking, because in the default setup, it works for me:

            Tried with 1.576 on Windows 8.1 Pro installed with the installer. Restart works without issues.

            2014-08-21 17:41:00 - Starting C:\Program Files (x86)\Jenkins\jre\bin\java -Xrs -Xmx256m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -jar "C:\Program Files (x86)\Jenkins\jenkins.war" --httpPort=8080
            2014-08-21 17:41:00 - Started 4088
            2014-08-21 17:42:26 - Stopping jenkins
            2014-08-21 17:42:26 - ProcessKill 4088
            2014-08-21 17:42:26 - Send SIGINT 4088
            2014-08-21 17:42:26 - SIGINT to 4088 failed - Killing as fallback
            2014-08-21 17:42:26 - Send SIGINT 4088
            2014-08-21 17:42:26 - SIGINT to 4088 failed - Killing as fallback
            2014-08-21 17:42:26 - Finished jenkins
            2014-08-21 17:42:28 - Starting C:\Program Files (x86)\Jenkins\jre\bin\java -Xrs -Xmx256m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -jar "C:\Program Files (x86)\Jenkins\jenkins.war" --httpPort=8080
            2014-08-21 17:42:28 - Started 568

            Web UI was available again as well.

            Then I tried 1.565.1 on the same machine (after resetting the VM to before I installed 1.576).

            Again it worked like a charm.

            Show
            danielbeck Daniel Beck added a comment - Are you running Jenkins as a non-default user? Maybe one who's not allowed to restart services? Try to give that user permission to use WMI: http://technet.microsoft.com/en-us/library/cc787533%28v=WS.10%29.aspx I'm asking, because in the default setup, it works for me: Tried with 1.576 on Windows 8.1 Pro installed with the installer. Restart works without issues. 2014-08-21 17:41:00 - Starting C:\Program Files (x86)\Jenkins\jre\bin\java -Xrs -Xmx256m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -jar "C:\Program Files (x86)\Jenkins\jenkins.war" --httpPort=8080 2014-08-21 17:41:00 - Started 4088 2014-08-21 17:42:26 - Stopping jenkins 2014-08-21 17:42:26 - ProcessKill 4088 2014-08-21 17:42:26 - Send SIGINT 4088 2014-08-21 17:42:26 - SIGINT to 4088 failed - Killing as fallback 2014-08-21 17:42:26 - Send SIGINT 4088 2014-08-21 17:42:26 - SIGINT to 4088 failed - Killing as fallback 2014-08-21 17:42:26 - Finished jenkins 2014-08-21 17:42:28 - Starting C:\Program Files (x86)\Jenkins\jre\bin\java -Xrs -Xmx256m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -jar "C:\Program Files (x86)\Jenkins\jenkins.war" --httpPort=8080 2014-08-21 17:42:28 - Started 568 Web UI was available again as well. Then I tried 1.565.1 on the same machine (after resetting the VM to before I installed 1.576). Again it worked like a charm.
            Hide
            jk Jan Klass added a comment -

            I also have the issue on a 1.565.1 installation.

            Jenkins.exe is running as user SYSTEM. I guess it should be able to restart services then!? (Not sure where to find WMI and its service permissions).

            Show
            jk Jan Klass added a comment - I also have the issue on a 1.565.1 installation. Jenkins.exe is running as user SYSTEM. I guess it should be able to restart services then!? (Not sure where to find WMI and its service permissions).
            Hide
            jk Jan Klass added a comment -

            Is the Jenkins auto-war-update sufficient? Or do I need to download and install Jenkins again (where I guess I would get a new Jenkins.exe)?

            Show
            jk Jan Klass added a comment - Is the Jenkins auto-war-update sufficient? Or do I need to download and install Jenkins again (where I guess I would get a new Jenkins.exe)?
            Hide
            jk Jan Klass added a comment -

            Indeed; replacing just the jenkins.exe with the one inside the 1.565.1 distribution zip makes the restart work again.
            I guess the other reports here likely come from the same misunderstanding. Merely (auto-)upgrading Jenkins (=replacing the war) is not enough.

            Show
            jk Jan Klass added a comment - Indeed; replacing just the jenkins.exe with the one inside the 1.565.1 distribution zip makes the restart work again. I guess the other reports here likely come from the same misunderstanding. Merely (auto-)upgrading Jenkins (=replacing the war) is not enough.
            Hide
            sschuberth Sebastian Schuberth added a comment -

            > Merely (auto-)upgrading Jenkins (=replacing the war) is not enough.

            Thanks for the clarification!

            Show
            sschuberth Sebastian Schuberth added a comment - > Merely (auto-)upgrading Jenkins (=replacing the war) is not enough. Thanks for the clarification!
            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: pom.xml http://jenkins-ci.org/commit/windows-slave-installer-module/95206829480d66058fedf75d781358166374142a Log: Update to winsw-1.18 Changes: https://github.com/kohsuke/winsw/issues?q=milestone%3Awinsw-1.17 and https://github.com/kohsuke/winsw/commit/2bf9e858a94c7caca41812d644543e61ba1c179e Jenkins issues: JENKINS-10547 , JENKINS-22685
            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: war/pom.xml http://jenkins-ci.org/commit/jenkins/22d9db09d56c1caa2a8a3e008e6bd29f5c9ed395 Log: Update windows-slave-installer dependency to 1.5.1 Jenkins issues: JENKINS-10547 , JENKINS-22685 Changes in winsw: https://github.com/kohsuke/winsw/issues?q=milestone%3Awinsw-1.17 kohsuke/winsw@2bf9e85 Changes in the module: https://github.com/jenkinsci/windows-slave-installer-module/commit/a4d2121812fe9fe1b38bbe9d2cb57b9cd5435a0d https://github.com/jenkinsci/windows-slave-installer-module/commit/3fbc8434aba05d5d36f52f345d9d0176e98459be
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            war/pom.xml
            http://jenkins-ci.org/commit/jenkins/ddfe4d207750352f123bfb510db70c0ea0820ebe
            Log:
            Merge pull request #1832 from oleg-nenashev/windows-slave-installer-1.5

            [JENKINS-10547, JENKINS-22685] Update windows-slave-installer dependency to 1.5

            Compare: https://github.com/jenkinsci/jenkins/compare/d86a88ab042c...ddfe4d207750

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: war/pom.xml http://jenkins-ci.org/commit/jenkins/ddfe4d207750352f123bfb510db70c0ea0820ebe Log: Merge pull request #1832 from oleg-nenashev/windows-slave-installer-1.5 [JENKINS-10547, JENKINS-22685] Update windows-slave-installer dependency to 1.5 Compare: https://github.com/jenkinsci/jenkins/compare/d86a88ab042c...ddfe4d207750

              People

              • Assignee:
                sdcgm JCK
                Reporter:
                sdcgm JCK
              • Votes:
                6 Vote for this issue
                Watchers:
                17 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: