Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-13153

"Upgrade Automatically" does not seem to work on Windows

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: core
    • Labels:
      None
    • Environment:
      Windows 2003 R3
      Java 1.6.0_26
      Apache httpd 2.2 as frontend (apj13 proxy)
    • Similar Issues:

      Description

      After moving to a Jenkins installation that runs as a service. I noticed it now had the option to download new versions of itself and upgrade. I tried that with 1.452 and was a bit confused that it restarted but still showed 1.452 as version number. But also gives me the option to downgrade to 1.452.

      After a couple more tries I think I can conclude that what happens is that it downloads the new version just fine (as jenkins.war.tmp) and copies the current version fine as well (jenkins.war.bak). But it never actually gets around to replacing the current running version with the new one.

      This appears to be the relevant part of the jenkins.err.log:

      INFO: Starting the installation of jenkins.war on behalf of kpe
      Mar 19, 2012 9:17:19 AM hudson.model.UpdateCenter$UpdateCenterConfiguration download
      INFO: Downloading jenkins.war
      Mar 19, 2012 9:17:19 AM hudson.model.UpdateCenter doSafeRestart
      INFO: Scheduling Jenkins reboot
      Mar 19, 2012 9:17:40 AM hudson.model.UpdateCenter$DownloadJob run
      INFO: Installation successful: jenkins.war
      

      But I am a bit confused on how this would ever actually work on Windows? Since files are generally locked up pretty tight. So replacing or overwriting a file that Java is actively using would be practically impossible. And instead it would require some sort of utility running before starting Jenkins to replace the war file with the newly downloaded one.

        Attachments

          Activity

          kristoffer Kristoffer Peterhänsel created issue -
          Hide
          tang Pei-Tang Huang added a comment -

          I have the same problem, and the most strange thing is that there exists no exception or warning during installation.

          Maybe jenkins.exe is a good place for doing this?

          Show
          tang Pei-Tang Huang added a comment - I have the same problem, and the most strange thing is that there exists no exception or warning during installation. Maybe jenkins.exe is a good place for doing this?
          Hide
          dhs Dirk Heinrichs added a comment -

          Maybe this functionality should generaly be disabled on Windows?

          Show
          dhs Dirk Heinrichs added a comment - Maybe this functionality should generaly be disabled on Windows?
          ircbot Jenkins IRC Bot made changes -
          Field Original Value New Value
          Component/s core [ 15593 ]
          Component/s update-center [ 15629 ]
          Hide
          danielbeck Daniel Beck added a comment -

          Is this still a problem in recent Jenkins versions?

          Show
          danielbeck Daniel Beck added a comment - Is this still a problem in recent Jenkins versions?
          Hide
          kristoffer Kristoffer Peterhänsel added a comment -

          Not a clue. I no longer work at the place where we had that installation. And don't really have anywhere to replicate the setup either.

          Show
          kristoffer Kristoffer Peterhänsel added a comment - Not a clue. I no longer work at the place where we had that installation. And don't really have anywhere to replicate the setup either.
          Hide
          danielbeck Daniel Beck added a comment -

          Thanks for the update Kristoffer!

          Resolving as incomplete as no information about the issue in recent Jenkins versions is known. (there were a few improvements to jenkins.exe since this was filed, so I have reason to believe this may have been fixed)

          If this happens in a recent (no older than six weeks) version of Jenkins, please just file a new issue. Thanks!

          Show
          danielbeck Daniel Beck added a comment - Thanks for the update Kristoffer! Resolving as incomplete as no information about the issue in recent Jenkins versions is known. (there were a few improvements to jenkins.exe since this was filed, so I have reason to believe this may have been fixed) If this happens in a recent (no older than six weeks) version of Jenkins, please just file a new issue. Thanks!
          danielbeck Daniel Beck made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Incomplete [ 4 ]
          Hide
          phancox Peter Hancox added a comment -

          This is still a problem as of version 1.574

          Attempted automatic upgrade from 1.574 to 1.582 and screen stuck on "Jenkins restarting ..." Stop the Jenkins service and copy "jenkins.war.tmp" to "jenkins.war" and restart service. Upgrade is now complete.

          Appears as if upgrade is waiting for the downloaded temporary war to replace the currently active war and this is hanging. My Jenkins is installed in the default "C:\Program Files (x86)\Jenkins\" directory so I would expect that the update process would need elevated privileges, though I suspect it already has them since it can place the new downloaded war file in that directory.

          Hope that helps.

          Show
          phancox Peter Hancox added a comment - This is still a problem as of version 1.574 Attempted automatic upgrade from 1.574 to 1.582 and screen stuck on "Jenkins restarting ..." Stop the Jenkins service and copy "jenkins.war.tmp" to "jenkins.war" and restart service. Upgrade is now complete. Appears as if upgrade is waiting for the downloaded temporary war to replace the currently active war and this is hanging. My Jenkins is installed in the default "C:\Program Files (x86)\Jenkins\" directory so I would expect that the update process would need elevated privileges, though I suspect it already has them since it can place the new downloaded war file in that directory. Hope that helps.
          Hide
          danielbeck Daniel Beck added a comment -

          Peter: Do you remember the Jenkins version you originally installed, and could look up the creation + last modification dates of jenkins.exe? AFAIK there is a (now resolved) bug that prevented that from getting updated on Jenkins update, and one of the fixed issues was a failure to restart Jenkins as well. So this appears to be an outdated jenkins.exe.

          Current jenkins.exe should have a date from early April, assuming it doesn't get changed when packaged into Jenkins.

          Show
          danielbeck Daniel Beck added a comment - Peter: Do you remember the Jenkins version you originally installed, and could look up the creation + last modification dates of jenkins.exe? AFAIK there is a (now resolved) bug that prevented that from getting updated on Jenkins update, and one of the fixed issues was a failure to restart Jenkins as well. So this appears to be an outdated jenkins.exe. Current jenkins.exe should have a date from early April, assuming it doesn't get changed when packaged into Jenkins.
          Hide
          phancox Peter Hancox added a comment -

          My jenkins.exe is dated 24-Mar-2014.

          Should I uninstall/install the latestWindows package?

          Show
          phancox Peter Hancox added a comment - My jenkins.exe is dated 24-Mar-2014. Should I uninstall/install the latestWindows package?
          Hide
          danielbeck Daniel Beck added a comment -

          Not sure how to best do this. Possibly through reinstall. IIRC a user reported he just replaced the jenkins.exe, so you could also try the following:

          If you find a file identical to your jenkins.exe among the exe files here (as sanity check; I'm not 100% sure this is the correct file): http://repo.jenkins-ci.org/releases/com/sun/winsw/winsw/
          ... then just replace it with the exe in the 1.16 folder.

          Show
          danielbeck Daniel Beck added a comment - Not sure how to best do this. Possibly through reinstall. IIRC a user reported he just replaced the jenkins.exe, so you could also try the following: If you find a file identical to your jenkins.exe among the exe files here (as sanity check; I'm not 100% sure this is the correct file): http://repo.jenkins-ci.org/releases/com/sun/winsw/winsw/ ... then just replace it with the exe in the 1.16 folder.
          Hide
          phancox Peter Hancox added a comment -

          Didn't uninstall. Just installed latest package over existing installation and reapplied changes I'd made to "jenkins.xml".

          So far so good. Have to wait and see if next auto update works.

          THANKS

          Show
          phancox Peter Hancox added a comment - Didn't uninstall. Just installed latest package over existing installation and reapplied changes I'd made to "jenkins.xml". So far so good. Have to wait and see if next auto update works. THANKS
          Hide
          phancox Peter Hancox added a comment -

          Still no luck unfortunately. Attempted automatic upgrade from 1.582 to 1.583 and experienced exactly the same behaviour as previously.

          REGARDS
          Peter

          Show
          phancox Peter Hancox added a comment - Still no luck unfortunately. Attempted automatic upgrade from 1.582 to 1.583 and experienced exactly the same behaviour as previously. REGARDS Peter
          Hide
          danielbeck Daniel Beck added a comment -

          Comments indicate this is still relevant.

          Show
          danielbeck Daniel Beck added a comment - Comments indicate this is still relevant.
          danielbeck Daniel Beck made changes -
          Resolution Incomplete [ 4 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Hide
          mcsf M Chon added a comment - - edited

          On Windows, did an automatic upgrade from 1.554.2 to 1.565.3, and in the jenkins.err.log I see this:

          INFO: Scheduled a replacement of jenkins.exe

          Although when Jenkins comes up, the message at the bottom indicates that 1.565.3 is running.

          Instead of checking the checkbox 'Restart Jenkins when installation is complete and no jobs are running', we always restart Jenkins via the Services control panel, after the downloads are complete.

          I ended up updating the file manually, then restarting Jenkins.
          After doing so, the 'INFO: Scheduled a replacement of jenkins.exe' message went away.
          It is kind of disturbing to see that jenkins.exe was never actually updated since 05/02/2014 until today, which is when I first noticed that message in the Jenkins log.

          05/02/2014 09:15 AM 50,624 jenkins.exe.old
          10/15/2014 06:18 PM 55,776 jenkins.exe.new
          10/15/2014 06:18 PM 55,776 jenkins.exe

          Show
          mcsf M Chon added a comment - - edited On Windows, did an automatic upgrade from 1.554.2 to 1.565.3, and in the jenkins.err.log I see this: INFO: Scheduled a replacement of jenkins.exe Although when Jenkins comes up, the message at the bottom indicates that 1.565.3 is running. Instead of checking the checkbox 'Restart Jenkins when installation is complete and no jobs are running', we always restart Jenkins via the Services control panel, after the downloads are complete. I ended up updating the file manually, then restarting Jenkins. After doing so, the 'INFO: Scheduled a replacement of jenkins.exe' message went away. It is kind of disturbing to see that jenkins.exe was never actually updated since 05/02/2014 until today, which is when I first noticed that message in the Jenkins log. 05/02/2014 09:15 AM 50,624 jenkins.exe.old 10/15/2014 06:18 PM 55,776 jenkins.exe.new 10/15/2014 06:18 PM 55,776 jenkins.exe
          Hide
          danielbeck Daniel Beck added a comment -

          Anyone still experiencing this on recent Jenkins versions, with up to date jenkins.exe?

          Show
          danielbeck Daniel Beck added a comment - Anyone still experiencing this on recent Jenkins versions, with up to date jenkins.exe?
          ganthore Mark Austin made changes -
          Attachment jenkins.xml [ 32942 ]
          ganthore Mark Austin made changes -
          Hide
          ganthore Mark Austin added a comment - - edited

          Hey Daniel,

          I've managed to reproduce this issue multiple times using a full admin windows account while using the latest copy of winsw (1.18). It seems to me that the process that writes jenkins.war.tmp fails silently. I can confirm that the correct update is being downloaded, stored as jenkins.war.tmp and making a copy of the old file as jenkins.war.bak.

          I suspect it's happening due to the jenkins.exe still holding a file handle on the original war file and fails silently based off how I understand what's going on in the core code:
          https://github.com/jenkinsci/jenkins/blob/cf64ba04fe8b0312bb4492766d62b5a00b790432/core/src/main/java/hudson/lifecycle/Lifecycle.java#L137-L170

          It may be helpful to have the

          canRewriteHudsonWar()

          function report success vs failure in the logger. I guess it would be best to avoid throwing an exception to help prevent introducing API breakage.

          The windows event viewer will throw the following details when the service is terminated (that is, when you check the restart box), but even the OS logs fail to report a problem, so this fails silently by all accounts right now. No additional errors are reported in the wrapper log. Here is the windows event at the time the JVM is restarted after the update is downloaded to the system:

          Log Name:      Application
          Source:        Jenkins
          Date:          6/8/2016 2:12:14 PM
          Event ID:      0
          Task Category: None
          Level:         Information
          Keywords:      Classic
          User:          N/A
          Computer:      reubenWin
          Description:
          Child process [4860 - C:\jenkins\java\jre1.8.0_92\bin\java -server -d64 -Xrs -Xms2048m -Xmx2048m -XX:+AlwaysPreTouch -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Djenkins.model.Jenkins.logStartupPerformance=true -Dhudson.udp=-1 -Dhudson.DNSMultiCast.disabled=true -jar "C:\jenkins\jenkins.war" --httpListenAddress=0.0.0.0 --httpsPort=-1 --httpPort=8080 --ajp13Port=-1 --webroot=C:\jenkins\webroot] terminated with -1073741510
          Event Xml:
          <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
            <System>
              <Provider Name="Jenkins" />
              <EventID Qualifiers="0">0</EventID>
              <Level>4</Level>
              <Task>0</Task>
              <Keywords>0x80000000000000</Keywords>
              <TimeCreated SystemTime="2016-06-08T18:12:14.000000000Z" />
              <EventRecordID>1159</EventRecordID>
              <Channel>Application</Channel>
              <Computer>reubenWin</Computer>
              <Security />
            </System>
            <EventData>
              <Data>Child process [4860 - C:\jenkins\java\jre1.8.0_92\bin\java -server -d64 -Xrs -Xms2048m -Xmx2048m -XX:+AlwaysPreTouch -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Djenkins.model.Jenkins.logStartupPerformance=true -Dhudson.udp=-1 -Dhudson.DNSMultiCast.disabled=true -jar "C:\jenkins\jenkins.war" --httpListenAddress=0.0.0.0 --httpsPort=-1 --httpPort=8080 --ajp13Port=-1 --webroot=C:\jenkins\webroot] terminated with -1073741510</Data>
            </EventData>
          </Event>
          

          I've attached a copy of my jenkins.xml file if that helps:
          jenkins.xml

          Note that I intentionally use custom folder paths to help prevent windows from locking up files in use, hence the separation of individual directories like this:

          c:\jenkins
          c:\jenkins\webroot
          c:\jenkins\tmp
          c:\jenkins\data
          
          Show
          ganthore Mark Austin added a comment - - edited Hey Daniel, I've managed to reproduce this issue multiple times using a full admin windows account while using the latest copy of winsw (1.18). It seems to me that the process that writes jenkins.war.tmp fails silently. I can confirm that the correct update is being downloaded, stored as jenkins.war.tmp and making a copy of the old file as jenkins.war.bak. I suspect it's happening due to the jenkins.exe still holding a file handle on the original war file and fails silently based off how I understand what's going on in the core code: https://github.com/jenkinsci/jenkins/blob/cf64ba04fe8b0312bb4492766d62b5a00b790432/core/src/main/java/hudson/lifecycle/Lifecycle.java#L137-L170 It may be helpful to have the canRewriteHudsonWar() function report success vs failure in the logger. I guess it would be best to avoid throwing an exception to help prevent introducing API breakage. The windows event viewer will throw the following details when the service is terminated (that is, when you check the restart box), but even the OS logs fail to report a problem, so this fails silently by all accounts right now. No additional errors are reported in the wrapper log. Here is the windows event at the time the JVM is restarted after the update is downloaded to the system: Log Name: Application Source: Jenkins Date: 6/8/2016 2:12:14 PM Event ID: 0 Task Category: None Level: Information Keywords: Classic User: N/A Computer: reubenWin Description: Child process [4860 - C:\jenkins\java\jre1.8.0_92\bin\java -server -d64 -Xrs -Xms2048m -Xmx2048m -XX:+AlwaysPreTouch -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -Djava.awt.headless= true -Djava.net.preferIPv4Stack= true -Djenkins.model.Jenkins.logStartupPerformance= true -Dhudson.udp=-1 -Dhudson.DNSMultiCast.disabled= true -jar "C:\jenkins\jenkins.war" --httpListenAddress=0.0.0.0 --httpsPort=-1 --httpPort=8080 --ajp13Port=-1 --webroot=C:\jenkins\webroot] terminated with -1073741510 Event Xml: <Event xmlns= "http: //schemas.microsoft.com/win/2004/08/events/event" > < System > <Provider Name= "Jenkins" /> <EventID Qualifiers= "0" >0</EventID> <Level>4</Level> <Task>0</Task> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime= "2016-06-08T18:12:14.000000000Z" /> <EventRecordID>1159</EventRecordID> <Channel>Application</Channel> <Computer>reubenWin</Computer> <Security /> </ System > <EventData> <Data>Child process [4860 - C:\jenkins\java\jre1.8.0_92\bin\java -server -d64 -Xrs -Xms2048m -Xmx2048m -XX:+AlwaysPreTouch -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -Djava.awt.headless= true -Djava.net.preferIPv4Stack= true -Djenkins.model.Jenkins.logStartupPerformance= true -Dhudson.udp=-1 -Dhudson.DNSMultiCast.disabled= true -jar "C:\jenkins\jenkins.war" --httpListenAddress=0.0.0.0 --httpsPort=-1 --httpPort=8080 --ajp13Port=-1 --webroot=C:\jenkins\webroot] terminated with -1073741510</Data> </EventData> </Event> I've attached a copy of my jenkins.xml file if that helps: jenkins.xml Note that I intentionally use custom folder paths to help prevent windows from locking up files in use, hence the separation of individual directories like this: c:\jenkins c:\jenkins\webroot c:\jenkins\tmp c:\jenkins\data
          rtyler R. Tyler Croy made changes -
          Workflow JNJira [ 143617 ] JNJira + In-Review [ 186049 ]
          Hide
          hp Halvor Platou added a comment -

          It seems that this happens when when running with a custom JENKINS_HOME. If, so the "jenkins.copies" file is written to JENKINS_HOME and will not be used by the service wrapper when restarting. I have created a pull-request [#2992|https://github.com/jenkinsci/jenkins/pull/2992] with a proposed solution. Using environment variable BASE which is set by the service wrapper. 

          Show
          hp Halvor Platou added a comment - It seems that this happens when when running with a custom JENKINS_HOME. If, so the "jenkins.copies" file is written to JENKINS_HOME and will not be used by the service wrapper when restarting. I have created a pull-request [#2992| https://github.com/jenkinsci/jenkins/pull/2992 ] with a proposed solution. Using environment variable BASE which is set by the service wrapper. 
          oleg_nenashev Oleg Nenashev made changes -
          Remote Link This issue links to "https://github.com/jenkinsci/jenkins/pull/2992 (Web Link)" [ 17498 ]
          oleg_nenashev Oleg Nenashev made changes -
          Status Reopened [ 4 ] In Progress [ 3 ]
          oleg_nenashev Oleg Nenashev made changes -
          Status In Progress [ 3 ] In Review [ 10005 ]
          oleg_nenashev Oleg Nenashev made changes -
          Assignee Halvor Platou [ hp ]
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: hplatou
          Path:
          core/src/main/java/hudson/lifecycle/WindowsServiceLifecycle.java
          http://jenkins-ci.org/commit/jenkins/0efdf8fb4f8c56f1f32fb390c472cb2e98e67f56
          Log:
          JENKINS-13153 - Use directory from env:BASE when writing jenkins.copies (#2992)

          JENKINS-13153 - Use directory from env:BASE when writing jenkins.copies

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: hplatou Path: core/src/main/java/hudson/lifecycle/WindowsServiceLifecycle.java http://jenkins-ci.org/commit/jenkins/0efdf8fb4f8c56f1f32fb390c472cb2e98e67f56 Log: JENKINS-13153 - Use directory from env:BASE when writing jenkins.copies (#2992) JENKINS-13153 - Use directory from env:BASE when writing jenkins.copies
          Hide
          danielbeck Daniel Beck added a comment -

          Resolved in 2.77 (I think).

          Show
          danielbeck Daniel Beck added a comment - Resolved in 2.77 (I think).
          danielbeck Daniel Beck made changes -
          Status In Review [ 10005 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]

            People

            • Assignee:
              hp Halvor Platou
              Reporter:
              kristoffer Kristoffer Peterhänsel
            • Votes:
              4 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: