Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48365

Command launcher plugin does not get installed on core update crossing 2.86

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      The plugin manager is supposed to install detached plugins if you upgrade from a version which previously had their functionality as part of core to a version which now has it detached. This is not working for some reason and we need to investigate why.

      Steps to reproduce:

      Install 2.84

      Pass setup wizard installing no plugins.

      Update to 2.93 for same home

      Expected:

      Update installs command-launcher plugin (plus deps as needed)

      Actual:

      No plugins installed

        Attachments

          Issue Links

            Activity

            danielbeck Daniel Beck created issue -
            Hide
            jglick Jesse Glick added a comment -

            Possibly related to this apparent bug. I think we have no test coverage for this kind of upgrade scenario because AFAIK JenkinsRule cannot express it.

            Show
            jglick Jesse Glick added a comment - Possibly related to this apparent bug . I think we have no test coverage for this kind of upgrade scenario because AFAIK JenkinsRule cannot express it.
            danielbeck Daniel Beck made changes -
            Field Original Value New Value
            Link This issue is related to JENKINS-41727 [ JENKINS-41727 ]
            danielbeck Daniel Beck made changes -
            Link This issue is related to JENKINS-41727 [ JENKINS-41727 ]
            recampbell Ryan Campbell made changes -
            Description h3. Steps to reproduce:

            Install 2.84

            Pass setup wizard installing no plugins.

            Update to 2.93 for same home
            h3. Expected:

            Update installs command-launcher plugin (plus deps as needed)
            h3. Actual:

            No plugins installed
            The plugin manager is supposed to install detached plugins if you upgrade from a version which previously had them installed to a version which now has them detached. This is not working for some reason and we need to investigate why.

            h3. Steps to reproduce:

            Install 2.84

            Pass setup wizard installing no plugins.

            Update to 2.93 for same home
            h3. Expected:

            Update installs command-launcher plugin (plus deps as needed)
            h3. Actual:

            No plugins installed
            danielbeck Daniel Beck made changes -
            Description The plugin manager is supposed to install detached plugins if you upgrade from a version which previously had them installed to a version which now has them detached. This is not working for some reason and we need to investigate why.

            h3. Steps to reproduce:

            Install 2.84

            Pass setup wizard installing no plugins.

            Update to 2.93 for same home
            h3. Expected:

            Update installs command-launcher plugin (plus deps as needed)
            h3. Actual:

            No plugins installed
            The plugin manager is supposed to install detached plugins if you upgrade from a version which previously had their functionality as part of core to a version which now has it detached. This is not working for some reason and we need to investigate why.
            h3. Steps to reproduce:

            Install 2.84

            Pass setup wizard installing no plugins.

            Update to 2.93 for same home
            h3. Expected:

            Update installs command-launcher plugin (plus deps as needed)
            h3. Actual:

            No plugins installed
            Hide
            danielbeck Daniel Beck added a comment -

            Can the ATH do upgrade tests? We could go from 2.85 to SUT and see whether the plugin gets installed (for every future pair of 'attached core version' and 'plugin')

            Show
            danielbeck Daniel Beck added a comment - Can the ATH do upgrade tests? We could go from 2.85 to SUT and see whether the plugin gets installed (for every future pair of 'attached core version' and 'plugin')
            cloudbees CloudBees Inc. made changes -
            Remote Link This issue links to "CloudBees Internal OSS-2588 (Web Link)" [ 18470 ]
            Hide
            dnusbaum Devin Nusbaum added a comment -

            I was able to reproduce the issue. I didn't see any log entries about upgrading during the upgrade process (even with a custom logger for PluginManager). I used the "Upgrade automatically" button from CoreUpdateMonitor, is it expected for the block in the following if statement to not be entered in that case?

            if (InstallState.UPGRADE.equals(installState)) {
                LOGGER.log(INFO, "Upgrading Jenkins. ...
                ...
            } else {
                ... code Jesse linked to ...
            }
            

            The logic in the else would only upgrade a detached plugin that is already installed which is not the case for command-launcher when upgrading from 2.84 to 2.93. Maybe a recent change has broken something with InstallState?

            Show
            dnusbaum Devin Nusbaum added a comment - I was able to reproduce the issue. I didn't see any log entries about upgrading during the upgrade process (even with a custom logger for PluginManager). I used the "Upgrade automatically" button from CoreUpdateMonitor, is it expected for the block in the following if statement to not be entered in that case? if (InstallState.UPGRADE.equals(installState)) { LOGGER.log(INFO, "Upgrading Jenkins. ... ... } else { ... code Jesse linked to ... } The logic in the else would only upgrade a detached plugin that is already installed which is not the case for command-launcher when upgrading from 2.84 to 2.93. Maybe a recent change has broken something with InstallState?
            dnusbaum Devin Nusbaum made changes -
            Assignee Devin Nusbaum [ dnusbaum ]
            dnusbaum Devin Nusbaum made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            Hide
            danielbeck Daniel Beck added a comment -

            Maybe a recent change has broken something with InstallState?

            That's my guess as well, but so far we have no idea when this last worked. Step 1 is determining when it broke. Note that there are plenty of other detached plugins that are supposed to be installed when you go from 'part of core' to 'in a plugin'. The previous last detached plugin was between 2.7.x and 2.19.x, so this may have been broken for a while (in my limited testing, it appears to have been broken even back then, or has some nontrivial precondition – going from 2.0 to 2.19.4, even having saved the global config and having a job set up, did not install bouncycastle-api).

            If you're going by whether a detached plugin gets installed, you need a variation of the usual bisect search for the culprit version ensuring there's always at least one plugin

            Show
            danielbeck Daniel Beck added a comment - Maybe a recent change has broken something with InstallState? That's my guess as well, but so far we have no idea when this last worked. Step 1 is determining when it broke. Note that there are plenty of other detached plugins that are supposed to be installed when you go from 'part of core' to 'in a plugin'. The previous last detached plugin was between 2.7.x and 2.19.x, so this may have been broken for a while (in my limited testing, it appears to have been broken even back then, or has some nontrivial precondition – going from 2.0 to 2.19.4, even having saved the global config and having a job set up, did not install bouncycastle-api). If you're going by whether a detached plugin gets installed, you need a variation of the usual bisect search for the culprit version ensuring there's always at least one plugin
            Hide
            dnusbaum Devin Nusbaum added a comment - - edited

            As far as I can tell this has never worked in 2.X. In 1.X all of the plugins bundled in the war get installed, so the detached plugins are added to new installations without problem. I tested various version combos with the first version before 2.16 (including 1.651), and the second after 2.18, but bouncycastle-api was never installed (I tried installing plugins, creating jobs, and changing config.xml before the upgrade but bouncycastle-api was never installed).

            After attaching a debugger and doing an upgrade, I realized PluginManager#loadDetachedPlugins is called before InstallUtil#getDefaultInstallState, so the upgrade branch will never be taken. Looking through Jenkins' constructor, the call to Jenkins#executeReactor (which will execute the 'Loading bundled plugins' task that calls PluginManager#loadDetachedPlugins) does indeed happen a few lines before the call to InstallUtil#proceedToNext (which will update the install state to UPGRADE if an upgrade occurs). As far as I can tell the call to executeReactor does not return until all tasks are complete (though the tasks run in parallel by default), so there isn't even a race condition that would allow this to work occasionally.

            I think the easiest fix here is to check InstallUtil.getLastExecVersion() against Jenkins.VERSION directly in PluginManager#loadDetachedPlugins since Jenkins.installState hasn't been initialized.

            Show
            dnusbaum Devin Nusbaum added a comment - - edited As far as I can tell this has never worked in 2.X. In 1.X all of the plugins bundled in the war get installed, so the detached plugins are added to new installations without problem. I tested various version combos with the first version before 2.16 (including 1.651), and the second after 2.18, but bouncycastle-api was never installed (I tried installing plugins, creating jobs, and changing config.xml before the upgrade but bouncycastle-api was never installed). After attaching a debugger and doing an upgrade, I realized PluginManager#loadDetachedPlugins is called before InstallUtil#getDefaultInstallState , so the upgrade branch will never be taken. Looking through Jenkins' constructor, the call to Jenkins#executeReactor (which will execute the 'Loading bundled plugins' task that calls PluginManager#loadDetachedPlugins ) does indeed happen a few lines before the call to  InstallUtil#proceedToNext (which will update the install state to UPGRADE if an upgrade occurs). As far as I can tell the call to executeReactor does not return until all tasks are complete (though the tasks run in parallel by default), so there isn't even a race condition that would allow this to work occasionally. I think the easiest fix here is to check InstallUtil.getLastExecVersion()  against Jenkins.VERSION  directly in PluginManager#loadDetachedPlugins since Jenkins.installState hasn't been initialized.
            jglick Jesse Glick made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "PR 3193 (Web Link)" [ 19359 ]
            danielbeck Daniel Beck made changes -
            Labels lts-candidate
            oleg_nenashev Oleg Nenashev made changes -
            Labels lts-candidate lts-candidate regression
            Hide
            danielbeck Daniel Beck added a comment -

            Fixed towards 2.96

            Show
            danielbeck Daniel Beck added a comment - Fixed towards 2.96
            danielbeck Daniel Beck made changes -
            Status In Review [ 10005 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            dnusbaum Devin Nusbaum made changes -
            Link This issue relates to JENKINS-48604 [ JENKINS-48604 ]
            danielbeck Daniel Beck made changes -
            Link This issue depends on JENKINS-48604 [ JENKINS-48604 ]
            olivergondza Oliver Gondža made changes -
            Labels lts-candidate regression 2.89.3-fixed regression
            andresrc Andres Rodriguez made changes -
            Link This issue relates to JENKINS-48899 [ JENKINS-48899 ]

              People

              • Assignee:
                dnusbaum Devin Nusbaum
                Reporter:
                danielbeck Daniel Beck
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: