Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-53318

Version for a given plugin should be taken from environments first when overridden

    XMLWordPrintable

    Details

    • Sprint:
      Evergreen - Milestone 1
    • Similar Issues:

      Description

      Issue

      In UL 33, it seems like the version for docker-commons that has been installed is the 1.5.

      This version is the one defined in the main section under spec.plugins.
      But this version is overridden to 1.9 under the flavor-specific environments sectionto 1.9.

      The .json available at https://evergreen.jenkins.io/ for 32a278882aee26f7075e2c219a76dcbacc7b7532 (I think this is the one for UL33) seems correct. So I think the bug might be coming from the service side selection for a given plugin.

      This caused the update of my test instance, and it failed to restart, with the following in the logs:

      [SEVERE][2018-08-29 10:08:59] Failed Loading plugin Docker plugin v1.1.5 (docker-plugin) (from jenkins.InitReactorRunner$1 onTaskFailed)
      java.io.IOException: Docker plugin v1.1.5 failed to load.
       - Docker Commons Plugin v1.5 is older than required. To fix, install v1.9 or later.
       - Token Macro Plugin v2.0 is older than required. To fix, install v2.3 or later.
              at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:652)
              at hudson.PluginManager$2$1$1.run(PluginManager.java:517)
              at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
              at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
              at jenkins.model.Jenkins$5.runTask(Jenkins.java:1066)
              at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
              at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      

      And here's the state on the browser side:

      Expected behavior

      When a plugin is overridden under the environment specific flavor part, the overridden version defined under the flavor should always be the selected one.

        Attachments

          Issue Links

            Activity

            Hide
            batmat Baptiste Mathus added a comment -

            Jesse Glick is hinting that we might just want to forbid this. I'm inclined to agree: we could keep the JS code simple/simpler, and just forbid at the `essentials.yaml` level by fordidding to have different versions of a given plugin for "default" and all flavors involved.

            I think this would likely save some headaches and discrepancies in the future where different flavors would be actually running different versions of plugins for the same Update Level.

            Show
            batmat Baptiste Mathus added a comment - Jesse Glick is hinting that we might just want to forbid this. I'm inclined to agree: we could keep the JS code simple/simpler, and just forbid at the `essentials.yaml` level by fordidding to have different versions of a given plugin for "default" and all flavors involved. I think this would likely save some headaches and discrepancies in the future where different flavors would be actually running different versions of plugins for the same Update Level.
            Hide
            batmat Baptiste Mathus added a comment -

            OK, so

            1) I confirm this one is really critical, I could not get a publicly running evergreen instance trying again just now. Not blocking anymore since I merged https://github.com/jenkins-infra/evergreen/commit/f7cb1f9fad38a4d36a9798dcd0960353b23604fa and it just got pushed as a new UL.
            2) What actually concerns me even more right now is I realized no test is failing. So I do not understand why when running locally a docker-cloud flavor, docker-commons gets installed correctly as 1.9

            Show
            batmat Baptiste Mathus added a comment - OK, so 1) I confirm this one is really critical, I could not get a publicly running evergreen instance trying again just now. Not blocking anymore since I merged https://github.com/jenkins-infra/evergreen/commit/f7cb1f9fad38a4d36a9798dcd0960353b23604fa and it just got pushed as a new UL. 2) What actually concerns me even more right now is I realized no test is failing. So I do not understand why when running locally a docker-cloud flavor, docker-commons gets installed correctly as 1.9
            Hide
            batmat Baptiste Mathus added a comment -

            Mandie Smith any idea of why the version could be chosen differently? The only thing I can thing about is the fact we have many UL on the official backend at https://evergreen.jenkins.io, when we have only one locally when testing. I do not know the backend service code enough for this part to form a hypothesis on what could lead to this.

            To clarify my previous comment: the new UL41 fixes this issue by working around it by forcing docker-commons all the time. So one will not be able to reproduce this issue using https://evergreen.jenkins.io. But because our tests are not failing for this case, this means we could well get back to this situation sooner than we hope (well, I do hope we get back to it soon so that we fix it now, and not when we have more users...).

            Show
            batmat Baptiste Mathus added a comment - Mandie Smith any idea of why the version could be chosen differently? The only thing I can thing about is the fact we have many UL on the official backend at https://evergreen.jenkins.io , when we have only one locally when testing. I do not know the backend service code enough for this part to form a hypothesis on what could lead to this. To clarify my previous comment: the new UL41 fixes this issue by working around it by forcing docker-commons all the time. So one will not be able to reproduce this issue using https://evergreen.jenkins.io . But because our tests are not failing for this case, this means we could well get back to this situation sooner than we hope (well, I do hope we get back to it soon so that we fix it now, and not when we have more users...).
            Hide
            batmat Baptiste Mathus added a comment -

            We just met with Mandie Smith and agreed on the following:

            • she is going to do some checks and add tests in this feature area
            • Because we can only reproduce it using the production deployed backend, it's very likely we should first upgrade the backend to latest before delving into this too much.

            I can see for instances commits like cc9ed7e60d441ecdea8737af8aac43a022ef92c1 or 21733e1228f9341b755d4727cc3ab08e52221352 or 7ac8b77de9a4f648916fc57dcaf6c48b02552831 that could well be fixes for just that.

            Show
            batmat Baptiste Mathus added a comment - We just met with Mandie Smith and agreed on the following: she is going to do some checks and add tests in this feature area Because we can only reproduce it using the production deployed backend, it's very likely we should first upgrade the backend to latest before delving into this too much. I can see for instances commits like cc9ed7e60d441ecdea8737af8aac43a022ef92c1 or 21733e1228f9341b755d4727cc3ab08e52221352 or 7ac8b77de9a4f648916fc57dcaf6c48b02552831 that could well be fixes for just that.
            Hide
            batmat Baptiste Mathus added a comment -

            After (supposedly) upgrading the public backend, I still see this, but on upgrade... I do not really have time to dig into it as I need to move forward JENKINS-53273, but we'll really need to understand this.

            Show
            batmat Baptiste Mathus added a comment - After (supposedly) upgrading the public backend, I still see this, but on upgrade... I do not really have time to dig into it as I need to move forward JENKINS-53273 , but we'll really need to understand this.
            Hide
            rtyler R. Tyler Croy added a comment -

            I think Baptiste Mathus and I may have identified another symptom of this bug. My demo instance had its docker plugin deleted after the first boot (seemingly) and as a result subsequent restarts of the Jenkins process fail because the docker plugin and some others are not present on the file system anymore (thanks to the recent deletes support!)

            The update manifest computed for this client was:

            {"meta":{"channel":"general","level":72},"core":{},"plugins":{"updates":[],"deletes":["docker-java-api","docker-plugin","ssh-slaves"]}}
            
            Show
            rtyler R. Tyler Croy added a comment - I think Baptiste Mathus and I may have identified another symptom of this bug. My demo instance had its docker plugin deleted after the first boot (seemingly) and as a result subsequent restarts of the Jenkins process fail because the docker plugin and some others are not present on the file system anymore (thanks to the recent deletes support!) The update manifest computed for this client was: { "meta" :{ "channel" : "general" , "level" :72}, "core" :{}, "plugins" :{ "updates" :[], "deletes" :[ "docker-java-api" , "docker-plugin" , "ssh-slaves" ]}}
            Hide
            rtyler R. Tyler Croy added a comment -

            I believe I have found a reproduction case and MAYBE even fixed this issue with this pull request: https://github.com/jenkins-infra/evergreen/pull/260

            Show
            rtyler R. Tyler Croy added a comment - I believe I have found a reproduction case and MAYBE even fixed this issue with this pull request: https://github.com/jenkins-infra/evergreen/pull/260
            Hide
            rtyler R. Tyler Croy added a comment -

            I believe I have found this bug and fixed it with this pull request: https://github.com/jenkins-infra/evergreen/pull/263

            Show
            rtyler R. Tyler Croy added a comment - I believe I have found this bug and fixed it with this pull request: https://github.com/jenkins-infra/evergreen/pull/263

              People

              • Assignee:
                rtyler R. Tyler Croy
                Reporter:
                batmat Baptiste Mathus
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: