Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54358

AWS flavor got broken by a wrong update

    Details

    • Similar Issues:
    • Sprint:
      Evergreen - Milestone 2

      Description

      Problem statement

      The AWS instance I have for testing got broken when updating from 134 to 135. (I'm attaching the updates.auditLog file for deeper analysis.
      In the UI, we get the usual

      java.lang.IllegalArgumentException: No hudson.slaves.Cloud implementation found for amazonEC2
      Possible solution: Try to install 'configuration-as-code-support' plugin
      	at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.findDescribableBySymbol(HeteroDescribableConfigurator.java:142)
      	at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.configure(HeteroDescribableConfigurator.java:93)
      	at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.check(HeteroDescribableConfigurator.java:101)
      	at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.check(HeteroDescribableConfigurator.java:43)
      	at io.jenkins.plugins.casc.BaseConfigurator.configure(BaseConfigurator.java:322)
      	at io.jenkins.plugins.casc.BaseConfigurator.check(BaseConfigurator.java:265)
      	at io.jenkins.plugins.casc.ConfigurationAsCode.lambda$checkWith$8(ConfigurationAsCode.java:640)
      	at io.jenkins.plugins.casc.ConfigurationAsCode.invokeWith(ConfigurationAsCode.java:605)
      	at io.jenkins.plugins.casc.ConfigurationAsCode.checkWith(ConfigurationAsCode.java:640)
      	at io.jenkins.plugins.casc.ConfigurationAsCode.configureWith(ConfigurationAsCode.java:627)
      	at io.jenkins.plugins.casc.ConfigurationAsCode.configureWith(ConfigurationAsCode.java:539)
      	at io.jenkins.plugins.casc.ConfigurationAsCode.configure(ConfigurationAsCode.java:270)
      	at io.jenkins.plugins.casc.ConfigurationAsCode.init(ConfigurationAsCode.java:262)
      Caused: java.lang.reflect.InvocationTargetException
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
      Caused: java.lang.Error
      	at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:110)
      	at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
      	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
      	at jenkins.model.Jenkins$5.runTask(Jenkins.java:1066)
      	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
      	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused: org.jvnet.hudson.reactor.ReactorException
      	at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:282)
      	at jenkins.InitReactorRunner.run(InitReactorRunner.java:48)
      	at jenkins.model.Jenkins.executeReactor(Jenkins.java:1100)
      	at jenkins.model.Jenkins.<init>(Jenkins.java:904)
      	at hudson.model.Hudson.<init>(Hudson.java:85)
      	at hudson.model.Hudson.<init>(Hudson.java:81)
      	at hudson.WebAppMain$3.run(WebAppMain.java:233)
      Caused: hudson.util.HudsonFailedToLoad
      	at hudson.WebAppMain$3.run(WebAppMain.java:250)
      

      (Exposed also is another issue with continuously rolling back, but let's process first the fact the instance got borked by, apparently, a wrong computation of plugins to be updated)

      Expected

      The update should go fine, and the result of the instance should be the target UL.

        Attachments

        1. updates.auditlog
          71 kB
        2. ul134.json
          41 kB
        3. ul135.json
          41 kB
        4. failed-restart.log
          399 kB

          Issue Links

            Activity

            Hide
            batmat Baptiste Mathus added a comment -

            So the update from 134 to 135 failed:

             

            cat updates.auditlog| jq .updateLevel | head
            134
            135
            134
            133
            132
            131
            130
            129
            128
            127
             

            So first interesting thing (just retrieved the full json from UL 134 and 135 using INFRA-1769, yey!):

            Despite the only difference between the metrics plugin:

            evergreen_prod=> select commit from updates where id=135;
                              commit
            ------------------------------------------
             6bffe54dc7265b90e714da21f1cb248e1e2f2319
            (1 row)
            
            $ diff ul134.json ul135.json
            2c2
            <   "timestamp": 1540568443089,
            ---
            >   "timestamp": 1540570564042,
            624,625c624,625
            <       "version": "4.0.2.2",
            <       "url": "https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/metrics/4.0.2.2/metrics-4.0.2.2.hpi",
            ---
            >       "version": "4.0.2.3-rc229.d9b0eb3c576d",
            >       "url": "https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/metrics/4.0.2.3-rc229.d9b0eb3c576d/metrics-4.0.2.3-rc229.d9b0eb3c576d.hpi",
            628c628
            <         "signature": "b60b49ab946cd8b960f999876b7da880fadf9ca07dd9524f9f0af63a5d0342eb"
            ---
            >         "signature": "1c1b95da5a0131a1af73ca81d90e4135ee3d05919a7382cc0c4f8195c9133aba"
            

            Here is what was served by the backend to the instance for this update:

            (as we can see, the part about metrics plugin is correct, but there's an chunk about EC2 and aws-java-sdk that shouldn't be here to be analyzed)

            {
              "timestamp": "2018-10-29T14:31:30.363Z",
              "updateLevel": 135,
              "manifest": {
                "meta": {
                  "channel": "general",
                  "level": 135
                },
                "core": {},
                "plugins": {
                  "updates": [
                    {
                      "url": "https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/metrics/4.0.2.3-rc229.d9b0eb3c576d/metrics-4.0.2.3-rc229.d9b0eb3c576d.hpi",
                      "artifactId": "metrics",
                      "checksum": {
                        "type": "sha256",
                        "signature": "1c1b95da5a0131a1af73ca81d90e4135ee3d05919a7382cc0c4f8195c9133aba"
                      }
                    },
                    {
                      "url": "https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/aws-java-sdk/1.11.341/aws-java-sdk-1.11.341.hpi",
                      "artifactId": "aws-java-sdk",
                      "checksum": {
                        "type": "sha256",
                        "signature": "ebe7cca7755dc020d138ef537a7bf7b61e3bbad41f7f0f6be4004917c0769238"
                      }
                    },
                    {
                      "url": "https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/ec2/1.39/ec2-1.39.hpi",
                      "artifactId": "ec2",
                      "checksum": {
                        "type": "sha256",
                        "signature": "fdd8670b1a71bcdacc5d7a358fa67fb8c44efc6f2ecc717d5e572c669934a0a7"
                      }
                    }
                  ],
                  "deletes": []
                }
              }
            }
            
            Show
            batmat Baptiste Mathus added a comment - So the update from 134 to 135 failed:   cat updates.auditlog| jq .updateLevel | head 134 135 134 133 132 131 130 129 128 127 So first interesting thing (just retrieved the full json from UL 134 and 135 using INFRA-1769 , yey!): Despite the only difference between the metrics plugin: evergreen_prod=> select commit from updates where id=135; commit ------------------------------------------ 6bffe54dc7265b90e714da21f1cb248e1e2f2319 (1 row) $ diff ul134.json ul135.json 2c2 < "timestamp": 1540568443089, --- > "timestamp": 1540570564042, 624,625c624,625 < "version": "4.0.2.2", < "url": "https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/metrics/4.0.2.2/metrics-4.0.2.2.hpi", --- > "version": "4.0.2.3-rc229.d9b0eb3c576d", > "url": "https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/metrics/4.0.2.3-rc229.d9b0eb3c576d/metrics-4.0.2.3-rc229.d9b0eb3c576d.hpi", 628c628 < "signature": "b60b49ab946cd8b960f999876b7da880fadf9ca07dd9524f9f0af63a5d0342eb" --- > "signature": "1c1b95da5a0131a1af73ca81d90e4135ee3d05919a7382cc0c4f8195c9133aba" Here is what was served by the backend to the instance for this update: (as we can see, the part about metrics plugin is correct, but there's an chunk about EC2 and aws-java-sdk that shouldn't be here to be analyzed) { "timestamp": "2018-10-29T14:31:30.363Z", "updateLevel": 135, "manifest": { "meta": { "channel": "general", "level": 135 }, "core": {}, "plugins": { "updates": [ { "url": "https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/metrics/4.0.2.3-rc229.d9b0eb3c576d/metrics-4.0.2.3-rc229.d9b0eb3c576d.hpi", "artifactId": "metrics", "checksum": { "type": "sha256", "signature": "1c1b95da5a0131a1af73ca81d90e4135ee3d05919a7382cc0c4f8195c9133aba" } }, { "url": "https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/aws-java-sdk/1.11.341/aws-java-sdk-1.11.341.hpi", "artifactId": "aws-java-sdk", "checksum": { "type": "sha256", "signature": "ebe7cca7755dc020d138ef537a7bf7b61e3bbad41f7f0f6be4004917c0769238" } }, { "url": "https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/ec2/1.39/ec2-1.39.hpi", "artifactId": "ec2", "checksum": { "type": "sha256", "signature": "fdd8670b1a71bcdacc5d7a358fa67fb8c44efc6f2ecc717d5e572c669934a0a7" } } ], "deletes": [] } } }

              People

              • Assignee:
                batmat Baptiste Mathus
                Reporter:
                batmat Baptiste Mathus
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: