Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54358

AWS flavor got broken by a wrong update

    Details

    • Sprint:
      Evergreen - Milestone 2
    • Similar Issues:

      Description

      Problem statement

      The AWS instance I have for testing got broken when updating from 134 to 135. (I'm attaching the updates.auditLog file for deeper analysis.
      In the UI, we get the usual

      java.lang.IllegalArgumentException: No hudson.slaves.Cloud implementation found for amazonEC2
      Possible solution: Try to install 'configuration-as-code-support' plugin
      	at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.findDescribableBySymbol(HeteroDescribableConfigurator.java:142)
      	at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.configure(HeteroDescribableConfigurator.java:93)
      	at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.check(HeteroDescribableConfigurator.java:101)
      	at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.check(HeteroDescribableConfigurator.java:43)
      	at io.jenkins.plugins.casc.BaseConfigurator.configure(BaseConfigurator.java:322)
      	at io.jenkins.plugins.casc.BaseConfigurator.check(BaseConfigurator.java:265)
      	at io.jenkins.plugins.casc.ConfigurationAsCode.lambda$checkWith$8(ConfigurationAsCode.java:640)
      	at io.jenkins.plugins.casc.ConfigurationAsCode.invokeWith(ConfigurationAsCode.java:605)
      	at io.jenkins.plugins.casc.ConfigurationAsCode.checkWith(ConfigurationAsCode.java:640)
      	at io.jenkins.plugins.casc.ConfigurationAsCode.configureWith(ConfigurationAsCode.java:627)
      	at io.jenkins.plugins.casc.ConfigurationAsCode.configureWith(ConfigurationAsCode.java:539)
      	at io.jenkins.plugins.casc.ConfigurationAsCode.configure(ConfigurationAsCode.java:270)
      	at io.jenkins.plugins.casc.ConfigurationAsCode.init(ConfigurationAsCode.java:262)
      Caused: java.lang.reflect.InvocationTargetException
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
      Caused: java.lang.Error
      	at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:110)
      	at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
      	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
      	at jenkins.model.Jenkins$5.runTask(Jenkins.java:1066)
      	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
      	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused: org.jvnet.hudson.reactor.ReactorException
      	at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:282)
      	at jenkins.InitReactorRunner.run(InitReactorRunner.java:48)
      	at jenkins.model.Jenkins.executeReactor(Jenkins.java:1100)
      	at jenkins.model.Jenkins.<init>(Jenkins.java:904)
      	at hudson.model.Hudson.<init>(Hudson.java:85)
      	at hudson.model.Hudson.<init>(Hudson.java:81)
      	at hudson.WebAppMain$3.run(WebAppMain.java:233)
      Caused: hudson.util.HudsonFailedToLoad
      	at hudson.WebAppMain$3.run(WebAppMain.java:250)
      

      (Exposed also is another issue with continuously rolling back, but let's process first the fact the instance got borked by, apparently, a wrong computation of plugins to be updated)

      Expected

      The update should go fine, and the result of the instance should be the target UL.

        Attachments

        1. failed-restart.log
          399 kB
        2. ul134.json
          41 kB
        3. ul135.json
          41 kB
        4. updates.auditlog
          71 kB

          Issue Links

            Activity

            Hide
            batmat Baptiste Mathus added a comment -

            So the update from 134 to 135 failed:

             

            cat updates.auditlog| jq .updateLevel | head
            134
            135
            134
            133
            132
            131
            130
            129
            128
            127
             

            So first interesting thing (just retrieved the full json from UL 134 and 135 using INFRA-1769, yey!):

            Despite the only difference between the metrics plugin:

            evergreen_prod=> select commit from updates where id=135;
                              commit
            ------------------------------------------
             6bffe54dc7265b90e714da21f1cb248e1e2f2319
            (1 row)
            
            $ diff ul134.json ul135.json
            2c2
            <   "timestamp": 1540568443089,
            ---
            >   "timestamp": 1540570564042,
            624,625c624,625
            <       "version": "4.0.2.2",
            <       "url": "https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/metrics/4.0.2.2/metrics-4.0.2.2.hpi",
            ---
            >       "version": "4.0.2.3-rc229.d9b0eb3c576d",
            >       "url": "https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/metrics/4.0.2.3-rc229.d9b0eb3c576d/metrics-4.0.2.3-rc229.d9b0eb3c576d.hpi",
            628c628
            <         "signature": "b60b49ab946cd8b960f999876b7da880fadf9ca07dd9524f9f0af63a5d0342eb"
            ---
            >         "signature": "1c1b95da5a0131a1af73ca81d90e4135ee3d05919a7382cc0c4f8195c9133aba"
            

            Here is what was served by the backend to the instance for this update:

            (as we can see, the part about metrics plugin is correct, but there's an chunk about EC2 and aws-java-sdk that shouldn't be here to be analyzed)

            {
              "timestamp": "2018-10-29T14:31:30.363Z",
              "updateLevel": 135,
              "manifest": {
                "meta": {
                  "channel": "general",
                  "level": 135
                },
                "core": {},
                "plugins": {
                  "updates": [
                    {
                      "url": "https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/metrics/4.0.2.3-rc229.d9b0eb3c576d/metrics-4.0.2.3-rc229.d9b0eb3c576d.hpi",
                      "artifactId": "metrics",
                      "checksum": {
                        "type": "sha256",
                        "signature": "1c1b95da5a0131a1af73ca81d90e4135ee3d05919a7382cc0c4f8195c9133aba"
                      }
                    },
                    {
                      "url": "https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/aws-java-sdk/1.11.341/aws-java-sdk-1.11.341.hpi",
                      "artifactId": "aws-java-sdk",
                      "checksum": {
                        "type": "sha256",
                        "signature": "ebe7cca7755dc020d138ef537a7bf7b61e3bbad41f7f0f6be4004917c0769238"
                      }
                    },
                    {
                      "url": "https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/ec2/1.39/ec2-1.39.hpi",
                      "artifactId": "ec2",
                      "checksum": {
                        "type": "sha256",
                        "signature": "fdd8670b1a71bcdacc5d7a358fa67fb8c44efc6f2ecc717d5e572c669934a0a7"
                      }
                    }
                  ],
                  "deletes": []
                }
              }
            }
            
            Show
            batmat Baptiste Mathus added a comment - So the update from 134 to 135 failed:   cat updates.auditlog| jq .updateLevel | head 134 135 134 133 132 131 130 129 128 127 So first interesting thing (just retrieved the full json from UL 134 and 135 using INFRA-1769 , yey!): Despite the only difference between the metrics plugin: evergreen_prod=> select commit from updates where id=135; commit ------------------------------------------ 6bffe54dc7265b90e714da21f1cb248e1e2f2319 (1 row) $ diff ul134.json ul135.json 2c2 < "timestamp": 1540568443089, --- > "timestamp": 1540570564042, 624,625c624,625 < "version": "4.0.2.2", < "url": "https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/metrics/4.0.2.2/metrics-4.0.2.2.hpi", --- > "version": "4.0.2.3-rc229.d9b0eb3c576d", > "url": "https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/metrics/4.0.2.3-rc229.d9b0eb3c576d/metrics-4.0.2.3-rc229.d9b0eb3c576d.hpi", 628c628 < "signature": "b60b49ab946cd8b960f999876b7da880fadf9ca07dd9524f9f0af63a5d0342eb" --- > "signature": "1c1b95da5a0131a1af73ca81d90e4135ee3d05919a7382cc0c4f8195c9133aba" Here is what was served by the backend to the instance for this update: (as we can see, the part about metrics plugin is correct, but there's an chunk about EC2 and aws-java-sdk that shouldn't be here to be analyzed) { "timestamp": "2018-10-29T14:31:30.363Z", "updateLevel": 135, "manifest": { "meta": { "channel": "general", "level": 135 }, "core": {}, "plugins": { "updates": [ { "url": "https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/metrics/4.0.2.3-rc229.d9b0eb3c576d/metrics-4.0.2.3-rc229.d9b0eb3c576d.hpi", "artifactId": "metrics", "checksum": { "type": "sha256", "signature": "1c1b95da5a0131a1af73ca81d90e4135ee3d05919a7382cc0c4f8195c9133aba" } }, { "url": "https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/aws-java-sdk/1.11.341/aws-java-sdk-1.11.341.hpi", "artifactId": "aws-java-sdk", "checksum": { "type": "sha256", "signature": "ebe7cca7755dc020d138ef537a7bf7b61e3bbad41f7f0f6be4004917c0769238" } }, { "url": "https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/ec2/1.39/ec2-1.39.hpi", "artifactId": "ec2", "checksum": { "type": "sha256", "signature": "fdd8670b1a71bcdacc5d7a358fa67fb8c44efc6f2ecc717d5e572c669934a0a7" } } ], "deletes": [] } } }

              People

              • Assignee:
                batmat Baptiste Mathus
                Reporter:
                batmat Baptiste Mathus
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: