Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55639

ec2-plugin only attempts to retrieve spot instance using old id

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Blocker
    • Resolution: Fixed
    • Component/s: ec2-plugin
    • Labels:
      None
    • Environment:
      Jenkins 2.150.2
      ec2-plugin 1.42
    • Similar Issues:

      Description

      When Jenkins attempts to provision spot instances, after spot requests are created (incorrectly, as per JENKINS-55618) Jenkins will try to retrieve the spot instance based on the last working instance id from before we upgraded the plugin, instead of the instance id returned by the spot request. 

      INFO: Spot instance id in provision: sir-34si4rxjINFO: Spot instance id in provision: sir-34si4rxjJan 17, 2019 4:32:10 AM hudson.plugins.ec2.CloudHelper getInstanceINFO: Unexpected number of reservations reported by EC2 for instance id 'i-0b934d05d88272fc4', expected 1 result, found []. Instance seems to be dead.Jan 17, 2019 4:32:10 AM hudson.plugins.ec2.EC2Cloud provisionWARNING: SlaveTemplate{ami='ami-09009aeca3492b598', labels='agent linux'}. Exception during provisioningcom.amazonaws.AmazonClientException: Unexpected number of reservations reported by EC2 for instance id 'i-0b934d05d88272fc4', expected 1 result, found []. Instance seems to be dead. at hudson.plugins.ec2.CloudHelper.getInstance(CloudHelper.java:54) at hudson.plugins.ec2.CloudHelper.getInstanceWithRetry(CloudHelper.java:25) at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:499) at hudson.plugins.ec2.EC2AbstractSlave.<init>(EC2AbstractSlave.java:159) at hudson.plugins.ec2.EC2SpotSlave.<init>(EC2SpotSlave.java:44) at hudson.plugins.ec2.EC2SpotSlave.<init>(EC2SpotSlave.java:37) at hudson.plugins.ec2.SlaveTemplate.newSpotSlave(SlaveTemplate.java:979) at hudson.plugins.ec2.SlaveTemplate.provisionSpot(SlaveTemplate.java:919) at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:464) at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:578) at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:594) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:715) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

      This hasn't happened with on-demand instances.

      This may be related to JENKINS-55618; I'm affected by that bug as well, but I wasn't sure how closely related it was so opted to open a new issue. 

        Attachments

          Activity

          Hide
          malberghini Mike Alberghini added a comment -

          We also faced this problem on Jenkins 2.157 with ec2-plugin 1.42.  Switching our spot instances to regular did not fix the issue, but downgrading the plugin to 1.41 did

          Show
          malberghini Mike Alberghini added a comment - We also faced this problem on Jenkins 2.157 with ec2-plugin 1.42.  Switching our spot instances to regular did not fix the issue, but downgrading the plugin to 1.41 did
          Hide
          vdczzz Victor Chavez added a comment - - edited

          On Jenkins 2.150.2 EC2 plugin 1.42 we get this behavior intermittently. We oscillate between this and JENKINS-55556.

          Show
          vdczzz Victor Chavez added a comment - - edited On Jenkins 2.150.2 EC2 plugin 1.42 we get this behavior intermittently. We oscillate between this and  JENKINS-55556 .
          Hide
          paulalphaorisg Paul P added a comment - - edited

          EDIT: Looks like I spoke too soon, the problem re-appeared.

          I had this issue and uninstalling the plugin and re-installing it fixed it.

          To be specific:

          1. backup your config.xml
          2. uninstall the plugin
          3. restart jenkins
          4. re-install the plugin (I lost my configuration)
          5. stop jenkins
          6. replace the config.xml
          7. start Jenkins

          I am using Jenkins 2.176 and ec2 plugin 1.42

          Show
          paulalphaorisg Paul P added a comment - - edited EDIT: Looks like I spoke too soon, the problem re-appeared. I had this issue and uninstalling the plugin and re-installing it fixed it. To be specific: backup your config.xml uninstall the plugin restart jenkins re-install the plugin (I lost my configuration) stop jenkins replace the config.xml start Jenkins I am using Jenkins 2.176 and ec2 plugin 1.42
          Hide
          thoulen FABRIZIO MANFREDI added a comment - - edited

          The 1.43 contains a fix on the missing default value of the new options that are not present for people that come from old plugin. Can you test ? 

          Show
          thoulen FABRIZIO MANFREDI added a comment - - edited The 1.43 contains a fix on the missing default value of the new options that are not present for people that come from old plugin. Can you test ? 
          Hide
          yanstoic753 Yan Corneille added a comment -

          Actully I can't, I'm sorry. I had to rollback and fix my instance EC2 plugin to v1.39 due to https://issues.jenkins-ci.org/browse/JENKINS-52362

          Show
          yanstoic753 Yan Corneille added a comment - Actully I can't, I'm sorry. I had to rollback and fix my instance EC2 plugin to v1.39 due to https://issues.jenkins-ci.org/browse/JENKINS-52362 . 
          Hide
          paulalphaorisg Paul P added a comment -

          I can confirm 1.43 & 2.177 work fine with Spot instances

          Show
          paulalphaorisg Paul P added a comment - I can confirm 1.43 & 2.177 work fine with Spot instances
          Hide
          tommoyang Tao Yang added a comment -

          Thanks FABRIZIO MANFREDI, it looks to me like it's working correctly now!

          Show
          tommoyang Tao Yang added a comment - Thanks FABRIZIO MANFREDI , it looks to me like it's working correctly now!

            People

            • Assignee:
              thoulen FABRIZIO MANFREDI
              Reporter:
              tommoyang Tao Yang
            • Votes:
              10 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: