JenkinsIcon indicating the project type

Jenkins

Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55618

When using spot instances, requests 3-4 nodes when one is needed

    Details

    • Similar Issues:

      Description

      When it is time to request a worker node, more than one can be requested.

      It requests one, then rapidly checks for if its up (node needs a minute or so to come up when using spot instances), and fires another one, then another – until one of spots is finally there on time. This one becomes Jenkins slave, rest linger there until Spot Marketplace kills it.

      Seems to be 1.42 specific: I reverted to version 1.41 and it seems to work fine.

      This is how it looks in Jenkins log file:
      (notice really short delays, only sir-bd78adpq gets into Jenkins slave list)

      Jan 15, 2019 2:17:40 PM hudson.plugins.ec2.SlaveTemplate provisionSpot
      INFO: Spot instance id in provision: sir-dvfib1pn
      Jan 15, 2019 2:17:40 PM hudson.plugins.ec2.EC2Cloud provision
      INFO: SlaveTemplate

      {ami='ami-XXXXX', labels=''}. Attempting provision finished, excess workload: 0
      Jan 15, 2019 2:17:40 PM hudson.plugins.ec2.EC2Cloud provision
      INFO: We have now 1 computers, waiting for 1 more
      Jan 15, 2019 2:17:40 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
      INFO: Started provisioning current_ami (ami-XXXXX) from ec2-ec2 cloud with 1 executors. Remaining excess workload: 0
      Jan 15, 2019 2:17:40 PM hudson.plugins.ec2.EC2Cloud$1 call
      WARNING: SlaveTemplate{ami='ami-XXXXX', labels=''}

      . Node terminated is neither pending, neither running, its

      {2}. Terminate provisioning
      Jan 15, 2019 2:17:48 PM hudson.plugins.ec2.EC2Cloud provision
      INFO: SlaveTemplate{ami='ami-XXXXX', labels=''}. Attempting to provision slave needed by excess workload of 1 units
      Jan 15, 2019 2:17:49 PM hudson.plugins.ec2.SlaveTemplate provisionSpot
      INFO: Launching ami-XXXXX for template current_ami

      Jan 15, 2019 2:17:50 PM hudson.plugins.ec2.SlaveTemplate provisionSpot
      INFO: Spot instance id in provision: sir-bieg943m
      Jan 15, 2019 2:17:50 PM hudson.plugins.ec2.EC2Cloud provision
      INFO: SlaveTemplate{ami='ami-XXXXX', labels=''}. Attempting provision finished, excess workload: 0
      Jan 15, 2019 2:17:50 PM hudson.plugins.ec2.EC2Cloud provision
      INFO: We have now 1 computers, waiting for 1 more
      Jan 15, 2019 2:17:50 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
      INFO: Started provisioning current_ami (ami-XXXXX) from ec2-ec2 cloud with 1 executors. Remaining excess workload: 0
      Jan 15, 2019 2:17:50 PM hudson.plugins.ec2.EC2Cloud$1 call
      WARNING: SlaveTemplate{ami='ami-XXXXX', labels=''}. Node terminated is neither pending, neither running, its {2}

      . Terminate provisioning
      Jan 15, 2019 2:17:58 PM hudson.plugins.ec2.EC2Cloud provision
      INFO: SlaveTemplate

      {ami='ami-XXXXX', labels=''}. Attempting to provision slave needed by excess workload of 1 units

      Jan 15, 2019 2:17:59 PM hudson.plugins.ec2.SlaveTemplate provisionSpot
      INFO: Launching ami-XXXXX for template current_ami

      Jan 15, 2019 2:18:00 PM hudson.plugins.ec2.SlaveTemplate provisionSpot
      INFO: Spot instance id in provision: sir-5mtr9erp
      Jan 15, 2019 2:18:00 PM hudson.plugins.ec2.EC2Cloud provision
      INFO: SlaveTemplate{ami='ami-XXXXX', labels=''}

      . Attempting provision finished, excess workload: 0
      Jan 15, 2019 2:18:00 PM hudson.plugins.ec2.EC2Cloud provision
      INFO: We have now 1 computers, waiting for 1 more
      Jan 15, 2019 2:18:00 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
      INFO: Started provisioning current_ami (ami-XXXXX) from ec2-ec2 cloud with 1 executors. Remaining excess workload: 0
      Jan 15, 2019 2:18:00 PM hudson.plugins.ec2.EC2Cloud$1 call
      WARNING: SlaveTemplate

      {ami='ami-XXXXX', labels=''}. Node terminated is neither pending, neither running, its {2}. Terminate provisioning
      Jan 15, 2019 2:18:08 PM hudson.plugins.ec2.EC2Cloud provision
      INFO: SlaveTemplate{ami='ami-XXXXX', labels=''}

      . Attempting to provision slave needed by excess workload of 1 units
      Jan 15, 2019 2:18:09 PM hudson.plugins.ec2.SlaveTemplate provisionSpot
      INFO: Launching ami-XXXXX for template current_ami
      Jan 15, 2019 2:18:10 PM hudson.plugins.ec2.SlaveTemplate provisionSpot

      INFO: Spot instance id in provision: sir-bd78adpq
      Jan 15, 2019 2:18:10 PM hudson.plugins.ec2.EC2Cloud provision
      INFO: SlaveTemplate

      {ami='ami-XXXXX', labels=''}

      . Attempting provision finished, excess workload: 0
      Jan 15, 2019 2:18:10 PM hudson.plugins.ec2.EC2Cloud provision
      INFO: We have now 1 computers, waiting for 1 more
      Jan 15, 2019 2:18:10 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
      INFO: Started provisioning current_ami (ami-XXXXX) from ec2-ec2 cloud with 1 executors. Remaining excess workload: 0
      Jan 15, 2019 2:18:21 PM hudson.plugins.ec2.EC2Cloud$1 call

        Attachments

          Activity

          Ascending order - Click to sort in descending order
          8 older comments
          Hide
          Permalink
          vnaum Vladislav Naumov added a comment -

          > how long takes your node to be ready for use ?

          Up to a minute with Spot instances. It used to be much worse few years ago – it could take up to 5 minutes back then.

          Most of this time is spent in that marketplace – on-demand instance comes up 2-3x times faster from same AMI.

          Show
          vnaum Vladislav Naumov added a comment - > how long takes your node to be ready for use ? Up to a minute with Spot instances. It used to be much worse few years ago – it could take up to 5 minutes back then. Most of this time is spent in that marketplace – on-demand instance comes up 2-3x times faster from same AMI.
          Hide
          Permalink
          smishraec Shubham Mishra added a comment -

          Can someone help me to resolve this issue as I am stuck and not able to resolve this issue.

          Vladislav Naumov, could you please help me to resolve this issue.

          Show
          smishraec Shubham Mishra added a comment - Can someone help me to resolve this issue as I am stuck and not able to resolve this issue. Vladislav Naumov , could you please help me to resolve this issue.
          Hide
          Permalink
          smishraec Shubham Mishra added a comment -

          Hi Sylvain LEBEAUPIN, I am also getting the same error in my error logs while using the ec2 plugin version 1.43.

          If I change the version to 1.40,would the spot instance work fine?

          Please suggest.

          Show
          smishraec Shubham Mishra added a comment - Hi Sylvain LEBEAUPIN , I am also getting the same error in my error logs while using the ec2 plugin version 1.43. If I change the version to 1.40,would the spot instance work fine? Please suggest.
          Hide
          Permalink
          vnaum Vladislav Naumov added a comment -

          > If I change the version to 1.40,would the spot instance work fine?

          Try rolling back to 1.41 first.
          It worked fine for me.
          But then again, 1.43 works for me, too.

          Show
          vnaum Vladislav Naumov added a comment - > If I change the version to 1.40,would the spot instance work fine? Try rolling back to 1.41 first. It worked fine for me. But then again, 1.43 works for me, too.
          Hide
          Permalink
          smishraec Shubham Mishra added a comment -

          Thanjs Vladislav Naumov, I will check and test it.

          BTW do we need to downgrade the jenkins version also because as of now I am using jenkins version 2.138 and ec2 plyugin version is 1.43.

          Please suggest.

          Show
          smishraec Shubham Mishra added a comment - Thanjs Vladislav Naumov , I will check and test it. BTW do we need to downgrade the jenkins version also because as of now I am using jenkins version 2.138 and ec2 plyugin version is 1.43. Please suggest.

            People

            • Assignee:
              thoulen FABRIZIO MANFREDI
              Reporter:
              vnaum Vladislav Naumov
            • Votes:
              13 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

              • Created:
                Updated: