Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-33879

can't proactively scale up ec2 slaves in anticipation demand

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Duplicate
    • Component/s: ec2-plugin
    • Labels:
      None
    • Environment:
      ec2 plugin 1.31
      jenkins 1.644
    • Similar Issues:

      Description

      Most of our nodes idle timeout overnight. We run a large Multi-configuration job early in the morning. We know we need a certain number of nodes online and used to be able to proactively provision instances via the /computer UI. In recent jenkins and ec2 plugin versions I observe the following:

      1. attempt to start a new slave of a given ami
      2. plugin finds there is an idle node of that particular type and returns its id, rather than creating a new one.

      Mar 29, 2016 4:45:31 PM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfo
      Launching ami-9fe6d8f5 for template Jenkins - Deployer Slave
      ...snip...
      Mar 29, 2016 4:45:32 PM FINE hudson.plugins.ec2.SlaveTemplate
       true - Node has capacity - can use it
      ...snip...
      Mar 29, 2016 4:45:32 PM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfo
      Using existing slave: i-b0614034
      

      In the above, i-b0614034 was an existing instance.

      This used to work just fine. Now it refuses to create a new node. The change in behavior seems to have been introduced in relation to commit 6b286b185ba41efc33ab8558a6c43969975e6238 for issues JENKINS-23787 EC2 not spooling up stopped nodes

      https://github.com/jenkinsci/ec2-plugin/commit/6b286b185ba41efc33ab8558a6c43969975e6238#diff-f2115e33148d3db7c133fe014ad9dfddR569

      I blame this code and a subsequent commit that added the additonal logging of " true - Node has capacity - can use it" where it returns the idle instance id rather than provisioning a new one as I explicity asked for.

      Exacerbating this is extremely slow instance provisioning in StandardStrategyImpl, which I intend to log as a separate story and cross reference.

        Attachments

          Issue Links

            Activity

            There are no comments yet on this issue.

              People

              • Assignee:
                francisu Francis Upton
                Reporter:
                trbaker Trevor Baker
              • Votes:
                1 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: