Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-23850

PATCH: EC2-plugin always starting new slaves instead of restarting existing

    Details

    • Similar Issues:

      Description

      Every time I start a build, Jenkins launches a new slave rather than restarting one of the stopped instances. It is successfully stopping the instance when it hits the idle time.

      During idle, the log shows lots of this (the _check, idleTimeout, stop entries appear once for every instance currently registered):

      Jul 16, 2014 12:40:47 PM hudson.model.AsyncPeriodicWork$1 run
      INFO: Started EC2 alive slaves monitor
      Jul 16, 2014 12:40:48 PM hudson.model.AsyncPeriodicWork$1 run
      INFO: Finished EC2 alive slaves monitor. 1172 ms
      Jul 16, 2014 12:41:53 PM hudson.plugins.ec2.EC2RetentionStrategy _check
      INFO: Idle timeout: edifestivalsapi build slave (i-ce32a08c)
      Jul 16, 2014 12:41:53 PM hudson.plugins.ec2.EC2AbstractSlave idleTimeout
      INFO: EC2 instance idle time expired: i-ce32a08c
      Jul 16, 2014 12:41:53 PM hudson.plugins.ec2.EC2AbstractSlave stop
      INFO: EC2 instance stopped: i-ce32a08c
      

      Then a build is triggered and the idle timeout checks run again (again, one set of entries for every instance):

      Jul 16, 2014 12:44:23 PM com.cloudbees.jenkins.GitHubPushTrigger$1 run
      INFO: SCM changes detected in edifestivalsapi-master. Triggering  #36
      Jul 16, 2014 12:45:53 PM hudson.plugins.ec2.EC2RetentionStrategy _check
      INFO: Idle timeout: edifestivalsapi build slave (i-ce32a08c)
      Jul 16, 2014 12:45:53 PM hudson.plugins.ec2.EC2AbstractSlave idleTimeout
      INFO: EC2 instance idle time expired: i-ce32a08c
      Jul 16, 2014 12:45:53 PM hudson.plugins.ec2.EC2AbstractSlave stop
      INFO: EC2 instance stopped: i-ce32a08c
      

      And then the plugin starts to provision a new instance - apparently without any attempt to restart a stopped slave.

      Jul 16, 2014 12:46:33 PM hudson.plugins.ec2.EC2Cloud provision
      INFO: Excess workload after pending Spot instances: 1
      Jul 16, 2014 12:46:33 PM hudson.plugins.ec2.EC2Cloud addProvisionedSlave
      INFO: Provisioning for AMI ami-57ea3d20; Estimated number of total slaves: 0; Estimated number of slaves for ami ami-57ea3d20: 0
      Launching ami-57ea3d20 for template edifestivalsapi build slave
      Jul 16, 2014 12:46:33 PM hudson.slaves.NodeProvisioner update
      INFO: Started provisioning edifestivalsapi build slave (ami-57ea3d20) from ec2-eu-west-1 with 1 executors. Remaining excess workload:0.0
      Looking for existing instances: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-57ea3d20]}, {Name: group-name,Values: [jenkins-build-slave]}, {Name: key-name,Values: [build-slave]}, {Name: instance-type,Values: [t1.micro]}, {Name: tag:Name,Values: [edifestivalsapi-build-slave]}, {Name: tag:Project,Values: [edifestivalsapi]}, {Name: instance-state-name,Values: [stopped, stopping]}],}
      No existing instance found - created: {InstanceId: i-eb35a8a9,ImageId: ami-57ea3d20,State: {Code: 0,Name: pending},"**REDACTED**}
      

      Then another block of the idle timeout checks while the instance is launched, and then this:

      Jul 16, 2014 12:47:44 PM hudson.slaves.NodeProvisioner update
      INFO: edifestivalsapi build slave (ami-57ea3d20) provisioningE successfully completed. We have now 8 computer(s)
      Jul 16, 2014 12:47:47 PM hudson.node_monitors.ResponseTimeMonitor$1 monitor
      WARNING: Making edifestivalsapi build slave (i-ce32a08c) offline because it’s not responding
      

      The UI shows all the slaves that were launched for previous jobs but shows them as offline with "Time out for last 5 try". When I manually start the instance (by clicking onto the slave page and clicking "Launch slave agent") I see that the stopped instance is restarted and comes online as expected.

      It does successfully run a subsequent build on the correct existing node if there is one still running.

      So my hunch is that that Jenkins somehow isn't detecting that it has a stopped instance for the given AMI?

      I have two slave types configured, both using the same AMI but with different labels.

      I had initially posted this as a comment on https://issues.jenkins-ci.org/browse/JENKINS-23787 but it sounds like it may not be related - he's seeing it waiting indefinitely for the instance to start, where I'm seeing it skipping and spinning up a new node straight away, leaving the old one marked as not available.

        Attachments

          Issue Links

            Activity

            Hide
            acoulton Andrew Coulton added a comment - - edited

            Originally thought this was related to https://issues.jenkins-ci.org/browse/JENKINS-23787 but it may not be.

            Show
            acoulton Andrew Coulton added a comment - - edited Originally thought this was related to https://issues.jenkins-ci.org/browse/JENKINS-23787 but it may not be.
            Hide
            acoulton Andrew Coulton added a comment -

            I've identified that this happens when using the eu-west-1 region, because the describeInstances API does not return results when used with the group-name filter. The same behaviour has been observed at https://github.com/aws/aws-sdk-java/issues/213 and I have opened a discussion on the AWS forum.

            Per the EC2 API docs the instance.group-name filter is functionally identical, and I have found that this query does produce expected results in eu-west-1 and elsewhere.

            I've submitted a 1-line pull request to switch to the instance.group-name filter at https://github.com/jenkinsci/ec2-plugin/pull/99 which I think will resolve this issue ahead of any response/resolution by the AWS team.

            Show
            acoulton Andrew Coulton added a comment - I've identified that this happens when using the eu-west-1 region, because the describeInstances API does not return results when used with the group-name filter. The same behaviour has been observed at https://github.com/aws/aws-sdk-java/issues/213 and I have opened a discussion on the AWS forum . Per the EC2 API docs the instance.group-name filter is functionally identical, and I have found that this query does produce expected results in eu-west-1 and elsewhere. I've submitted a 1-line pull request to switch to the instance.group-name filter at https://github.com/jenkinsci/ec2-plugin/pull/99 which I think will resolve this issue ahead of any response/resolution by the AWS team.
            Hide
            ringerc Craig Ringer added a comment -

            Consider adding some instrumentation to SlaveTemplate.java, particularly the provisionOndemand(...) function.

            this changeset: https://github.com/jenkinsci/ec2-plugin/pull/101

            might be interesting for the purpose.

            Show
            ringerc Craig Ringer added a comment - Consider adding some instrumentation to SlaveTemplate.java, particularly the provisionOndemand(...) function. this changeset: https://github.com/jenkinsci/ec2-plugin/pull/101 might be interesting for the purpose.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Andrew Coulton
            Path:
            src/main/java/hudson/plugins/ec2/SlaveTemplate.java
            http://jenkins-ci.org/commit/ec2-plugin/14d537a08728f0baef88ed2d33e73696f681433f
            Log:
            [FIXED JENKINS-23850] Workaround AWS bug that hides stopped slaves

            Use instance.group-name instead of group-name as the describeInstances
            filter when searching for stopped slaves to restart.

            Some EC2 regions are incorrectly returning empty describeInstances
            results when the group-name filter is used. This causes jenkins to start
            new slaves instead of restarting stopped instances.

            instance.group-name is functionally identical and seems to work
            consistently.

            https://issues.jenkins-ci.org/browse/JENKINS-23850

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Coulton Path: src/main/java/hudson/plugins/ec2/SlaveTemplate.java http://jenkins-ci.org/commit/ec2-plugin/14d537a08728f0baef88ed2d33e73696f681433f Log: [FIXED JENKINS-23850] Workaround AWS bug that hides stopped slaves Use instance.group-name instead of group-name as the describeInstances filter when searching for stopped slaves to restart. Some EC2 regions are incorrectly returning empty describeInstances results when the group-name filter is used. This causes jenkins to start new slaves instead of restarting stopped instances. instance.group-name is functionally identical and seems to work consistently. https://issues.jenkins-ci.org/browse/JENKINS-23850

              People

              • Assignee:
                francisu Francis Upton
                Reporter:
                acoulton Andrew Coulton
              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: