Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-33945

can't provision new nodes when a matching node is marked offline

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Duplicate
    • Component/s: ec2-plugin
    • Labels:
      None
    • Environment:
      ec2 1.31
    • Similar Issues:

      Description

      You can not provision a new node if there is an existing node from the same SlaveTemplate which is marked offline. Offline nodes instance id will be returned and no new node will be provisioned.

        Attachments

          Issue Links

            Activity

            Hide
            trbaker Trevor Baker added a comment -
            Show
            trbaker Trevor Baker added a comment - I provided PR 187 https://github.com/jenkinsci/ec2-plugin/pull/187
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Trevor Baker
            Path:
            src/main/java/hudson/plugins/ec2/SlaveTemplate.java
            http://jenkins-ci.org/commit/ec2-plugin/8226bdb66579342b85766cc1eb12994e895f4685
            Log:
            JENKINS-33945 allow node provisioning when existing matching node is offline
            You can not provision a new node when an existing node from the template
            is marked offline. This change consults isOffline() to allow new node
            provisioning. Nodes can either be offline due to channel errors or user
            interaction. Either way, don't block new nodes.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Trevor Baker Path: src/main/java/hudson/plugins/ec2/SlaveTemplate.java http://jenkins-ci.org/commit/ec2-plugin/8226bdb66579342b85766cc1eb12994e895f4685 Log: JENKINS-33945 allow node provisioning when existing matching node is offline You can not provision a new node when an existing node from the template is marked offline. This change consults isOffline() to allow new node provisioning. Nodes can either be offline due to channel errors or user interaction. Either way, don't block new nodes.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Francis Upton IV
            Path:
            src/main/java/hudson/plugins/ec2/SlaveTemplate.java
            http://jenkins-ci.org/commit/ec2-plugin/1227f790c1049a24094ae503aab2ce5ea89cdf36
            Log:
            Merge pull request #187 from tjbaker/offline

            JENKINS-33945 allow node provisioning when matching node is offline

            Compare: https://github.com/jenkinsci/ec2-plugin/compare/e3bfb31528fa...1227f790c104

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Francis Upton IV Path: src/main/java/hudson/plugins/ec2/SlaveTemplate.java http://jenkins-ci.org/commit/ec2-plugin/1227f790c1049a24094ae503aab2ce5ea89cdf36 Log: Merge pull request #187 from tjbaker/offline JENKINS-33945 allow node provisioning when matching node is offline Compare: https://github.com/jenkinsci/ec2-plugin/compare/e3bfb31528fa...1227f790c104
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Francis Upton IV
            Path:
            src/main/java/hudson/plugins/ec2/SlaveTemplate.java
            http://jenkins-ci.org/commit/ec2-plugin/7a3973e1129054029d5f260e6e4df2ddd17dcbaa
            Log:
            JENKINS-33945 Allow offline nodes to be provisions (back out change)

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Francis Upton IV Path: src/main/java/hudson/plugins/ec2/SlaveTemplate.java http://jenkins-ci.org/commit/ec2-plugin/7a3973e1129054029d5f260e6e4df2ddd17dcbaa Log: JENKINS-33945 Allow offline nodes to be provisions (back out change)
            Hide
            francisu Francis Upton added a comment -

            Trevor Baker Does this issue apply only to manually provisioning nodes? I think this is related to JENKINS-32690. I would like to fix this by ignoring offline nodes only in the manual provisioning case (as opposed to automatic provisioning due to workload). Would this be acceptable?

            Show
            francisu Francis Upton added a comment - Trevor Baker Does this issue apply only to manually provisioning nodes? I think this is related to JENKINS-32690 . I would like to fix this by ignoring offline nodes only in the manual provisioning case (as opposed to automatic provisioning due to workload). Would this be acceptable?
            Hide
            trbaker Trevor Baker added a comment -

            It is also a problem for nodes provisioned automatically via StandardStrategyImpl

            Create a 3x3 axis Multi-configuration job. Launch a single slave with the label that can run the job, then mark if offline. It won't start any new nodes and the job will be stuck.

            Mar 31, 2016 6:46:45 PM FINER hudson.slaves.NodeProvisioner
            Consulting hudson.slaves.NodeProvisioner$StandardStrategyImpl@6437ee5a provisioning strategy with state StrategyState{label=deployer, snapshot=LoadStatisticsSnapshot{definedExecutors=1, onlineExecutors=0, connectingExecutors=0, busyExecutors=0, idleExecutors=0, availableExecutors=0, queueLength=9}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0}
            Mar 31, 2016 6:46:45 PM FINE hudson.slaves.NodeProvisioner
            Excess workload 7.906 detected. (planned capacity=0,connecting capacity=0,Qlen=7.906,available=0.174&0,online=0,m=0.5)
            Mar 31, 2016 6:46:45 PM FINE hudson.plugins.ec2.EC2Cloud
            Attempting provision, excess workload: 8
            Mar 31, 2016 6:46:45 PM FINE hudson.plugins.ec2.EC2Cloud
            Counting current slaves:  All AMIS
            Mar 31, 2016 6:46:45 PM FINE hudson.plugins.ec2.EC2Cloud
            Existing instance found: i-3a616ebe AMI: ami-ce2d26a4 Template: null
            Mar 31, 2016 6:46:45 PM FINE hudson.plugins.ec2.EC2Cloud
            Counting current slaves:  AMI: ami-ce2d26a4
            Mar 31, 2016 6:46:45 PM FINE hudson.plugins.ec2.EC2Cloud
            Existing instance found: i-3a616ebe AMI: ami-ce2d26a4 Template: Jenkins - Deployer Slave
            Mar 31, 2016 6:46:45 PM FINE hudson.plugins.ec2.EC2Cloud
            Available Total Slaves: 2147483646 Available AMI slaves: 2147483646 AMI: ami-ce2d26a4 TemplateDesc: Jenkins - Deployer Slave
            Mar 31, 2016 6:46:45 PM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfo
            Launching ami-ce2d26a4 for template Jenkins - Deployer Slave
            Mar 31, 2016 6:46:46 PM FINE hudson.plugins.ec2.SlaveTemplate
            Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-ce2d26a4]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-xxx]}, {Name: instance.group-id,Values: [sg-xxx]}, {Name: key-name,Values: [dev]}, {Name: instance-type,Values: [t2.small]}, {Name: tag:Name,Values: [DEV-JENKINSSLAVE]}, {Name: tag:ENV,Values: [DEV]}],}
            Mar 31, 2016 6:46:46 PM FINE hudson.plugins.ec2.SlaveTemplate
            checkInstance: {InstanceId: i-3a616ebe,ImageId: snip}
            Mar 31, 2016 6:46:46 PM FINE hudson.plugins.ec2.SlaveTemplate
            Found existing corresponding Jenkins slave: i-3a616ebe
            Mar 31, 2016 6:46:46 PM FINE hudson.plugins.ec2.SlaveTemplate
             true - Node has capacity - can use it
            Mar 31, 2016 6:46:46 PM FINE hudson.plugins.ec2.SlaveTemplate
            Found existing instance: {InstanceId: i-3a616ebe,snip}
            node: i-3a616ebe
            Mar 31, 2016 6:46:46 PM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfo
            Found existing pending or running: running instance: {InstanceId: i-3a616ebe,snip}
            Mar 31, 2016 6:46:46 PM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfo
            Using existing slave: i-3a616ebe
            

            Instance i-3a616ebe in the above example is the temporarily offline node.

            Show
            trbaker Trevor Baker added a comment - It is also a problem for nodes provisioned automatically via StandardStrategyImpl Create a 3x3 axis Multi-configuration job. Launch a single slave with the label that can run the job, then mark if offline. It won't start any new nodes and the job will be stuck. Mar 31, 2016 6:46:45 PM FINER hudson.slaves.NodeProvisioner Consulting hudson.slaves.NodeProvisioner$StandardStrategyImpl@6437ee5a provisioning strategy with state StrategyState{label=deployer, snapshot=LoadStatisticsSnapshot{definedExecutors=1, onlineExecutors=0, connectingExecutors=0, busyExecutors=0, idleExecutors=0, availableExecutors=0, queueLength=9}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0} Mar 31, 2016 6:46:45 PM FINE hudson.slaves.NodeProvisioner Excess workload 7.906 detected. (planned capacity=0,connecting capacity=0,Qlen=7.906,available=0.174&0,online=0,m=0.5) Mar 31, 2016 6:46:45 PM FINE hudson.plugins.ec2.EC2Cloud Attempting provision, excess workload: 8 Mar 31, 2016 6:46:45 PM FINE hudson.plugins.ec2.EC2Cloud Counting current slaves: All AMIS Mar 31, 2016 6:46:45 PM FINE hudson.plugins.ec2.EC2Cloud Existing instance found: i-3a616ebe AMI: ami-ce2d26a4 Template: null Mar 31, 2016 6:46:45 PM FINE hudson.plugins.ec2.EC2Cloud Counting current slaves: AMI: ami-ce2d26a4 Mar 31, 2016 6:46:45 PM FINE hudson.plugins.ec2.EC2Cloud Existing instance found: i-3a616ebe AMI: ami-ce2d26a4 Template: Jenkins - Deployer Slave Mar 31, 2016 6:46:45 PM FINE hudson.plugins.ec2.EC2Cloud Available Total Slaves: 2147483646 Available AMI slaves: 2147483646 AMI: ami-ce2d26a4 TemplateDesc: Jenkins - Deployer Slave Mar 31, 2016 6:46:45 PM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfo Launching ami-ce2d26a4 for template Jenkins - Deployer Slave Mar 31, 2016 6:46:46 PM FINE hudson.plugins.ec2.SlaveTemplate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-ce2d26a4]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-xxx]}, {Name: instance.group-id,Values: [sg-xxx]}, {Name: key-name,Values: [dev]}, {Name: instance-type,Values: [t2.small]}, {Name: tag:Name,Values: [DEV-JENKINSSLAVE]}, {Name: tag:ENV,Values: [DEV]}],} Mar 31, 2016 6:46:46 PM FINE hudson.plugins.ec2.SlaveTemplate checkInstance: {InstanceId: i-3a616ebe,ImageId: snip} Mar 31, 2016 6:46:46 PM FINE hudson.plugins.ec2.SlaveTemplate Found existing corresponding Jenkins slave: i-3a616ebe Mar 31, 2016 6:46:46 PM FINE hudson.plugins.ec2.SlaveTemplate true - Node has capacity - can use it Mar 31, 2016 6:46:46 PM FINE hudson.plugins.ec2.SlaveTemplate Found existing instance: {InstanceId: i-3a616ebe,snip} node: i-3a616ebe Mar 31, 2016 6:46:46 PM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfo Found existing pending or running: running instance: {InstanceId: i-3a616ebe,snip} Mar 31, 2016 6:46:46 PM INFO hudson.plugins.ec2.SlaveTemplate logProvisionInfo Using existing slave: i-3a616ebe Instance i-3a616ebe in the above example is the temporarily offline node.

              People

              • Assignee:
                francisu Francis Upton
                Reporter:
                trbaker Trevor Baker
              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: