Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60657

EC2 plugins with multiple templates: fails to start an instance for job in the middle of the queue

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Component/s: ec2-plugin
    • Labels:
    • Environment:
      Jenkins: 2.204.1
      Amazon EC2 plugin (ID: ec2): 1.47
    • Similar Issues:

      Description

      We have two templates for the ec2 plugin:

      template1
      label: test default
      instance cap: 1
      usage: NORMAL
      template2
      label: report
      instance cap: 3
      usage: EXCLUSIVE

        

      Jenkins doesn't create a new instance with template 2 when there are jobs in queue waiting for an available excutor on a running instance created with template 1, for example in our queue:

      1. test_job (Restrict where this project can be run unchecked. Waiting for template1:test as all exectuors on template1:test are taken and we only want 1 instance with template1:test)

      2. report_job (Restrict where this project can be run checked. Label expression: report. Waiting for template2:report as it does not exist)

      report_job will be blocked until test_job is scheduled as Jenkins keeps trying to schedule test_job and ignores the request of an instance with template2 from report_job

       

      UPDATE: I can confirm that this only happens when the first job in the queue has "Restrict where this project can be run" unchecked.

       

      And I believe the problem is from Jenkins core not the ec2 plugin as the NodeProvisioner doesn't not look for all the labels as it should be (https://github.com/jenkinsci/jenkins/blob/jenkins-2.204.1/core/src/main/java/hudson/slaves/NodeProvisioner.java#L815)

       

      Here is the log from hudson.slaves.NodeProvisioner class

      Jan 16, 2020 9:38:08 PM FINER hudson.slaves.NodeProvisioner
      Consulting hudson.plugins.ec2.NoDelayProvisionerStrategy@1cfb55fa provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0}
      Jan 16, 2020 9:38:09 PM FINER hudson.slaves.NodeProvisioner
      Consulting hudson.slaves.NodeProvisioner$StandardStrategyImpl@6a13a6e5 provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0}
      Jan 16, 2020 9:38:09 PM FINE hudson.slaves.NodeProvisioner
      Excess workload 117.81 detected. (planned capacity=0,connecting capacity=0,Qlen=117.81,available=0&0,online=6,m=0.106)
      Jan 16, 2020 9:38:18 PM FINER hudson.slaves.NodeProvisioner
      Consulting hudson.plugins.ec2.NoDelayProvisionerStrategy@1cfb55fa provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0}
      Jan 16, 2020 9:38:18 PM FINER hudson.slaves.NodeProvisioner
      Consulting hudson.slaves.NodeProvisioner$StandardStrategyImpl@6a13a6e5 provisioning strategy with state StrategyState{label=null, snapshot=LoadStatisticsSnapshot{definedExecutors=6, onlineExecutors=6, connectingExecutors=0, busyExecutors=6, idleExecutors=0, availableExecutors=0, queueLength=127}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0}
      Jan 16, 2020 9:38:18 PM FINE hudson.slaves.NodeProvisioner
      Excess workload 118.729 detected. (planned capacity=0,connecting capacity=0,Qlen=118.729,available=0&0,online=6,m=0.106)
      

       

        Attachments

          Activity

          Hide
          h35gao Handi Gao added a comment - - edited

          This may be solved by commit https://github.com/jenkinsci/jenkins/commit/1125b1b4d7a5280ce0bd0d4c3eb464e172bf94ec

          I will close the ticket once we confirm that this is solved in version 2.209. Need to wait for Blueocean to release a new version before we can upgrade Jenkins to the latest version as we use jenkinsci/blueocean as base image.

          Show
          h35gao Handi Gao added a comment - - edited This may be solved by commit  https://github.com/jenkinsci/jenkins/commit/1125b1b4d7a5280ce0bd0d4c3eb464e172bf94ec I will close the ticket once we confirm that this is solved in version 2.209. Need to wait for Blueocean to release a new version before we can upgrade Jenkins to the latest version as we use jenkinsci/blueocean as base image.

            People

            • Assignee:
              thoulen FABRIZIO MANFREDI
              Reporter:
              h35gao Handi Gao
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: