We have two templates for the ec2 plugin:
Jenkins doesn't create a new instance with template 2 when there are jobs in queue waiting for an available excutor on a running instance created with template 1, for example in our queue:
1. test_job (Restrict where this project can be run unchecked. Waiting for template1:test as all exectuors on template1:test are taken and we only want 1 instance with template1:test)
2. report_job (Restrict where this project can be run checked. Label expression: report. Waiting for template2:report as it does not exist)
report_job will be blocked until test_job is scheduled as Jenkins keeps trying to schedule test_job and ignores the request of an instance with template2 from report_job
UPDATE: I can confirm that this only happens when the first job in the queue has "Restrict where this project can be run" unchecked.
And I believe the problem is from Jenkins core not the ec2 plugin as the NodeProvisioner doesn't not look for all the labels as it should be (https://github.com/jenkinsci/jenkins/blob/jenkins-2.204.1/core/src/main/java/hudson/slaves/NodeProvisioner.java#L815)
Here is the log from hudson.slaves.NodeProvisioner class