-
Bug
-
Resolution: Not A Defect
-
Blocker
-
None
-
Jenkins 2.187
AWS EC2 1.47
I'm trying to provision AWS EC2 agents as build slaves for our Jenkins master, whenever a job matching the ami labels (defined in Jenkins config) enters the build queue.
Cloud config:
All the builds that I schedule will wait for the existing AWS node to finish building, even though the Instance Cap is set to 30 (I've tried to tie multiple jobs to the same label and they all wait for the one single node to finish building):
The job is configured to run concurrent builds:
In the Jenkins logs I see this recurrently:
Planned 0 new nodes Nov 29, 2019 1:37:37 PM FINE hudson.plugins.ec2.NoDelayProvisionerStrategy applyAfter provisioning, available capacity=0, currentDemand=1 Nov 29, 2019 1:37:37 PM FINE hudson.plugins.ec2.NoDelayProvisionerStrategy applyProvisioning not complete, consulting remaining strategies Nov 29, 2019 1:37:37 PM INFO hudson.plugins.ec2.EC2Cloud provisionSlaveTemplate{ami='ami-0266c7b78a0d89503', labels='c42xlarge'}. Attempting to provision slave needed by excess workload of 1 units Nov 29, 2019 1:37:37 PM FINE hudson.plugins.ec2.EC2Cloud countCurrentEC2SlavesCounting current slaves: All AMIS Jenkins Server: http://deb-jenkins-prd.ullink.lan/ Nov 29, 2019 1:37:37 PM FINE hudson.plugins.ec2.EC2Cloud countCurrentEC2SlavesExisting instance found: i-01e4a3e64c7dac1e7 AMI: ami-0266c7b78a0d89503 Jenkins Server: http://deb-jenkins-prd.ullink.lan/ Nov 29, 2019 1:37:37 PM FINE hudson.plugins.ec2.EC2Cloud countCurrentEC2SlavesCounting current slaves: AMI: ami-0266c7b78a0d89503 TemplateDesc: AWS Windows Node Jenkins Server: http://deb-jenkins-prd.ullink.lan/ Nov 29, 2019 1:37:37 PM FINE hudson.plugins.ec2.EC2Cloud countCurrentEC2SlavesExisting instance found: i-01e4a3e64c7dac1e7 AMI: ami-0266c7b78a0d89503 Template: AWS Windows Node Jenkins Server: http://deb-jenkins-prd.ullink.lan/ Nov 29, 2019 1:37:37 PM FINE hudson.plugins.ec2.EC2Cloud getPossibleNewSlavesCountAvailable Total Slaves: 0 Available AMI slaves: 29 AMI: ami-0266c7b78a0d89503 TemplateDesc: AWS Windows Node Nov 29, 2019 1:37:37 PM INFO hudson.plugins.ec2.EC2Cloud getNewOrExistingAvailableSlaveSlaveTemplate{ami='ami-0266c7b78a0d89503', labels='c42xlarge'}. Cannot provision - no capacity for instances: 0 Nov 29, 2019 1:37:37 PM WARNING hudson.plugins.ec2.EC2Cloud provisionCan't raise nodes for SlaveTemplate{ami='ami-0266c7b78a0d89503', labels='c42xlarge'} Nov 29, 2019 1:37:46 PM FINE hudson.plugins.ec2.NoDelayProvisionerStrategy applyAvailable capacity=0, currentDemand=1 Nov 29, 2019 1:37:47 PM INFO hudson.plugins.ec2.EC2Cloud provisionSlaveTemplate{ami='ami-0266c7b78a0d89503', labels='c42xlarge'}. Attempting to provision slave needed by excess workload of 1 units Nov 29, 2019 1:37:47 PM FINE hudson.plugins.ec2.EC2Cloud countCurrentEC2SlavesCounting current slaves: All AMIS Jenkins Server: http://deb-jenkins-prd.ullink.lan/ Nov 29, 2019 1:37:47 PM FINE hudson.plugins.ec2.EC2Cloud countCurrentEC2SlavesExisting instance found: i-01e4a3e64c7dac1e7 AMI: ami-0266c7b78a0d89503 Jenkins Server: http://deb-jenkins-prd.ullink.lan/ Nov 29, 2019 1:37:47 PM FINE hudson.plugins.ec2.EC2Cloud countCurrentEC2SlavesCounting current slaves: AMI: ami-0266c7b78a0d89503 TemplateDesc: AWS Windows Node Jenkins Server: http://deb-jenkins-prd.ullink.lan/ Nov 29, 2019 1:37:47 PM FINE hudson.plugins.ec2.EC2Cloud countCurrentEC2SlavesExisting instance found: i-01e4a3e64c7dac1e7 AMI: ami-0266c7b78a0d89503 Template: AWS Windows Node Jenkins Server: http://deb-jenkins-prd.ullink.lan/ Nov 29, 2019 1:37:47 PM FINE hudson.plugins.ec2.EC2Cloud getPossibleNewSlavesCountAvailable Total Slaves: 0 Available AMI slaves: 29 AMI: ami-0266c7b78a0d89503 TemplateDesc: AWS Windows Node Nov 29, 2019 1:37:47 PM INFO hudson.plugins.ec2.EC2Cloud getNewOrExistingAvailableSlaveSlaveTemplate{ami='ami-0266c7b78a0d89503', labels='c42xlarge'}. Cannot provision - no capacity for instances: 0 Nov 29, 2019 1:37:47 PM WARNING hudson.plugins.ec2.EC2Cloud provisionCan't raise nodes for SlaveTemplate{ami='ami-0266c7b78a0d89503', labels='c42xlarge'}
When trying to manually provision additional nodes via [http://localhost:8080/cloud/ec2-Itiviti%20AWS/provision:
|http://localhost:8080/cloud/ec2-Itiviti%20AWS/provision]
I get the following:
java.lang.Exception: Cloud or AMI instance cap would be exceeded for: AWS Windows Node at org.kohsuke.stapler.HttpResponses.error(HttpResponses.java:83) at hudson.plugins.ec2.EC2Cloud.doProvision(EC2Cloud.java:356) at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627) at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145) at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:535) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878) at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:280) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:676) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:225) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:92) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:99) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1700) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1667) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:505) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804) at java.lang.Thread.run(Thread.java:748)
Any help is much appreciated!
