[JIRA] (JENKINS-60330) EC2 number of instances capped at 1 even though Instance Cap is greater

19 views
Skip to first unread message

cimpoiesgeorge@gmail.com (JIRA)

unread,
Nov 29, 2019, 7:39:04 AM11/29/19
to jenkinsc...@googlegroups.com
George Cimpoies created an issue
 
Jenkins / Bug JENKINS-60330
EC2 number of instances capped at 1 even though Instance Cap is greater
Issue Type: Bug Bug
Assignee: FABRIZIO MANFREDI
Attachments: image-2019-11-29-14-34-52-692.png, image-2019-11-29-14-35-28-900.png, image-2019-11-29-14-36-49-367.png, image-2019-11-29-14-36-54-791.png, image-2019-11-29-14-37-36-917.png
Components: ec2-plugin
Created: 2019-11-29 12:38
Environment: Jenkins 2.187
AWS EC2 1.47
Priority: Blocker Blocker
Reporter: George Cimpoies

I'm trying to provision AWS EC2 agents as build slaves for our Jenkins master, whenever a job matching the ami labels (dedined in Jenkins config) enters the build queue.

 

Cloud config:

All the builds that I schedule will wait for the existing AWS node to finish building, even though the Instance Cap is set to 30:

The job is configured to run concurrent builds:

In the Jenkins logs I see this recurrently:

 

Planned 0 new nodes
Nov 29, 2019 1:37:37 PM FINE hudson.plugins.ec2.NoDelayProvisionerStrategy applyAfter provisioning, available capacity=0, currentDemand=1
Nov 29, 2019 1:37:37 PM FINE hudson.plugins.ec2.NoDelayProvisionerStrategy applyProvisioning not complete, consulting remaining strategies
Nov 29, 2019 1:37:37 PM INFO hudson.plugins.ec2.EC2Cloud provisionSlaveTemplate{ami='ami-0266c7b78a0d89503', labels='c42xlarge'}. Attempting to provision slave needed by excess workload of 1 units
Nov 29, 2019 1:37:37 PM FINE hudson.plugins.ec2.EC2Cloud countCurrentEC2SlavesCounting current slaves:  All AMIS Jenkins Server: http://deb-jenkins-prd.ullink.lan/
Nov 29, 2019 1:37:37 PM FINE hudson.plugins.ec2.EC2Cloud countCurrentEC2SlavesExisting instance found: i-01e4a3e64c7dac1e7 AMI: ami-0266c7b78a0d89503 Jenkins Server: http://deb-jenkins-prd.ullink.lan/
Nov 29, 2019 1:37:37 PM FINE hudson.plugins.ec2.EC2Cloud countCurrentEC2SlavesCounting current slaves:  AMI: ami-0266c7b78a0d89503 TemplateDesc: AWS Windows Node Jenkins Server: http://deb-jenkins-prd.ullink.lan/
Nov 29, 2019 1:37:37 PM FINE hudson.plugins.ec2.EC2Cloud countCurrentEC2SlavesExisting instance found: i-01e4a3e64c7dac1e7 AMI: ami-0266c7b78a0d89503 Template: AWS Windows Node Jenkins Server: http://deb-jenkins-prd.ullink.lan/
Nov 29, 2019 1:37:37 PM FINE hudson.plugins.ec2.EC2Cloud getPossibleNewSlavesCountAvailable Total Slaves: 0 Available AMI slaves: 29 AMI: ami-0266c7b78a0d89503 TemplateDesc: AWS Windows Node
Nov 29, 2019 1:37:37 PM INFO hudson.plugins.ec2.EC2Cloud getNewOrExistingAvailableSlaveSlaveTemplate{ami='ami-0266c7b78a0d89503', labels='c42xlarge'}. Cannot provision - no capacity for instances: 0
Nov 29, 2019 1:37:37 PM WARNING hudson.plugins.ec2.EC2Cloud provisionCan't raise nodes for SlaveTemplate{ami='ami-0266c7b78a0d89503', labels='c42xlarge'}
Nov 29, 2019 1:37:46 PM FINE hudson.plugins.ec2.NoDelayProvisionerStrategy applyAvailable capacity=0, currentDemand=1
Nov 29, 2019 1:37:47 PM INFO hudson.plugins.ec2.EC2Cloud provisionSlaveTemplate{ami='ami-0266c7b78a0d89503', labels='c42xlarge'}. Attempting to provision slave needed by excess workload of 1 units
Nov 29, 2019 1:37:47 PM FINE hudson.plugins.ec2.EC2Cloud countCurrentEC2SlavesCounting current slaves:  All AMIS Jenkins Server: http://deb-jenkins-prd.ullink.lan/
Nov 29, 2019 1:37:47 PM FINE hudson.plugins.ec2.EC2Cloud countCurrentEC2SlavesExisting instance found: i-01e4a3e64c7dac1e7 AMI: ami-0266c7b78a0d89503 Jenkins Server: http://deb-jenkins-prd.ullink.lan/
Nov 29, 2019 1:37:47 PM FINE hudson.plugins.ec2.EC2Cloud countCurrentEC2SlavesCounting current slaves:  AMI: ami-0266c7b78a0d89503 TemplateDesc: AWS Windows Node Jenkins Server: http://deb-jenkins-prd.ullink.lan/
Nov 29, 2019 1:37:47 PM FINE hudson.plugins.ec2.EC2Cloud countCurrentEC2SlavesExisting instance found: i-01e4a3e64c7dac1e7 AMI: ami-0266c7b78a0d89503 Template: AWS Windows Node Jenkins Server: http://deb-jenkins-prd.ullink.lan/
Nov 29, 2019 1:37:47 PM FINE hudson.plugins.ec2.EC2Cloud getPossibleNewSlavesCountAvailable Total Slaves: 0 Available AMI slaves: 29 AMI: ami-0266c7b78a0d89503 TemplateDesc: AWS Windows Node
Nov 29, 2019 1:37:47 PM INFO hudson.plugins.ec2.EC2Cloud getNewOrExistingAvailableSlaveSlaveTemplate{ami='ami-0266c7b78a0d89503', labels='c42xlarge'}. Cannot provision - no capacity for instances: 0
Nov 29, 2019 1:37:47 PM WARNING hudson.plugins.ec2.EC2Cloud provisionCan't raise nodes for SlaveTemplate{ami='ami-0266c7b78a0d89503', labels='c42xlarge'}

Any help is much appreciated!
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

cimpoiesgeorge@gmail.com (JIRA)

unread,
Nov 29, 2019, 7:40:03 AM11/29/19
to jenkinsc...@googlegroups.com
George Cimpoies updated an issue
Change By: George Cimpoies
I'm trying to provision AWS EC2 agents as build slaves for our Jenkins master, whenever a job matching the ami labels ( dedined defined in Jenkins config) enters the build queue.

 

Cloud config:

!image-2019-11-29-14-35-28-900.png!


All the builds that I schedule will wait for the existing AWS node to finish building, even though the Instance Cap is set to 30:

!image-2019-11-29-14-36-54-791.png!


The job is configured to run concurrent builds:

!image-2019-11-29-14-37-36-917.png!


In the Jenkins logs I see this recurrently:

 
{code:java}

Planned 0 new nodes
Nov 29, 2019 1:37:37 PM FINE hudson.plugins.ec2.NoDelayProvisionerStrategy applyAfter provisioning, available capacity=0, currentDemand=1
Nov 29, 2019 1:37:37 PM FINE hudson.plugins.ec2.NoDelayProvisionerStrategy applyProvisioning not complete, consulting remaining strategies
Nov 29, 2019 1:37:37 PM INFO hudson.plugins.ec2.EC2Cloud provisionSlaveTemplate{ami='ami-0266c7b78a0d89503', labels='c42xlarge'}. Attempting to provision slave needed by excess workload of 1 units
Nov 29, 2019 1:37:37 PM FINE hudson.plugins.ec2.EC2Cloud countCurrentEC2SlavesCounting current slaves:  All AMIS Jenkins Server: http://deb-jenkins-prd.ullink.lan/
Nov 29, 2019 1:37:37 PM FINE hudson.plugins.ec2.EC2Cloud countCurrentEC2SlavesExisting instance found: i-01e4a3e64c7dac1e7 AMI: ami-0266c7b78a0d89503 Jenkins Server: http://deb-jenkins-prd.ullink.lan/
Nov 29, 2019 1:37:37 PM FINE hudson.plugins.ec2.EC2Cloud countCurrentEC2SlavesCounting current slaves:  AMI: ami-0266c7b78a0d89503 TemplateDesc: AWS Windows Node Jenkins Server: http://deb-jenkins-prd.ullink.lan/
Nov 29, 2019 1:37:37 PM FINE hudson.plugins.ec2.EC2Cloud countCurrentEC2SlavesExisting instance found: i-01e4a3e64c7dac1e7 AMI: ami-0266c7b78a0d89503 Template: AWS Windows Node Jenkins Server: http://deb-jenkins-prd.ullink.lan/
Nov 29, 2019 1:37:37 PM FINE hudson.plugins.ec2.EC2Cloud getPossibleNewSlavesCountAvailable Total Slaves: 0 Available AMI slaves: 29 AMI: ami-0266c7b78a0d89503 TemplateDesc: AWS Windows Node
Nov 29, 2019 1:37:37 PM INFO hudson.plugins.ec2.EC2Cloud getNewOrExistingAvailableSlaveSlaveTemplate{ami='ami-0266c7b78a0d89503', labels='c42xlarge'}. Cannot provision - no capacity for instances: 0
Nov 29, 2019 1:37:37 PM WARNING hudson.plugins.ec2.EC2Cloud provisionCan't raise nodes for SlaveTemplate{ami='ami-0266c7b78a0d89503', labels='c42xlarge'}
Nov 29, 2019 1:37:46 PM FINE hudson.plugins.ec2.NoDelayProvisionerStrategy applyAvailable capacity=0, currentDemand=1
Nov 29, 2019 1:37:47 PM INFO hudson.plugins.ec2.EC2Cloud provisionSlaveTemplate{ami='ami-0266c7b78a0d89503', labels='c42xlarge'}. Attempting to provision slave needed by excess workload of 1 units
Nov 29, 2019 1:37:47 PM FINE hudson.plugins.ec2.EC2Cloud countCurrentEC2SlavesCounting current slaves:  All AMIS Jenkins Server: http://deb-jenkins-prd.ullink.lan/
Nov 29, 2019 1:37:47 PM FINE hudson.plugins.ec2.EC2Cloud countCurrentEC2SlavesExisting instance found: i-01e4a3e64c7dac1e7 AMI: ami-0266c7b78a0d89503 Jenkins Server: http://deb-jenkins-prd.ullink.lan/
Nov 29, 2019 1:37:47 PM FINE hudson.plugins.ec2.EC2Cloud countCurrentEC2SlavesCounting current slaves:  AMI: ami-0266c7b78a0d89503 TemplateDesc: AWS Windows Node Jenkins Server: http://deb-jenkins-prd.ullink.lan/
Nov 29, 2019 1:37:47 PM FINE hudson.plugins.ec2.EC2Cloud countCurrentEC2SlavesExisting instance found: i-01e4a3e64c7dac1e7 AMI: ami-0266c7b78a0d89503 Template: AWS Windows Node Jenkins Server: http://deb-jenkins-prd.ullink.lan/
Nov 29, 2019 1:37:47 PM FINE hudson.plugins.ec2.EC2Cloud getPossibleNewSlavesCountAvailable Total Slaves: 0 Available AMI slaves: 29 AMI: ami-0266c7b78a0d89503 TemplateDesc: AWS Windows Node
Nov 29, 2019 1:37:47 PM INFO hudson.plugins.ec2.EC2Cloud getNewOrExistingAvailableSlaveSlaveTemplate{ami='ami-0266c7b78a0d89503', labels='c42xlarge'}. Cannot provision - no capacity for instances: 0
Nov 29, 2019 1:37:47 PM WARNING hudson.plugins.ec2.EC2Cloud provisionCan't raise nodes for SlaveTemplate{ami='ami-0266c7b78a0d89503', labels='c42xlarge'}
{code}

Any help is much appreciated!

cimpoiesgeorge@gmail.com (JIRA)

unread,
Nov 29, 2019, 8:01:02 AM11/29/19
to jenkinsc...@googlegroups.com
George Cimpoies updated an issue
I'm trying to provision AWS EC2 agents as build slaves for our Jenkins master, whenever a job matching the ami labels (defined in Jenkins config) enters the build queue.

 

Cloud config:

!image-2019-11-29-14-35-28-900.png!

All the builds that I schedule will wait for the existing AWS node to finish building, even though the Instance Cap is set to 30 (I've tried to tie multiple jobs to the same label and they all wait for the one single node to finish building) :

cimpoiesgeorge@gmail.com (JIRA)

unread,
Nov 29, 2019, 8:14:02 AM11/29/19
to jenkinsc...@googlegroups.com
George Cimpoies updated an issue
Change By: George Cimpoies
Environment: Jenkins 2.187
AWS EC2 1.47 (https://github.com/jenkinsci/ec2-plugin/pull/410)

cimpoiesgeorge@gmail.com (JIRA)

unread,
Nov 29, 2019, 9:10:03 AM11/29/19
to jenkinsc...@googlegroups.com
George Cimpoies updated an issue
Change By: George Cimpoies
Attachment: image-2019-11-29-16-09-13-234.png

cimpoiesgeorge@gmail.com (JIRA)

unread,
Nov 29, 2019, 9:10:04 AM11/29/19
to jenkinsc...@googlegroups.com
George Cimpoies updated an issue
When trying to manually provision additional nodes via [http://localhost:8080/cloud/ec2-Itiviti%20AWS/provision:
!image-2019-11-29-16-09-13-234.png!|http://localhost:8080/cloud/ec2-Itiviti%20AWS/provision]

I get the following: 
{code:java}
java.lang.Exception: Cloud or AMI instance cap would be exceeded for: AWS Windows Node
at org.kohsuke.stapler.HttpResponses.error(HttpResponses.java:83)
at hudson.plugins.ec2.EC2Cloud.doProvision(EC2Cloud.java:356)
at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)
at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408)
at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212)
at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145)
at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:535)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)
at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:280)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:676)
at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:225)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:92)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:99)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1700)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1667)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:505)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804)
at java.lang.Thread.run(Thread.java:748)
{code}
Any help is much appreciated!

cimpoiesgeorge@gmail.com (JIRA)

unread,
Jan 7, 2020, 3:35:03 AM1/7/20
to jenkinsc...@googlegroups.com
George Cimpoies updated an issue
Change By: George Cimpoies
Environment: Jenkins 2.187
AWS EC2 1.47 (https://github.com/jenkinsci/ec2-plugin/pull/410)

cimpoiesgeorge@gmail.com (JIRA)

unread,
Jan 7, 2020, 7:11:02 AM1/7/20
to jenkinsc...@googlegroups.com
Change By: George Cimpoies
Status: Fixed but Unreleased Resolved

cimpoiesgeorge@gmail.com (JIRA)

unread,
Jan 7, 2020, 7:11:02 AM1/7/20
to jenkinsc...@googlegroups.com
 

Although AMI instance cap was set to 10, global instance cap was set to 1
Change By: George Cimpoies
Status: Open Fixed but Unreleased
Assignee: FABRIZIO MANFREDI George Cimpoies
Resolution: Not A Defect
Reply all
Reply to author
Forward
0 new messages