[JIRA] (JENKINS-38556) Jenkins spawns infinite windows spot instances on aws.

8 views
Skip to first unread message

tyler@monkeypox.org (JIRA)

unread,
Sep 27, 2016, 6:35:03 PM9/27/16
to jenkinsc...@googlegroups.com
R. Tyler Croy moved an issue
 
Jenkins / Bug JENKINS-38556
Jenkins spawns infinite windows spot instances on aws.
Change By: R. Tyler Croy
Project: Infrastructure Jenkins
Key: INFRA JENKINS - 934 38556
Workflow: classic default workflow JNJira + In-Review
Component/s: ec2-plugin
Component/s: ci
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

sbrain@imagus.com.au (JIRA)

unread,
Sep 28, 2016, 8:50:02 PM9/28/16
to jenkinsc...@googlegroups.com
Steve Brain commented on Bug JENKINS-38556
 
Re: Jenkins spawns infinite windows spot instances on aws.

Same problem,

I updated to the latest aws-java-sdk 1.11.37 from 1.10.50 library and restarted my Jenkins instance, and noticed no nodes were being created. I looked at my ec2 page and had the maximum number of allowed machines running or starting up. I terminated these and it filled up again.

I immediately undid the upgrade and it started working correctly.

The jenkins log was filled with these

Sep 29, 2016 12:01:08 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting to provision slave from template hudson.plugins.ec2.SlaveTemplate@1a01b2a0 needed by excess workload of 1 units of label 'cxx'
Considering launching ami-63112100 for template Linux AMI
Sep 29, 2016 12:01:10 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Considering launching ami-63112100 for template Linux AMI
Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Stop
Sep 29, 2016 12:01:10 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Stop
Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [

{Name: image-id,Values: [ami-63112100]}

,

{Name: instance.group-name,Values: [JenkinsSecurity]}

,

{Name: key-name,Values: [Jenkins_keys]}

,

{Name: instance-type,Values: [c3.large]}

,

{Name: tag:Name,Values: [Centos Jenkins Slave CXX]}

,

{Name: tag:Jenkins,Values: [Slave]}

],}
Sep 29, 2016 12:01:11 AM hudson.triggers.SafeTimerTask run
SEVERE: Timer task hudson.slaves.NodeProvisioner$NodeProvisionerInvoker@459472fa failed
java.lang.NoSuchMethodError: com.amazonaws.services.ec2.AmazonEC2.createTags(Lcom/amazonaws/services/ec2/model/CreateTagsRequest;)V
at hudson.plugins.ec2.SlaveTemplate.updateRemoteTags(SlaveTemplate.java:882)
at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:585)
at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:377)
at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:515)
at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:532)
at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:701)
at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:307)
at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:60)
at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:798)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:50)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Sep 29, 2016 12:01:18 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting to provision slave from template hudson.plugins.ec2.SlaveTemplate@1a01b2a0 needed by excess workload of 1 units of label 'cxx'
Considering launching ami-63112100 for template Linux AMI
Sep 29, 2016 12:01:19 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Considering launching ami-63112100 for template Linux AMI
Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Stop
Sep 29, 2016 12:01:19 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Stop
Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [

{Name: image-id,Values: [ami-63112100]}

,

{Name: instance.group-name,Values: [JenkinsSecurity]}

,

{Name: key-name,Values: [Jenkins_keys]}

,

{Name: instance-type,Values: [c3.large]}

,

{Name: tag:Name,Values: [Centos Jenkins Slave CXX]}

,

{Name: tag:Jenkins,Values: [Slave]}

],}
Sep 29, 2016 12:01:20 AM hudson.triggers.SafeTimerTask run
SEVERE: Timer task hudson.slaves.NodeProvisioner$NodeProvisionerInvoker@459472fa failed
java.lang.NoSuchMethodError: com.amazonaws.services.ec2.AmazonEC2.createTags(Lcom/amazonaws/services/ec2/model/CreateTagsRequest;)V
at hudson.plugins.ec2.SlaveTemplate.updateRemoteTags(SlaveTemplate.java:882)
at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:585)
at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:377)
at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:515)
at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:532)
at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:701)
at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:307)
at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:60)
at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:798)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:50)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

sbrain@imagus.com.au (JIRA)

unread,
Sep 28, 2016, 9:00:01 PM9/28/16
to jenkinsc...@googlegroups.com
Steve Brain updated an issue
 
Change By: Steve Brain
Component/s: aws-java-sdk-plugin

me@joshma.com (JIRA)

unread,
Sep 30, 2016, 7:31:01 PM9/30/16
to jenkinsc...@googlegroups.com
Joshua Ma commented on Bug JENKINS-38556
 
Re: Jenkins spawns infinite windows spot instances on aws.

We're seeing the same thing with Linux non-spot instances. 80+ nodes launched (all missing tags, weirdly) - I terminated them at night, only to have them come back the next day. (It might've come back immediately, I just didn't check.) Would be great to have this bug be investigated - it's going to accidentally cost a lot of people a lot of money.

me@joshma.com (JIRA)

unread,
Sep 30, 2016, 7:31:02 PM9/30/16
to jenkinsc...@googlegroups.com
Joshua Ma updated an issue
 
Change By: Joshua Ma
Priority: Major Critical

me@joshma.com (JIRA)

unread,
Sep 30, 2016, 7:34:01 PM9/30/16
to jenkinsc...@googlegroups.com
 
Re: Jenkins spawns infinite windows spot instances on aws.

(Looks like JENKINS-38481 is the issue I'm seeing and is likely the same issue as the original poster.)

dbeck@cloudbees.com (JIRA)

unread,
Oct 14, 2016, 1:47:04 AM10/14/16
to jenkinsc...@googlegroups.com
Daniel Beck resolved as Duplicate
 

Amazon EC2 plugin
Allow Jenkins to start slaves on EC2 or Eucalyptus on demand, and kill them as they get unused.
1.35
Uninstall

Amazon Web Services SDK
This plugin provides AWS SDK for Java for other plugins.
1.11.37
DOWNGRADE TO 1.10.50
Uninstall

Fixed in EC2 Plugin 1.36

Change By: Daniel Beck
Status: Open Resolved
Resolution: Duplicate
Reply all
Reply to author
Forward
0 new messages