[JIRA] (JENKINS-36516) EC2 plugin: Plugin fails to automatically launch slaves to meet demand

76 views
Skip to first unread message

chin.godawita@me.com (JIRA)

unread,
Jul 8, 2016, 2:03:02 AM7/8/16
to jenkinsc...@googlegroups.com
Chin Godawita created an issue
 
Jenkins / Bug JENKINS-36516
EC2 plugin: Plugin fails to automatically launch slaves to meet demand
Issue Type: Bug Bug
Assignee: Francis Upton
Components: ec2-plugin
Created: 2016/Jul/08 6:02 AM
Environment: Jenkins v2.12 / Ubuntu 14.04.4 x86_64 / Oracle Java 1.80_91-b14
Priority: Minor Minor
Reporter: Chin Godawita

Hi,

We've recently started seeing a problem auto-launching EC2 build slaves. The plugin detects that it needs to launch a slave to meet demand and that no current instances are runnign - it just doesn't launch anything.

Log (some info sanitised):

Jul 08, 2016 3:47:34 PM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting to provision slave from template hudson.plugins.ec2.SlaveTemplate@6a756eca needed by excess workload of 1 units of label 'null'
Jul 08, 2016 3:47:34 PM hudson.plugins.ec2.EC2Cloud provision
WARNING: Label is null - can't calculate how many executors slave will have. Using 2 number of executors
Considering launching ami-6c14310f for template Remote Ubuntu 14.04 Docker Machine
Jul 08, 2016 3:47:34 PM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Considering launching ami-6c14310f for template Remote Ubuntu 14.04 Docker Machine
Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Jul 08, 2016 3:47:34 PM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-6c14310f]}, {Name: subnet-id,Values: [subnet-XXXXX]}, {Name: instance.group-id,Values: [sg-XXXXX]}, {Name: key-name,Values: [XXXXX]}, {Name: instance-type,Values: [t2.small]}, {Name: tag:Name,Values: [Jenkins Build Slave]}],}
No existing instance found - but cannot create new instance
Jul 08, 2016 3:47:34 PM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision - finished, excess workload: 1
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

chin.godawita@me.com (JIRA)

unread,
Jul 8, 2016, 2:04:01 AM7/8/16
to jenkinsc...@googlegroups.com

chin.godawita@me.com (JIRA)

unread,
Jul 8, 2016, 2:05:02 AM7/8/16
to jenkinsc...@googlegroups.com
Chin Godawita updated an issue
Hi,

We've recently started seeing a problem auto-launching EC2 build slaves. The plugin detects that it needs to launch a slave to meet demand and that no current instances are runnign - it just doesn't launch anything.

Manual provisioning works.

Log (some info sanitised):
{noformat}

Jul 08, 2016 3:47:34 PM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting to provision slave from template hudson.plugins.ec2.SlaveTemplate@6a756eca needed by excess workload of 1 units of label 'null'
Jul 08, 2016 3:47:34 PM hudson.plugins.ec2.EC2Cloud provision
WARNING: Label is null - can't calculate how many executors slave will have. Using 2 number of executors
Considering launching ami-6c14310f for template Remote Ubuntu 14.04 Docker Machine
Jul 08, 2016 3:47:34 PM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Considering launching ami-6c14310f for template Remote Ubuntu 14.04 Docker Machine
Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Jul 08, 2016 3:47:34 PM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-6c14310f]}, {Name: subnet-id,Values: [subnet-XXXXX]}, {Name: instance.group-id,Values: [sg-XXXXX]}, {Name: key-name,Values: [XXXXX]}, {Name: instance-type,Values: [t2.small]}, {Name: tag:Name,Values: [Jenkins Build Slave]}],}
No existing instance found - but cannot create new instance
Jul 08, 2016 3:47:34 PM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision - finished, excess workload: 1
{noformat}

francisu@gmail.com (JIRA)

unread,
Jul 8, 2016, 10:59:02 AM7/8/16
to jenkinsc...@googlegroups.com

chin.godawita@me.com (JIRA)

unread,
Jul 10, 2016, 7:24:01 PM7/10/16
to jenkinsc...@googlegroups.com

It was happening on 1.33 but upgrading to 1.35 didn't fix it.

kjayaram@asmltd.com (JIRA)

unread,
Jul 13, 2016, 2:19:01 AM7/13/16
to jenkinsc...@googlegroups.com

kjayaram@asmltd.com (JIRA)

unread,
Jul 13, 2016, 2:20:04 AM7/13/16
to jenkinsc...@googlegroups.com
karthic jayaraman commented on Bug JENKINS-36516
 
Re: EC2 plugin: Plugin fails to automatically launch slaves to meet demand

I've observed the same issue in the latest Jenkins Version (2.10).

No existing instance found - but cannot create new instance
Jul 13, 2016 6:05:33 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision - finished, excess workload: 1
Jul 13, 2016 6:05:41 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision, excess workload: 1
Launching ami-e8e952ff for template GSC Cloud Slave
Jul 13, 2016 6:05:42 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Launching ami-e8e952ff for template GSC Cloud Slave
Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Jul 13, 2016 6:05:42 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsa...@fourv.com"]}],}
No existing instance found - but cannot create new instance
Jul 13, 2016 6:05:43 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision - finished, excess workload: 1

kjayaram@asmltd.com (JIRA)

unread,
Jul 13, 2016, 2:30:01 AM7/13/16
to jenkinsc...@googlegroups.com
karthic jayaraman edited a comment on Bug JENKINS-36516
I've observed the same issue in the latest Jenkins Version (2.10).

Manual provisioning works.


{noformat}

No existing instance found - but cannot create new instance
Jul 13, 2016 6:05:33 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision - finished, excess workload: 1
Jul 13, 2016 6:05:41 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision, excess workload: 1
Launching ami-e8e952ff for template GSC Cloud Slave
Jul 13, 2016 6:05:42 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Launching ami-e8e952ff for template GSC Cloud Slave
Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Jul 13, 2016 6:05:42 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsa...@fourv.com"]}],}
No existing instance found - but cannot create new instance
Jul 13, 2016 6:05:43 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision - finished, excess workload: 1
{noformat}

francisu@gmail.com (JIRA)

unread,
Jul 13, 2016, 2:33:02 AM7/13/16
to jenkinsc...@googlegroups.com

Can you try increasing the log level so we get the debug/fine level of logging?

kjayaram@asmltd.com (JIRA)

unread,
Jul 13, 2016, 2:42:03 AM7/13/16
to jenkinsc...@googlegroups.com

Here you go Francis Upton

INFO: Finished EC2 alive slaves monitor. 402 ms
Jul 13, 2016 1:45:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started EC2 alive slaves monitor
Jul 13, 2016 1:45:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished EC2 alive slaves monitor. 229 ms
Jul 13, 2016 1:55:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started EC2 alive slaves monitor
Jul 13, 2016 1:55:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished EC2 alive slaves monitor. 184 ms
Jul 13, 2016 2:05:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started EC2 alive slaves monitor
Jul 13, 2016 2:05:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished EC2 alive slaves monitor. 162 ms
Jul 13, 2016 2:15:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started EC2 alive slaves monitor
Jul 13, 2016 2:15:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished EC2 alive slaves monitor. 200 ms
Jul 13, 2016 2:25:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started EC2 alive slaves monitor
Jul 13, 2016 2:25:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished EC2 alive slaves monitor. 175 ms
Jul 13, 2016 2:35:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started EC2 alive slaves monitor
Jul 13, 2016 2:35:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished EC2 alive slaves monitor. 234 ms
Jul 13, 2016 2:45:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started EC2 alive slaves monitor
Jul 13, 2016 2:45:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished EC2 alive slaves monitor. 182 ms
Jul 13, 2016 2:55:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started EC2 alive slaves monitor
Jul 13, 2016 2:55:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished EC2 alive slaves monitor. 185 ms
Jul 13, 2016 3:05:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started EC2 alive slaves monitor
Jul 13, 2016 3:05:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished EC2 alive slaves monitor. 155 ms
Jul 13, 2016 3:15:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started EC2 alive slaves monitor
Jul 13, 2016 3:15:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished EC2 alive slaves monitor. 138 ms
Jul 13, 2016 3:25:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started EC2 alive slaves monitor
Jul 13, 2016 3:25:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished EC2 alive slaves monitor. 163 ms
Jul 13, 2016 3:35:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started EC2 alive slaves monitor
Jul 13, 2016 3:35:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished EC2 alive slaves monitor. 167 ms
Jul 13, 2016 3:45:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started EC2 alive slaves monitor
Jul 13, 2016 3:45:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished EC2 alive slaves monitor. 166 ms
Jul 13, 2016 3:55:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started EC2 alive slaves monitor
Jul 13, 2016 3:55:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished EC2 alive slaves monitor. 169 ms
Jul 13, 2016 4:05:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started EC2 alive slaves monitor
Jul 13, 2016 4:05:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished EC2 alive slaves monitor. 165 ms
Jul 13, 2016 4:15:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started EC2 alive slaves monitor
Jul 13, 2016 4:15:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished EC2 alive slaves monitor. 161 ms
Jul 13, 2016 4:25:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started EC2 alive slaves monitor
Jul 13, 2016 4:25:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished EC2 alive slaves monitor. 163 ms
Jul 13, 2016 4:35:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started EC2 alive slaves monitor
Jul 13, 2016 4:35:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished EC2 alive slaves monitor. 167 ms
Jul 13, 2016 4:45:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started EC2 alive slaves monitor
Jul 13, 2016 4:45:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished EC2 alive slaves monitor. 163 ms
Jul 13, 2016 4:55:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started EC2 alive slaves monitor
Jul 13, 2016 4:55:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished EC2 alive slaves monitor. 157 ms
Jul 13, 2016 5:05:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started EC2 alive slaves monitor
Jul 13, 2016 5:05:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished EC2 alive slaves monitor. 172 ms
Jul 13, 2016 5:15:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started EC2 alive slaves monitor
Jul 13, 2016 5:15:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished EC2 alive slaves monitor. 171 ms
Jul 13, 2016 5:25:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started EC2 alive slaves monitor
Jul 13, 2016 5:25:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished EC2 alive slaves monitor. 175 ms
Jul 13, 2016 5:35:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started EC2 alive slaves monitor
Jul 13, 2016 5:35:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished EC2 alive slaves monitor. 160 ms
Jul 13, 2016 5:39:14 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started Download metadata
Jul 13, 2016 5:39:16 AM hudson.model.UpdateSite updateData
INFO: Obtained the latest update center data file for UpdateSource default
Jul 13, 2016 5:39:16 AM hudson.model.DownloadService$Downloadable load
INFO: Obtained the updated data file for hudson.tasks.Maven.MavenInstaller
Jul 13, 2016 5:39:16 AM hudson.model.DownloadService$Downloadable load
INFO: Obtained the updated data file for hudson.tasks.Ant.AntInstaller
Jul 13, 2016 5:39:16 AM hudson.model.DownloadService$Downloadable load
INFO: Obtained the updated data file for hudson.plugins.gradle.GradleInstaller
Jul 13, 2016 5:39:16 AM hudson.model.DownloadService$Downloadable load
INFO: Obtained the updated data file for hudson.tools.JDKInstaller
Jul 13, 2016 5:39:16 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished Download metadata. 1,668 ms
Jul 13, 2016 5:45:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started EC2 alive slaves monitor
Jul 13, 2016 5:45:50 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished EC2 alive slaves monitor. 185 ms
Jul 13, 2016 5:49:08 AM hudson.plugins.ec2.EC2OndemandSlave terminate
INFO: Terminated EC2 instance (terminated): i-d81e5a5e
Jul 13, 2016 5:49:08 AM hudson.plugins.ec2.EC2OndemandSlave terminate
INFO: Removed EC2 instance from jenkins master: i-d81e5a5e
Jul 13, 2016 5:52:24 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision, excess workload: 1
Launching ami-e8e952ff for template GSC Cloud Slave
Jul 13, 2016 5:52:26 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Launching ami-e8e952ff for template GSC Cloud Slave
Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Jul 13, 2016 5:52:26 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsa...@fourv.com"]}],}
No existing instance found - but cannot create new instance
Jul 13, 2016 5:52:26 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision - finished, excess workload: 1
Jul 13, 2016 5:52:34 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision, excess workload: 1
Launching ami-e8e952ff for template GSC Cloud Slave
Jul 13, 2016 5:52:36 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Launching ami-e8e952ff for template GSC Cloud Slave
Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Jul 13, 2016 5:52:36 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsa...@fourv.com"]}],}
No existing instance found - but cannot create new instance
Jul 13, 2016 5:52:36 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision - finished, excess workload: 1
Jul 13, 2016 5:52:44 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision, excess workload: 1
Launching ami-e8e952ff for template GSC Cloud Slave
Jul 13, 2016 5:52:46 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Launching ami-e8e952ff for template GSC Cloud Slave
Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Jul 13, 2016 5:52:46 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsa...@fourv.com"]}],}
No existing instance found - but cannot create new instance
Jul 13, 2016 5:52:46 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision - finished, excess workload: 1
Jul 13, 2016 5:52:54 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision, excess workload: 1
Launching ami-e8e952ff for template GSC Cloud Slave
Jul 13, 2016 5:52:56 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Launching ami-e8e952ff for template GSC Cloud Slave
Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Jul 13, 2016 5:52:56 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsa...@fourv.com"]}],}
No existing instance found - but cannot create new instance
Jul 13, 2016 5:52:56 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision - finished, excess workload: 1
Jul 13, 2016 5:53:04 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision, excess workload: 1
Launching ami-e8e952ff for template GSC Cloud Slave
Jul 13, 2016 5:53:06 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Launching ami-e8e952ff for template GSC Cloud Slave
Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Jul 13, 2016 5:53:06 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsa...@fourv.com"]}],}
No existing instance found - but cannot create new instance
Jul 13, 2016 5:53:06 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision - finished, excess workload: 1
Jul 13, 2016 5:53:14 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision, excess workload: 1
Launching ami-e8e952ff for template GSC Cloud Slave
Jul 13, 2016 5:53:16 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Launching ami-e8e952ff for template GSC Cloud Slave
Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Jul 13, 2016 5:53:16 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsa...@fourv.com"]}],}
No existing instance found - but cannot create new instance
Jul 13, 2016 5:53:16 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision - finished, excess workload: 1
Jul 13, 2016 5:53:24 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision, excess workload: 1
Launching ami-e8e952ff for template GSC Cloud Slave
Jul 13, 2016 5:53:25 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Launching ami-e8e952ff for template GSC Cloud Slave
Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Jul 13, 2016 5:53:25 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsa...@fourv.com"]}],}
No existing instance found - but cannot create new instance
Jul 13, 2016 5:53:25 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision - finished, excess workload: 1
Jul 13, 2016 5:53:34 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision, excess workload: 1
Launching ami-e8e952ff for template GSC Cloud Slave
Jul 13, 2016 5:53:35 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Launching ami-e8e952ff for template GSC Cloud Slave
Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Jul 13, 2016 5:53:35 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsa...@fourv.com"]}],}
No existing instance found - but cannot create new instance
Jul 13, 2016 5:53:35 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision - finished, excess workload: 1
Jul 13, 2016 5:53:44 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision, excess workload: 1
Launching ami-e8e952ff for template GSC Cloud Slave
Jul 13, 2016 5:53:45 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Launching ami-e8e952ff for template GSC Cloud Slave
Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Jul 13, 2016 5:53:45 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsa...@fourv.com"]}],}
No existing instance found - but cannot create new instance
Jul 13, 2016 5:53:45 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision - finished, excess workload: 1
Jul 13, 2016 5:53:54 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision, excess workload: 1
Launching ami-e8e952ff for template GSC Cloud Slave
Jul 13, 2016 5:53:55 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Launching ami-e8e952ff for template GSC Cloud Slave
Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Jul 13, 2016 5:53:55 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsa...@fourv.com"]}],}
No existing instance found - but cannot create new instance
Jul 13, 2016 5:53:56 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision - finished, excess workload: 1
Jul 13, 2016 5:54:02 AM jenkins.model.Jenkins$26 run
INFO: Restart in 10 seconds
Jul 13, 2016 5:54:12 AM jenkins.model.Jenkins$26 run
SEVERE: Restarting VM as requested by anonymous
Jul 13, 2016 5:54:12 AM jenkins.model.Jenkins cleanUp
INFO: Stopping Jenkins
Jul 13, 2016 5:54:12 AM jenkins.model.Jenkins$21 onAttained
INFO: Started termination
Jul 13, 2016 5:54:12 AM jenkins.model.Jenkins$21 onAttained
INFO: Completed termination
Jul 13, 2016 5:54:12 AM jenkins.model.Jenkins _cleanUpDisconnectComputers
INFO: Starting node disconnection
Jul 13, 2016 5:54:13 AM jenkins.model.Jenkins _cleanUpShutdownPluginManager
INFO: Stopping plugin manager
Jul 13, 2016 5:54:13 AM jenkins.model.Jenkins _cleanUpPersistQueue
INFO: Persisting build queue
Jul 13, 2016 5:54:13 AM jenkins.model.Jenkins _cleanUpAwaitDisconnects
INFO: Waiting for node disconnection completion
Jul 13, 2016 5:54:13 AM jenkins.model.Jenkins cleanUp
INFO: Jenkins stopped
Running from: /usr/share/jenkins/jenkins.war
Jul 13, 2016 5:54:13 AM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: Logging initialized @380ms
Jul 13, 2016 5:54:13 AM winstone.Logger logInternal
INFO: Beginning extraction from war file
Jul 13, 2016 5:54:13 AM org.eclipse.jetty.util.log.JavaUtilLog warn
WARNING: Empty contextPath
Jul 13, 2016 5:54:13 AM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: jetty-9.2.z-SNAPSHOT
Jul 13, 2016 5:54:14 AM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
Jenkins home directory: /var/lib/jenkins found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
Jul 13, 2016 5:54:15 AM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: Started w.@1e125710{/,file:/var/cache/jenkins/war/,AVAILABLE}{/var/cache/jenkins/war}
Jul 13, 2016 5:54:15 AM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: Started ServerConnector@42eded9{HTTP/1.1}{0.0.0.0:8080}
Jul 13, 2016 5:54:15 AM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: Started @1689ms
Jul 13, 2016 5:54:15 AM winstone.Logger logInternal
INFO: Winstone Servlet Engine v2.0 running: controlPort=disabled
Jul 13, 2016 5:54:15 AM jenkins.InitReactorRunner$1 onAttained
INFO: Started initialization
Jul 13, 2016 5:54:15 AM jenkins.InitReactorRunner$1 onAttained
INFO: Listed all plugins
Jul 13, 2016 5:54:15 AM jenkins.bouncycastle.api.SecurityProviderInitializer addSecurityProvider
INFO: Initializing Bouncy Castle security provider.
Jul 13, 2016 5:54:16 AM jenkins.bouncycastle.api.SecurityProviderInitializer addSecurityProvider
INFO: Bouncy Castle security provider initialized.
Jul 13, 2016 5:54:20 AM jenkins.InitReactorRunner$1 onAttained
INFO: Prepared all plugins
Jul 13, 2016 5:54:20 AM hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1 error
INFO: Failed to instantiate optional component org.jenkinsci.plugins.workflow.steps.scm.GitStep$DescriptorImpl; skipping
Jul 13, 2016 5:54:21 AM jenkins.InitReactorRunner$1 onAttained
INFO: Started all plugins
Jul 13, 2016 5:54:21 AM jenkins.InitReactorRunner$1 onAttained
INFO: Augmented all extensions
Jul 13, 2016 5:54:21 AM hudson.slaves.SlaveComputer tryReconnect
INFO: Attempting to reconnect gscqascheduler113.fourvinternal.com
Jul 13, 2016 5:54:21 AM hudson.slaves.SlaveComputer tryReconnect
INFO: Attempting to reconnect gscqascheduler120.fourvinternal.com
Jul 13, 2016 5:54:21 AM hudson.slaves.SlaveComputer tryReconnect
INFO: Attempting to reconnect qa04scheduler.fourvinternal.com
Jul 13, 2016 5:54:21 AM jenkins.InitReactorRunner$1 onAttained
INFO: Loaded all jobs
Jul 13, 2016 5:54:22 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started Download metadata
Jul 13, 2016 5:54:22 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished Download metadata. 1 ms
Jul 13, 2016 5:54:22 AM org.jenkinsci.main.modules.sshd.SSHD start
INFO: Started SSHD at port 37928
Jul 13, 2016 5:54:22 AM jenkins.InitReactorRunner$1 onAttained
INFO: Completed initialization
Jul 13, 2016 5:54:22 AM hudson.WebAppMain$3 run
INFO: Jenkins is fully up and running
[07/13/16 05:54:24] SSH Launch of gscqascheduler120.fourvinternal.com on 10.187.27.222 completed in 2,567 ms
[07/13/16 05:54:26] SSH Launch of gscqascheduler113.fourvinternal.com on 10.187.29.0 completed in 4,673 ms
[07/13/16 05:54:26] SSH Launch of qa04scheduler.fourvinternal.com on 10.187.29.100 completed in 4,840 ms
Jul 13, 2016 5:56:01 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision, excess workload: 1
Launching ami-e8e952ff for template GSC Cloud Slave
Jul 13, 2016 5:56:04 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Launching ami-e8e952ff for template GSC Cloud Slave
Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Jul 13, 2016 5:56:04 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsa...@fourv.com"]}],}
No existing instance found - but cannot create new instance
Jul 13, 2016 5:56:04 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision - finished, excess workload: 1
Jul 13, 2016 5:56:11 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision, excess workload: 1
Launching ami-e8e952ff for template GSC Cloud Slave
Jul 13, 2016 5:56:12 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Launching ami-e8e952ff for template GSC Cloud Slave
Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Jul 13, 2016 5:56:12 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsa...@fourv.com"]}],}
No existing instance found - but cannot create new instance
Jul 13, 2016 5:56:12 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision - finished, excess workload: 1
Jul 13, 2016 5:56:21 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision, excess workload: 1
Launching ami-e8e952ff for template GSC Cloud Slave
Jul 13, 2016 5:56:22 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Launching ami-e8e952ff for template GSC Cloud Slave
Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Jul 13, 2016 5:56:22 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsa...@fourv.com"]}],}
No existing instance found - but cannot create new instance
Jul 13, 2016 5:56:22 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision - finished, excess workload: 1
Jul 13, 2016 5:56:31 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision, excess workload: 1
Launching ami-e8e952ff for template GSC Cloud Slave
Jul 13, 2016 5:56:32 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Launching ami-e8e952ff for template GSC Cloud Slave
Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Jul 13, 2016 5:56:32 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsa...@fourv.com"]}],}
No existing instance found - but cannot create new instance
Jul 13, 2016 5:56:32 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision - finished, excess workload: 1
Jul 13, 2016 5:56:41 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision, excess workload: 1
Launching ami-e8e952ff for template GSC Cloud Slave
Jul 13, 2016 5:56:42 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Launching ami-e8e952ff for template GSC Cloud Slave
Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Jul 13, 2016 5:56:42 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsa...@fourv.com"]}],}
No existing instance found - but cannot create new instance
Jul 13, 2016 5:56:43 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision - finished, excess workload: 1
Jul 13, 2016 5:56:51 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision, excess workload: 1
Launching ami-e8e952ff for template GSC Cloud Slave
Jul 13, 2016 5:56:52 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Launching ami-e8e952ff for template GSC Cloud Slave
Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Jul 13, 2016 5:56:52 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsa...@fourv.com"]}],}
No existing instance found - but cannot create new instance
Jul 13, 2016 5:56:52 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision - finished, excess workload: 1
Jul 13, 2016 5:57:01 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision, excess workload: 1
Launching ami-e8e952ff for template GSC Cloud Slave
Jul 13, 2016 5:57:03 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Launching ami-e8e952ff for template GSC Cloud Slave
Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Jul 13, 2016 5:57:03 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsa...@fourv.com"]}],}
No existing instance found - but cannot create new instance
Jul 13, 2016 5:57:03 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision - finished, excess workload: 1
Launching ami-e8e952ff for template GSC Cloud Slave
Jul 13, 2016 5:57:26 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Launching ami-e8e952ff for template GSC Cloud Slave
Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Jul 13, 2016 5:57:26 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsa...@fourv.com"]}],}

francisu@gmail.com (JIRA)

unread,
Jul 13, 2016, 3:43:01 AM7/13/16
to jenkinsc...@googlegroups.com

Thanks for the additional tracing (it's good to get it from the beginning), but I also need the logging level changed so we get the "fine" tracing as well (not just "info")

chin.godawita@me.com (JIRA)

unread,
Jul 27, 2016, 10:10:01 PM7/27/16
to jenkinsc...@googlegroups.com

I realise this is not much help but strangely enough, it started working for us again. No Jenkins/plugin upgrades have been done since the issue happened.

The immediate set of actions I did before it started working was:

  • To get around this issue, we'd provisioned (through the EC2 plugin) an always-running instance (with no idle timeout)
  • Today, because we'd run out of space, I deleted it
  • In the interim before I'd had a chance to , we had a build get triggered via a commit
  • This automatically brought up a new instance as it used to

I'm also trying to get a debug log across but it's a little difficult as I need to lock in a period of time where our master build server is inaccessible.

In the meantime, here's the log from when it started working:

Jul 28, 2016 11:43:26 AM hudson.plugins.ec2.EC2OndemandSlave terminate
INFO: Terminated EC2 instance (terminated): i-xxxxxx
Jul 28, 2016 11:43:26 AM hudson.plugins.ec2.EC2OndemandSlave terminate
INFO: Removed EC2 instance from jenkins master: i-xxxxxx
Jul 28, 2016 11:44:24 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting to provision slave from template hudson.plugins.ec2.SlaveTemplate@xxx needed by excess workload of 1 units of label 'null'
Jul 28, 2016 11:44:24 AM hudson.plugins.ec2.EC2Cloud provision
WARNING: Label is null - can't calculate how many executors slave will have. Using 2 number of executors
Considering launching ami-xxxxxx for template Remote Ubuntu 14.04 Docker Machine
Jul 28, 2016 11:44:24 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Considering launching ami-xxxxxx for template Remote Ubuntu 14.04 Docker Machine
Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Jul 28, 2016 11:44:24 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-xxxxxx]}, {Name: subnet-id,Values: [subnet-xxxxxx]}, {Name: instance.group-id,Values: [sg-xxxxxx]}, {Name: key-name,Values: [jenkins-slave]}, {Name: instance-type,Values: [t2.small]}, {Name: tag:Name,Values: [Jenkins Build Slave]}],}
No existing instance found - created new instance: {InstanceId: i-xxxxxx,ImageId: ami-xxxxxx,State: {Code: 0,Name: pending},PrivateDnsName: ip-x-x-x-x.ap-southeast-2.compute.internal,PublicDnsName: ,StateTransitionReason: ,KeyName: jenkins-slave,AmiLaunchIndex: 0,ProductCodes: [],InstanceType: t2.small,LaunchTime: Thu Jul 28 11:44:03 AEST 2016,Placement: {AvailabilityZone: ap-southeast-2a,GroupName: ,Tenancy: default,},Monitoring: {State: disabled},SubnetId: subnet-xxxxxx,VpcId: vpc-xxxxxx,PrivateIpAddress: x.x.x.x,StateReason: {Code: pending,Message: pending},Architecture: x86_64,RootDeviceType: ebs,RootDeviceName: /dev/sda1,BlockDeviceMappings: [],VirtualizationType: hvm,ClientToken: x-x-x-x-x,Tags: [{Key: jenkins_slave_type,Value: demand_Remote Ubuntu 14.04 Docker Machine}, {Key: Name,Value: Jenkins Build Slave}],SecurityGroups: [{GroupName: jenkins,GroupId: sg-xxxxxx}],SourceDestCheck: true,Hypervisor: xen,NetworkInterfaces: [{NetworkInterfaceId: eni-xxxxxx,SubnetId: subnet-xxxxxx,VpcId: vpc-xxxxxx,Description: ,OwnerId: xxxxxx,Status: in-use,MacAddress: xxx,PrivateIpAddress: x.x.x.x,PrivateDnsName: ip-x-x-x-x.ap-southeast-2.compute.internal,SourceDestCheck: true,Groups: [{GroupName: jenkins,GroupId: sg-xxxxxx}],Attachment: {AttachmentId: eni-attach-xxxxxx,DeviceIndex: 0,Status: attaching,AttachTime: Thu Jul 28 11:44:03 AEST 2016,DeleteOnTermination: true},PrivateIpAddresses: [{PrivateIpAddress: x.x.x.x,PrivateDnsName: ip-x-x-x-x.ap-southeast-2.compute.internal,Primary: true,}]}],EbsOptimized: false,}
Jul 28, 2016 11:44:26 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: No existing instance found - created new instance: {InstanceId: i-xxxxxx,ImageId: ami-xxxxxx,State: {Code: 0,Name: pending},PrivateDnsName: ip-x-x-x-x.ap-southeast-2.compute.internal,PublicDnsName: ,StateTransitionReason: ,KeyName: jenkins-slave,AmiLaunchIndex: 0,ProductCodes: [],InstanceType: t2.small,LaunchTime: Thu Jul 28 11:44:03 AEST 2016,Placement: {AvailabilityZone: ap-southeast-2a,GroupName: ,Tenancy: default,},Monitoring: {State: disabled},SubnetId: subnet-xxxxxx,VpcId: vpc-xxxxxx,PrivateIpAddress: x.x.x.x,StateReason: {Code: pending,Message: pending},Architecture: x86_64,RootDeviceType: ebs,RootDeviceName: /dev/sda1,BlockDeviceMappings: [],VirtualizationType: hvm,ClientToken: x-x-x-x-x,Tags: [{Key: jenkins_slave_type,Value: demand_Remote Ubuntu 14.04 Docker Machine}, {Key: Name,Value: Jenkins Build Slave}],SecurityGroups: [{GroupName: jenkins,GroupId: sg-xxxxxx}],SourceDestCheck: true,Hypervisor: xen,NetworkInterfaces: [{NetworkInterfaceId: eni-xxxxxx,SubnetId: subnet-xxxxxx,VpcId: vpc-xxxxxx,Description: ,OwnerId: xxxxxx,Status: in-use,MacAddress: xxx,PrivateIpAddress: x.x.x.x,PrivateDnsName: ip-x-x-x-x.ap-southeast-2.compute.internal,SourceDestCheck: true,Groups: [{GroupName: jenkins,GroupId: sg-xxxxxx}],Attachment: {AttachmentId: eni-attach-xxxxxx,DeviceIndex: 0,Status: attaching,AttachTime: Thu Jul 28 11:44:03 AEST 2016,DeleteOnTermination: true},PrivateIpAddresses: [{PrivateIpAddress: x.x.x.x,PrivateDnsName: ip-x-x-x-x.ap-southeast-2.compute.internal,Primary: true,}]}],EbsOptimized: false,}
Jul 28, 2016 11:44:26 AM hudson.plugins.ec2.EC2Cloud provision
INFO: We have now 1 computers
Jul 28, 2016 11:44:26 AM hudson.plugins.ec2.EC2RetentionStrategy start
INFO: Start requested for Remote Ubuntu 14.04 Docker Machine (i-xxxxxx)
Jul 28, 2016 11:44:26 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Added node named: Remote Ubuntu 14.04 Docker Machine (i-xxxxxx), We have now 2 computers
Jul 28, 2016 11:44:26 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Attempting provision - finished, excess workload: -1
Jul 28, 2016 11:44:26 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Remote Ubuntu 14.04 Docker Machine (ami-xxxxxx) from ec2-jenkins-user with 2 executors. Remaining excess workload: -1
Jul 28, 2016 11:44:38 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Started EC2 alive slaves monitor
Jul 28, 2016 11:44:38 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished EC2 alive slaves monitor. 55 ms

jmcginnis@ccpgames.com (JIRA)

unread,
Nov 2, 2016, 10:35:03 AM11/2/16
to jenkinsc...@googlegroups.com

I am seeing what appear to me to be the same issue on EC2 1.36. After adding an additional EC2 node to a previously working system, auto-re-launch of timed out instances doesn't occur. Manually launching will work. My logs look very similar.

vincent.brouillet@gmail.com (JIRA)

unread,
Nov 17, 2016, 11:54:01 PM11/17/16
to jenkinsc...@googlegroups.com

dstark75@gmail.com (JIRA)

unread,
Dec 12, 2016, 2:11:03 PM12/12/16
to jenkinsc...@googlegroups.com

I'm also experiencing this issue with 1.36. I'm using spot instances if that matters. Here's the exception I'm seeing:

Dec 12, 2016 7:02:19 PM hudson.plugins.ec2.EC2Cloud provision
WARNING: Exception during provisioning
com.amazonaws.SdkClientException: Unable to unmarshall response (Read timed out). Response Code: 200, Response Text: OK
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1468)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1151)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:964)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:676)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:650)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:633)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:601)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:583)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:447)
        at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:11396)
        at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:11372)
        at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:5542)
        at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:5554)
        at hudson.plugins.ec2.EC2Cloud.countCurrentEC2Slaves(EC2Cloud.java:363)
        at hudson.plugins.ec2.EC2Cloud.getPossibleNewSlavesCount(EC2Cloud.java:499)
        at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:518)
        at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:547)
        at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:701)
        at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:307)
        at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:60)
        at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:798)
        at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:50)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.ctc.wstx.exc.WstxIOException: Read timed out
        at com.ctc.wstx.sr.StreamScanner.throwFromIOE(StreamScanner.java:708)
        at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1086)
        at com.ctc.wstx.evt.WstxEventReader.peek(WstxEventReader.java:306)
        at com.amazonaws.transform.StaxUnmarshallerContext.nextEvent(StaxUnmarshallerContext.java:220)
        at com.amazonaws.services.ec2.model.transform.InstanceStaxUnmarshaller.unmarshall(InstanceStaxUnmarshaller.java:42)
        at com.amazonaws.services.ec2.model.transform.ReservationStaxUnmarshaller.unmarshall(ReservationStaxUnmarshaller.java:69)
        at com.amazonaws.services.ec2.model.transform.DescribeInstancesResultStaxUnmarshaller.unmarshall(DescribeInstancesResultStaxUnmarshaller.java:49)
        at com.amazonaws.services.ec2.model.transform.DescribeInstancesResultStaxUnmarshaller.unmarshall(DescribeInstancesResultStaxUnmarshaller.java:31)
        at com.amazonaws.http.StaxResponseHandler.handle(StaxResponseHandler.java:101)
        at com.amazonaws.http.StaxResponseHandler.handle(StaxResponseHandler.java:43)
        at com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:70)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1444)
        ... 28 more
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:152)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
        at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:554)
        at sun.security.ssl.InputRecord.read(InputRecord.java:509)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:944)
        at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:901)
        at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
        at org.apache.http.impl.conn.LoggingInputStream.read(LoggingInputStream.java:87)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
        at org.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:200)
        at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:191)
        at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137)
        at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
        at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:181)
        at java.io.FilterInputStream.read(FilterInputStream.java:107)
        at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:365)
        at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:110)
        at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84)
        at com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57)
        at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:992)
        at com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4628)
        at com.ctc.wstx.sr.BasicStreamReader.readCoalescedText(BasicStreamReader.java:4126)
        at com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3701)
        at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1023)
        ... 38 more

francisu@gmail.com (JIRA)

unread,
Mar 15, 2018, 3:02:02 AM3/15/18
to jenkinsc...@googlegroups.com

Please try 1.39 of the plugin, particularly if you are upgrading from 1.36 or earlier.

This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
Atlassian logo

bahdad@bigmir.net (JIRA)

unread,
Aug 8, 2018, 2:39:03 PM8/8/18
to jenkinsc...@googlegroups.com
Sasha Miroshnychenko assigned an issue to Sasha Miroshnychenko
 
Change By: Sasha Miroshnychenko
Assignee: Francis Upton Sasha Miroshnychenko
This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396)

bahdad@bigmir.net (JIRA)

unread,
Aug 8, 2018, 2:39:05 PM8/8/18
to jenkinsc...@googlegroups.com

ben.copeland@menlosecurity.com (JIRA)

unread,
Mar 22, 2019, 3:48:02 PM3/22/19
to jenkinsc...@googlegroups.com
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

raihaan.shouhell@autodesk.com (JIRA)

unread,
Sep 2, 2019, 2:47:03 AM9/2/19
to jenkinsc...@googlegroups.com

ovidiuc@gmail.com (JIRA)

unread,
Jan 18, 2020, 5:04:03 AM1/18/20
to jenkinsc...@googlegroups.com

still an issue with 1.48
I can see this log but the spotinst mentioned is never spawned, no other error:

```
Spot instance id in provision: sir-px3sj14h
SlaveTemplate

{ami='ami-17de70f54e3bad981', labels='windows'}

. Attempting provision finished, excess workload: 0
We have now 5 computers, waiting for 1 more
```

and then
```


Started EC2 alive slaves monitor

Finished EC2 alive slaves monitor. 112 ms
```

the build queue keeps having that one waiting build to pe pushed to a provisioned slave.

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

ovidiuc@gmail.com (JIRA)

unread,
Jan 18, 2020, 5:06:05 AM1/18/20
to jenkinsc...@googlegroups.com
ovi craciun edited a comment on Bug JENKINS-36516
still an issue with 1.48
I can see this log but the spotinst mentioned is never spawned, no other error:

```
Spot instance id in provision: sir-px3sj14h
SlaveTemplate{ami='ami-17de70f54e3bad981', labels='windows'}. Attempting provision finished, excess workload: 0
We have now 5 computers, waiting for 1 more
```

and then
```
Started EC2 alive slaves monitor
Finished EC2 alive slaves monitor. 112 ms
```

the build queue keeps having that one waiting build to pe pushed to a provisioned slave but the slave is never provisioned .

ovidiuc@gmail.com (JIRA)

unread,
Jan 18, 2020, 5:36:05 AM1/18/20
to jenkinsc...@googlegroups.com
ovi craciun edited a comment on Bug JENKINS-36516
still an issue with 1.48
I can see this log but the spotinst mentioned is never spawned, no other error:

```
Spot instance id in provision: sir-px3sj14h
SlaveTemplate{ami='ami-17de70f54e3bad981', labels='windows'}. Attempting provision finished, excess workload: 0
We have now 5 computers, waiting for 1 more
```

and then
```
Started EC2 alive slaves monitor
Finished EC2 alive slaves monitor. 112 ms
```

and
```
Available capacity=1, currentDemand=1
Provisioning completedFinished
```

however capacity is 0 cause there is a build in progress already.

and thus,
the build queue keeps having that one waiting build to pe pushed to a provisioned slave but the slave is never provisioned.

ovidiuc@gmail.com (JIRA)

unread,
Jan 18, 2020, 5:36:07 AM1/18/20
to jenkinsc...@googlegroups.com
ovi craciun edited a comment on Bug JENKINS-36516
still an issue with 1.48
I can see this log but the spotinst mentioned is never spawned, no other error:

```
Spot instance id in provision: sir-px3sj14h
SlaveTemplate{ami='ami-17de70f54e3bad981', labels='windows'}. Attempting provision finished, excess workload: 0
We have now 5 computers, waiting for 1 more
```

and then
```
Started EC2 alive slaves monitor
EC2 alive slaves monitor. 112 ms
```

and
```
Available capacity=1, currentDemand=1
Provisioning completedFinished
```

however capacity is 0 cause there is a build in progress already and one in the waiting queue .


and thus, the build queue keeps having that one waiting build to pe pushed to a provisioned slave but the slave is never provisioned.

ovidiuc@gmail.com (JIRA)

unread,
Jan 20, 2020, 6:14:08 AM1/20/20
to jenkinsc...@googlegroups.com
ovi craciun edited a comment on Bug JENKINS-36516
still an issue with 1.48
I can see this log but the spotinst mentioned is never spawned, no other error:

```
Spot instance id in provision: sir-px3sj14h
SlaveTemplate{ami='ami-17de70f54e3bad981', labels='windows'}. Attempting provision finished, excess workload: 0
We have now 5 computers, waiting for 1 more
```

and then
```
Started EC2 alive slaves monitor
EC2 alive slaves monitor. 112 ms
```

and
```
Available capacity=1, currentDemand=1
Provisioning completedFinished
```

however capacity is 0 cause there is a build in progress already and one in the waiting queue.

and thus, the build queue keeps having that one waiting build to pe pushed to a provisioned slave but the slave is never provisioned.


Later: even when there's no build in progress, I see this message
`Available capacity=1, currentDemand=1`
and a new slave is not provisioned when there's a build waiting in the build queue.

How can I get around this mess?
Reply all
Reply to author
Forward
0 new messages