[JIRA] (JENKINS-55639) ec2-plugin only attempts to retrieve spot instance using old id

14 views
Skip to first unread message

tommoyang@gmail.com (JIRA)

unread,
Jan 17, 2019, 12:21:02 AM1/17/19
to jenkinsc...@googlegroups.com
Tao Yang created an issue
 
Jenkins / Bug JENKINS-55639
ec2-plugin only attempts to retrieve spot instance using old id
Issue Type: Bug Bug
Assignee: FABRIZIO MANFREDI
Components: ec2-plugin
Created: 2019-01-17 05:20
Environment: Jenkins 2.150.2
ec2-plugin 1.42
Priority: Blocker Blocker
Reporter: Tao Yang

When Jenkins attempts to provision spot instances, after spot requests are created (incorrectly, as per JENKINS-55618) Jenkins will try to retrieve the spot instance based on the last working instance id from before the plugin was upgraded, instead of the instance id returned by the spot request.

 

INFO: Spot instance id in provision: sir-34si4rxjINFO: Spot instance id in provision: sir-34si4rxjJan 17, 2019 4:32:10 AM hudson.plugins.ec2.CloudHelper getInstanceINFO: Unexpected number of reservations reported by EC2 for instance id 'i-0b934d05d88272fc4', expected 1 result, found []. Instance seems to be dead.Jan 17, 2019 4:32:10 AM hudson.plugins.ec2.EC2Cloud provisionWARNING: SlaveTemplate{ami='ami-09009aeca3492b598', labels='agent linux'}. Exception during provisioningcom.amazonaws.AmazonClientException: Unexpected number of reservations reported by EC2 for instance id 'i-0b934d05d88272fc4', expected 1 result, found []. Instance seems to be dead. at hudson.plugins.ec2.CloudHelper.getInstance(CloudHelper.java:54) at hudson.plugins.ec2.CloudHelper.getInstanceWithRetry(CloudHelper.java:25) at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:499) at hudson.plugins.ec2.EC2AbstractSlave.<init>(EC2AbstractSlave.java:159) at hudson.plugins.ec2.EC2SpotSlave.<init>(EC2SpotSlave.java:44) at hudson.plugins.ec2.EC2SpotSlave.<init>(EC2SpotSlave.java:37) at hudson.plugins.ec2.SlaveTemplate.newSpotSlave(SlaveTemplate.java:979) at hudson.plugins.ec2.SlaveTemplate.provisionSpot(SlaveTemplate.java:919) at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:464) at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:578) at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:594) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:715) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

This hasn't happened with on-demand instances.

This may be related to JENKINS-55618; I'm affected by that bug as well, but I wasn't sure how closely related it was so opted to open a new issue.

 
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

tommoyang@gmail.com (JIRA)

unread,
Jan 17, 2019, 12:54:02 AM1/17/19
to jenkinsc...@googlegroups.com
Tao Yang updated an issue
Change By: Tao Yang
When Jenkins attempts to provision spot instances, after spot requests are created (incorrectly, as per JENKINS-55618) Jenkins will try to retrieve the spot instance based on the last working instance id from before we upgraded the plugin was upgraded , instead of the instance id returned by the spot request.

 
{code:java}
INFO: Spot instance id in provision: sir-34si4rxjINFO: Spot instance id in provision: sir-34si4rxjJan 17, 2019 4:32:10 AM hudson.plugins.ec2.CloudHelper getInstanceINFO: Unexpected number of reservations reported by EC2 for instance id 'i-0b934d05d88272fc4', expected 1 result, found []. Instance seems to be dead.Jan 17, 2019 4:32:10 AM hudson.plugins.ec2.EC2Cloud provisionWARNING: SlaveTemplate{ami='ami-09009aeca3492b598', labels='agent linux'}. Exception during provisioningcom.amazonaws.AmazonClientException: Unexpected number of reservations reported by EC2 for instance id 'i-0b934d05d88272fc4', expected 1 result, found []. Instance seems to be dead. at hudson.plugins.ec2.CloudHelper.getInstance(CloudHelper.java:54) at hudson.plugins.ec2.CloudHelper.getInstanceWithRetry(CloudHelper.java:25) at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:499) at hudson.plugins.ec2.EC2AbstractSlave.<init>(EC2AbstractSlave.java:159) at hudson.plugins.ec2.EC2SpotSlave.<init>(EC2SpotSlave.java:44) at hudson.plugins.ec2.EC2SpotSlave.<init>(EC2SpotSlave.java:37) at hudson.plugins.ec2.SlaveTemplate.newSpotSlave(SlaveTemplate.java:979) at hudson.plugins.ec2.SlaveTemplate.provisionSpot(SlaveTemplate.java:919) at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:464) at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:578) at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:594) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:715) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748){code}

This hasn't happened with on-demand instances.

This may be related to JENKINS-55618; I'm affected by that bug as well, but I wasn't sure how closely related it was so opted to open a new issue.

michael.alberghini@keysight.com (JIRA)

unread,
Jan 17, 2019, 12:54:02 PM1/17/19
to jenkinsc...@googlegroups.com
Mike Alberghini commented on Bug JENKINS-55639
 
Re: ec2-plugin only attempts to retrieve spot instance using old id

We also faced this problem on Jenkins 2.157 with ec2-plugin 1.42.  Switching our spot instances to regular did not fix the issue, but downgrading the plugin to 1.41 did

vdc@zymergen.com (JIRA)

unread,
Feb 4, 2019, 11:17:02 PM2/4/19
to jenkinsc...@googlegroups.com

On 1.42 we get this behavior intermittently. We oscillate between this and JENKINS-55556.

vdc@zymergen.com (JIRA)

unread,
Feb 4, 2019, 11:25:02 PM2/4/19
to jenkinsc...@googlegroups.com
Victor Chavez edited a comment on Bug JENKINS-55639
On Jenkins 2.150.2 EC2 plugin 1.42 we get this behavior intermittently. We oscillate between this and JENKINS-55556.

paul@alphaori.sg (JIRA)

unread,
May 7, 2019, 5:30:03 PM5/7/19
to jenkinsc...@googlegroups.com
Paul P commented on Bug JENKINS-55639

I had this issue and uninstalling the plugin and re-installing it fixed it.

To be specific:

  1. backup your config.xml
  2. uninstall the plugin
  3. restart jenkins
  4. re-install the plugin (I lost my configuration)
  5. stop jenkins
  6. replace the config.xml
  7. start Jenkins

I am using Jenkins 2.176 and ec2 plugin 1.42

paul@alphaori.sg (JIRA)

unread,
May 7, 2019, 6:23:02 PM5/7/19
to jenkinsc...@googlegroups.com
Paul P edited a comment on Bug JENKINS-55639
EDIT: Looks like I spoke too soon, the problem re-appeared.

I had this issue and uninstalling the plugin and re-installing it fixed it.

To be specific:
# backup your config.xml
# uninstall the plugin
# restart jenkins
# re-install the plugin (I lost my configuration)
# stop jenkins
# replace the config.xml
# start Jenkins


I am using Jenkins 2.176 and ec2 plugin 1.42

fabrizio.manfredi@gmail.com (JIRA)

unread,
May 17, 2019, 5:14:02 AM5/17/19
to jenkinsc...@googlegroups.com

The 1.43 contains a fix on the missing default value of the new options that are not present for people that come from old plugin.

fabrizio.manfredi@gmail.com (JIRA)

unread,
May 17, 2019, 5:15:04 AM5/17/19
to jenkinsc...@googlegroups.com
FABRIZIO MANFREDI edited a comment on Bug JENKINS-55639
The 1.43 contains a fix on the missing default value of the new options that are not present for people that come from old plugin. Can you test ? 

yan@sutoiku.com (JIRA)

unread,
May 17, 2019, 5:24:02 AM5/17/19
to jenkinsc...@googlegroups.com

Actully I can't, I'm sorry. I had to rollback and fix my instance EC2 plugin to v1.39 due to https://issues.jenkins-ci.org/browse/JENKINS-52362. 

paul@alphaori.sg (JIRA)

unread,
May 17, 2019, 4:37:02 PM5/17/19
to jenkinsc...@googlegroups.com
Paul P commented on Bug JENKINS-55639

I can confirm 1.43 & 2.177 work fine with Spot instances

tommoyang@gmail.com (JIRA)

unread,
May 19, 2019, 10:10:03 PM5/19/19
to jenkinsc...@googlegroups.com
Tao Yang commented on Bug JENKINS-55639

Thanks FABRIZIO MANFREDI, it looks to me like it's working correctly now!

tommoyang@gmail.com (JIRA)

unread,
May 19, 2019, 10:11:04 PM5/19/19
to jenkinsc...@googlegroups.com
Tao Yang closed an issue as Fixed
 
Change By: Tao Yang
Status: Open Closed
Resolution: Fixed
Reply all
Reply to author
Forward
0 new messages