[JIRA] (JENKINS-59661) EC2 fails to provision due incorrect instance cap

[ryan.mortimore@locationsciences.ai (JIRA)]

ryan M created an issue

Jenkins / JENKINS-59661 EC2 fails to provision due incorrect instance cap Issue Type: Bug Assignee: FABRIZIO MANFREDI Attachments: Screenshot_2019-10-04_16-27-43.png Components: ec2-plugin Created: 2019-10-04 15:44 Labels: ec2 ec2-plugin plug-in slave node plugin aws Priority: Major Reporter: ryan M We have been having issues with the EC2 plugin versions above 1.39 where jenkins wont connect to nodes once stopped. Our nodes are all set at instance caps of 1 however, when jenkins tries to bring back up a node it successfully launches it but fails to provision as it thinks we have an instance cap of 0 `Cannot provision - no capacity for instances: 0` Changing our instance cap to 2 instances allows jenkins to connect to one of the nodes but when it tries to re-startup a second node we then see `Cannot provision - no capacity for instances: 0` It seems like the instance cap is off by one. ``` SlaveTemplate{ami='ami-0095fb81fe067be99', labels='devops'}. Attempting to provision slave needed by excess workload of 1 units Oct 04, 2019 3:17:02 PM INFO hudson.plugins.ec2.EC2Cloud getNewOrExistingAvailableSlave SlaveTemplate{ami='ami-0095fb81fe067be99', labels='devops'}. Cannot provision - no capacity for instances: 0 ```   We are running jenkins 2.190.1 with Amazon EC2 plugin version 1.46.1 on an EC2 instance running ubuntu 18.04 Add Comment

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)

[ryan.mortimore@locationsciences.ai (JIRA)]

ryan M updated an issue

Jenkins / JENKINS-59661 EC2 fails to provision due incorrect instance cap

Change By: ryan M

We have been having issues with the EC2 plugin versions above 1.39 where jenkins wont connect to nodes once stopped.

Our nodes are all set at instance caps of 1 however, when jenkins tries to bring back up a node it successfully launches it but fails to provision as it thinks we have an instance cap of 0 ` _{color:#FF0000} Cannot provision - no capacity for instances: 0 ` {color}_

Changing our instance cap to 2 instances allows jenkins to connect to one of the nodes but when it tries to re-startup a second node we then see `Cannot provision - no capacity for instances: 0` It seems like the instance cap is off by one.

```   {code:java} SlaveTemplate \ {ami='ami-0095fb81fe067be99', labels='devops'}. Attempting to provision slave needed by excess workload of 1 units

Oct 04, 2019 3:17:02 PM INFO hudson.plugins.ec2.EC2Cloud getNewOrExistingAvailableSlave

SlaveTemplate \ {ami='ami-0095fb81fe067be99', labels='devops'}. Cannot provision - no capacity for instances: 0 {code}   ```

We are running jenkins 2.190.1 with Amazon EC2 plugin version 1.46.1 on an EC2 instance running ubuntu 18.04

Add Comment

[denis.bel@gmail.com (JIRA)]

Denis Bel commented on JENKINS-59661

Re: EC2 fails to provision due incorrect instance cap ryan M we are experiencing the same issue, but in our case agents marked as Offline in Jenkins actually are still running in AWS. After stop those EC2 instances manually via AWS console, we can see increased capacity again, you could check this out too.

Add Comment

[h35gao@edu.uwaterloo.ca (JIRA)]

Handi Gao commented on JENKINS-59661

Re: EC2 fails to provision due incorrect instance cap

ryan M We are having this issue as well and noticed the same as Denis Bel. The current workaround we have is that instead of stopping the instances after idle timeout, we terminate the instances, i.e. uncheck the option "Stop/Disconnect on Idle Timeout". I think the root cause is somehow releated to JENKINS-57795 which has an open pr addressing it https://github.com/jenkinsci/ec2-plugin/pull/399

Add Comment

[ryan.mortimore@locationsciences.ai (JIRA)]

ryan M commented on JENKINS-59661

Re: EC2 fails to provision due incorrect instance cap

If jenkins has stopped the instance and tries to connect to it I get the instance cap error but the EC2 instance will be running in the EC2 console. I think terminating the instances instead of stopping will work for us. However, we have environment variables set for each node that need to be manually added when a new node is brought up.

Add Comment

[h35gao@edu.uwaterloo.ca (JIRA)]

Handi Gao commented on JENKINS-59661

Re: EC2 fails to provision due incorrect instance cap

ryan M Yeah, the probelm they are having is that the EC2 instance gets restarted in AWS but it will take some time for it to show as "Pending" and the plugin only checks the state of the instance once... if the state is neither running or pending, Jenkins stops provisioning the node... the pr adds the idea of retrying when waking up stopped instances so it should be able to solve the issue we are having right now Oct 16, 2019 5:06:19 PM WARNING hudson.plugins.ec2.EC2Cloud$1 call SlaveTemplate{ami='ami-02eac2c0129f6376b', labels='default master_ec2'}. Node stopped is neither pending, neither running, its {2}. Terminate provisioning For the environment variables, I'm not sure if the init script can add them for you... you could check it out, may save you some effort

Add Comment

[h35gao@edu.uwaterloo.ca (JIRA)]

Handi Gao edited a comment on JENKINS-59661

Re: EC2 fails to provision due incorrect instance cap

[~ryan_m] Yeah, the probelm they are having is that the EC2 instance gets restarted in AWS but it will take some time for it to show as "Pending" and the plugin only checks the state of the instance once... if the state is neither running or pending, Jenkins the plugin stops provisioning the node... the pr adds the idea of retrying when waking up stopped instances so it should be able to solve the issue we are having right now {code:java}

Oct 16, 2019 5:06:19 PM WARNING hudson.plugins.ec2.EC2Cloud$1 call SlaveTemplate{ami='ami-02eac2c0129f6376b', labels='default master_ec2'}. Node stopped is neither pending, neither running, its {2}. Terminate provisioning

{code} For the environment variables, I'm not sure if the init script can add them for you... you could check it out, may save you some effort :P

Add Comment