[JIRA] (JENKINS-56871) EC2 Plugin fails to terminate spot instances after agent deleted

gordon.shieh@safe.com (JIRA)

unread,

Apr 3, 2019, 8:35:02 PM4/3/19

to jenkinsc...@googlegroups.com

Gordon Shieh created an issue

Jenkins /

JENKINS-56871

EC2 Plugin fails to terminate spot instances after agent deleted

Issue Type:	Bug
Assignee:	FABRIZIO MANFREDI
Components:	ec2-plugin
Created:	2019-04-04 00:34
Environment:	CJE Managed Master 2.138.1.2-rolling EC2Plugin 1.42
Priority:	Critical
Reporter:	Gordon Shieh

We had a case where idle instances that were supposedly deleted still running on EC2.
To reproduce:
Launch EC2 template with spot instance enabled
Immediately delete that instance.
On the Spot request console, the request will be "cancelled"
But the associated EC2 instance is still running, and orphaned.

We had a case where we had so many of these orphaned instances that we reached the instance cap of our account.

Add Comment

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

ravibagri2@gmail.com (JIRA)

unread,

May 2, 2019, 1:12:03 PM5/2/19

to jenkinsc...@googlegroups.com

Ravi Bagri commented on

JENKINS-56871

Re: EC2 Plugin fails to terminate spot instances after agent deleted

I recently faced the same issue. later after investing into my cloud-trail logs i found that jenkins role was missing the permission to terminate the instance.

Below is the log snippet.

_{"eventName": "TerminateInstances",}
_{"awsRegion": "us-west-2",}
_{"userAgent": "aws-sdk-java/1.11.457 Linux/4.9.0-6-amd64 OpenJDK_64-Bit_Server_VM/25.212-b01 java/1.8.0_212 groovy/2.4.12",}
_{"errorCode": "Client.UnauthorizedOperation",}
_{"errorMessage": "You are not authorized to perform this operation. Encoded authorization failure message:}
After adding the appropriate permission it seems to have resolved.

Add Comment

ravibagri2@gmail.com (JIRA)

unread,

May 2, 2019, 1:14:02 PM5/2/19

to jenkinsc...@googlegroups.com

Ravi Bagri edited a comment on

JENKINS-56871

Re: EC2 Plugin fails to terminate spot instances after agent deleted

I recently faced the same issue. later after investing into my cloud-trail logs i found that jenkins role was missing had the permission to cancel the spot request but the permission to terminate instance was missing. So that's why the spot request was getting cancelled but the instance was still persisting .

Below is the log snippet.

~"eventName": "TerminateInstances",~
~"awsRegion": "us-west-2",~
~"userAgent": "aws-sdk-java/1.11.457 Linux/4.9.0-6-amd64 OpenJDK_64-Bit_Server_VM/25.212-b01 java/1.8.0_212 groovy/2.4.12",~
~"errorCode": "Client.UnauthorizedOperation",~
~"errorMessage": "You are not authorized to perform this operation. Encoded authorization failure message:~

After adding the appropriate permission it seems to have resolved.