EC2 Plugin: cannot automatically connect to "stopped on idle" instances

45 views

Skip to first unread message

Ahmed Akram

unread,

Jul 7, 2017, 11:03:28 AM7/7/17

to Jenkins Users

Hello all,

I have a Jenkins setup with 1 master and 9 slaves on AWS and using the EC2 Plugin to manage them.

When setting-up my slave I ticked the "Stop/Disconnect on idle" option so my slaves won't be terminated on idle timeout but rather stopped so it's faster when rebooting them. However, when I run my jobs the master can't provision those stopped instances giving me the following:

Attempting to provision slave from template hudson.plugins.ec2.SlaveTemplate@7adb3f72 needed by excess workload of 3 units of label 'jenkins-slaves'
Considering launching ami-62f4ae74 for template jenkins-slaves
Jul 07, 2017 2:38:34 PM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Considering launching ami-62f4ae74 for template jenkins-slaves
Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Stop
Jul 07, 2017 2:38:34 PM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Stop
Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-62f4ae74]}, {Name: instance.group-name,Values: [AWS-OpsWorks-LB-Server]}, {Name: key-name,Values: [jenkins]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:Name,Values: [jenkins-slave]}],}
checkInstance: {InstanceId: i-0db88147e686a3030,ImageId: ami-62f4ae74,State: {Code: 16,Name: running},PrivateDnsName: ip-172-31-44-107.ec2.internal,PublicDnsName: ec2-52-202-79-47.compute-1.amazonaws.com,StateTransitionReason: ,KeyName: jenkins,AmiLaunchIndex: 0,ProductCodes: [],InstanceType: t2.medium,LaunchTime: Fri Jul 07 13:42:15 UTC 2017,Placement: {AvailabilityZone: us-east-1b,GroupName: ,Tenancy: default,},Monitoring: {State: disabled},SubnetId: subnet-d5e2befd,VpcId: vpc-795ca41c,PrivateIpAddress: 172.31.44.107,PublicIpAddress: 52.202.79.47,Architecture: x86_64,RootDeviceType: ebs,RootDeviceName: /dev/sda1,BlockDeviceMappings: [{DeviceName: /dev/sda1,Ebs: {VolumeId: vol-085ac48e9316d664a,Status: attached,AttachTime: Fri Jul 07 13:42:16 UTC 2017,DeleteOnTermination: true}}],VirtualizationType: hvm,ClientToken: 4a00c997-c822-4caf-b323-6901a2080305,Tags: [{Key: jenkins_slave_type,Value: demand_jenkins-slaves}, {Key: Name,Value: jenkins-slave}],SecurityGroups: [{GroupName: AWS-OpsWorks-LB-Server,GroupId: sg-c806e8ac}],SourceDestCheck: true,Hypervisor: xen,NetworkInterfaces: [{NetworkInterfaceId: eni-1087bcbc,SubnetId: subnet-d5e2befd,VpcId: vpc-795ca41c,Description: ,OwnerId: 568067863565,Status: in-use,MacAddress: 12:89:f5:d8:18:fe,PrivateIpAddress: 172.31.44.107,PrivateDnsName: ip-172-31-44-107.ec2.internal,SourceDestCheck: true,Groups: [{GroupName: AWS-OpsWorks-LB-Server,GroupId: sg-c806e8ac}],Attachment: {AttachmentId: eni-attach-c8fb7b2b,DeviceIndex: 0,Status: attached,AttachTime: Fri Jul 07 13:42:15 UTC 2017,DeleteOnTermination: true},Association: {PublicIp: 52.202.79.47,PublicDnsName: ec2-52-202-79-47.compute-1.amazonaws.com,IpOwnerId: amazon},PrivateIpAddresses: [{PrivateIpAddress: 172.31.44.107,PrivateDnsName: ip-172-31-44-107.ec2.internal,Primary: true,Association: {PublicIp: 52.202.79.47,PublicDnsName: ec2-52-202-79-47.compute-1.amazonaws.com,IpOwnerId: amazon}}],Ipv6Addresses: []}],EbsOptimized: false,EnaSupport: true}
Found existing corresponding Jenkins slave: i-0db88147e686a3030
false - Node is not partially idle
No existing instance found - but cannot create new instance

So it can detect that I have an instance meeting the criteria "Found existing corresponding Jenkins slave: i-0db88147e686a3030", but it cannot' start it up with the "Node is not parially idle". When I manually launch the slaves from the Jenkins slave UI, it normally works, connecting to my stopped instances and running my jobs.