[JIRA] [ec2-plugin] (JENKINS-31474) After unsuccessful terminate, ec2-plugin reaches a state where no new slaves are being initialized

[j.smolenaers@icloud.com (JIRA)]

Jeroen Smolenaers created an issue

Jenkins / JENKINS-31474 After unsuccessful terminate, ec2-plugin reaches a state where no new slaves are being initialized Issue Type: Bug Assignee: Francis Upton Components: ec2-plugin Created: 10/Nov/15 11:23 AM Environment: Jenkins 1.631, ec2-plugin 1.29 Priority: Minor Reporter: Jeroen Smolenaers We use ec2-plugin to start on-demand slaves and terminate them after 30m of idle time. This was working fine, until last week when we noticed a full build queue and an empty list of build executors. On EC2, there actually was a slave running (i-1e42a2a7). Below is the relevant fragment from the log: WARNING: Failed to fetch EC2 instance: i-1e42a2a7 com.amazonaws.AmazonServiceException: The service is unavailable. Please try again shortly. (Service: AmazonEC2; Status Code: 503; Error Code: Unavailable; Re-rw-r---quest ID: ff9e539b-f87f-4143-80b6-e11fbf61378e) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1160) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:748) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:467) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:302) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:11768) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:5899) at hudson.plugins.ec2.EC2AbstractSlave.getInstance(EC2AbstractSlave.java:232) at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:378) at hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:350) at hudson.plugins.ec2.EC2OndemandSlave.terminate(EC2OndemandSlave.java:63) at hudson.plugins.ec2.EC2AbstractSlave.idleTimeout(EC2AbstractSlave.java:297) at hudson.plugins.ec2.EC2RetentionStrategy._check(EC2RetentionStrategy.java:99) at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:71) at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:40) at hudson.slaves.ComputerRetentionWork$1.run(ComputerRetentionWork.java:70) at hudson.model.Queue._withLock(Queue.java:1286) at hudson.model.Queue.withLock(Queue.java:1169) at hudson.slaves.ComputerRetentionWork.doRun(ComputerRetentionWork.java:61) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Oct 30, 2015 12:50:32 PM hudson.plugins.ec2.EC2OndemandSlave terminate INFO: EC2 instance already terminated: i-1e42a2a7 Oct 30, 2015 12:50:32 PM hudson.plugins.ec2.EC2OndemandSlave terminate INFO: Removed EC2 instance from jenkins master: i-1e42a2a7 Add Comment

This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)

[raihaan.shouhell@autodesk.com (JIRA)]

Raihaan Shouhell closed an issue as Cannot Reproduce

Can't reproduce this issue

Jenkins / JENKINS-31474 After unsuccessful terminate, ec2-plugin reaches a state where no new slaves are being initialized

Change By: Raihaan Shouhell Status: Open Closed Resolution: Cannot Reproduce

Add Comment

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

[n3v3rf411@gmail.com (JIRA)]

Willie Loyd Tandingan commented on JENKINS-31474

Re: After unsuccessful terminate, ec2-plugin reaches a state where no new slaves are being initialized I have experienced this but the timing is sporadic, and is pretty hard to replicate. Stop on terminate is turned on. When this issue occurs, the node is marked as offline in Jenkins, but the EC2 instance is still running. Since the EC2 filter only searches for stopped instances, this node will never be used.

Add Comment

This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)