Instance is not terminated by chaos monkey
124 views
Skip to first unread message
Svetlin Slavov
Feb 18, 2016, 8:20:03 AM2/18/16
to Simian Army Users
Hello, I am evaluating the SimianArmy project on a local machine. When I run the ChaosMonkey, it recognizes the ASG I have set up and successfully writes information to the SimpleDB. However, Instance is not actually terminated, instead it throws this exception.
I have double-checked that the email is correct, and also verified on Amazon SimpleEmailService. What am I missing?
I have double-checked that the email is correct, and also verified on Amazon SimpleEmailService. What am I missing?
java.lang.RuntimeException: Failed to send email to <my-email>
at com.netflix.simianarmy.aws.AWSEmailNotifier.sendEmail(AWSEmailNotifier.java:86) ~[main/:na]
at com.netflix.simianarmy.basic.chaos.BasicChaosEmailNotifier.buildAndSendEmail(BasicChaosEmailNotifier.java:133) ~[main/:na]
at com.netflix.simianarmy.basic.chaos.BasicChaosEmailNotifier.sendTerminationGlobalNotification(BasicChaosEmailNotifier.java:76) ~[main/:na]
at com.netflix.simianarmy.basic.chaos.BasicChaosMonkey.sendTerminationNotification(BasicChaosMonkey.java:445) [main/:na]
at com.netflix.simianarmy.basic.chaos.BasicChaosMonkey.terminateInstance(BasicChaosMonkey.java:374) [main/:na]
at com.netflix.simianarmy.basic.chaos.BasicChaosMonkey.doMonkeyBusiness(BasicChaosMonkey.java:127) [main/:na]
at com.netflix.simianarmy.Monkey.run(Monkey.java:134) [main/:na]
at com.netflix.simianarmy.Monkey$1.run(Monkey.java:155) [main/:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_95]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_95]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_95]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_95]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_95]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_95]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_95]
Caused by: com.amazonaws.AmazonServiceException: Invalid message subject: Blah BlahInstance i-832ca544 of ASG monkey-target is being terminated by Chaos monkey.
Chaos type: ShutdownInstance. Fucking blah (Service: AmazonSimpleEmailService; Status Code: 400; Error Code: InvalidParameterValue; Request ID: 4998558a-d63c-11e5-b80c-d99e4e418566)
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1160) ~[aws-java-sdk-core-1.10.5.1.jar:na]
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:748) ~[aws-java-sdk-core-1.10.5.1.jar:na]
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:467) ~[aws-java-sdk-core-1.10.5.1.jar:na]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:302) ~[aws-java-sdk-core-1.10.5.1.jar:na]
at com.amazonaws.services.simpleemail.AmazonSimpleEmailServiceClient.invoke(AmazonSimpleEmailServiceClient.java:1731) ~[aws-java-sdk-ses-1.10.5.1.jar:na]
at com.amazonaws.services.simpleemail.AmazonSimpleEmailServiceClient.sendEmail(AmazonSimpleEmailServiceClient.java:1461) ~[aws-java-sdk-ses-1.10.5.1.jar:na]
at com.netflix.simianarmy.aws.AWSEmailNotifier.sendEmail(AWSEmailNotifier.java:84) ~[main/:na]
... 14 common frames omitted
14:36:52.405 [pool-1-thread-1] INFO com.netflix.simianarmy.Monkey - Reporting what I did...
14:36:52.406 [pool-1-thread-1] ERROR com.netflix.simianarmy.Monkey - CHAOS Monkey Error:
java.lang.RuntimeException: failed to terminate instance i-832ca544
at com.netflix.simianarmy.basic.chaos.BasicChaosMonkey.handleTerminationError(BasicChaosMonkey.java:202) ~[main/:na]
at com.netflix.simianarmy.basic.chaos.BasicChaosMonkey.terminateInstance(BasicChaosMonkey.java:387) ~[main/:na]
at com.netflix.simianarmy.basic.chaos.BasicChaosMonkey.doMonkeyBusiness(BasicChaosMonkey.java:127) ~[main/:na]
at com.netflix.simianarmy.Monkey.run(Monkey.java:134) ~[main/:na]
at com.netflix.simianarmy.Monkey$1.run(Monkey.java:155) ~[main/:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_95]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_95]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_95]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_95]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_95]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_95]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_95]
Caused by: java.lang.RuntimeException: Failed to send email to <my-email>
at com.netflix.simianarmy.aws.AWSEmailNotifier.sendEmail(AWSEmailNotifier.java:86) ~[main/:na]
at com.netflix.simianarmy.basic.chaos.BasicChaosEmailNotifier.buildAndSendEmail(BasicChaosEmailNotifier.java:133) ~[main/:na]
at com.netflix.simianarmy.basic.chaos.BasicChaosEmailNotifier.sendTerminationGlobalNotification(BasicChaosEmailNotifier.java:76) ~[main/:na]
at com.netflix.simianarmy.basic.chaos.BasicChaosMonkey.sendTerminationNotification(BasicChaosMonkey.java:445) ~[main/:na]
at com.netflix.simianarmy.basic.chaos.BasicChaosMonkey.terminateInstance(BasicChaosMonkey.java:374) ~[main/:na]
... 10 common frames omitted
Caused by: com.amazonaws.AmazonServiceException: Invalid message subject: Blah BlahInstance i-832ca544 of ASG monkey-target is being terminated by Chaos monkey.
Chaos type: ShutdownInstance. Fucking blah (Service: AmazonSimpleEmailService; Status Code: 400; Error Code: InvalidParameterValue; Request ID: 4998558a-d63c-11e5-b80c-d99e4e418566)
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1160) ~[aws-java-sdk-core-1.10.5.1.jar:na]
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:748) ~[aws-java-sdk-core-1.10.5.1.jar:na]
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:467) ~[aws-java-sdk-core-1.10.5.1.jar:na]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:302) ~[aws-java-sdk-core-1.10.5.1.jar:na]
at com.amazonaws.services.simpleemail.AmazonSimpleEmailServiceClient.invoke(AmazonSimpleEmailServiceClient.java:1731) ~[aws-java-sdk-ses-1.10.5.1.jar:na]
at com.amazonaws.services.simpleemail.AmazonSimpleEmailServiceClient.sendEmail(AmazonSimpleEmailServiceClient.java:1461) ~[aws-java-sdk-ses-1.10.5.1.jar:na]
at com.netflix.simianarmy.aws.AWSEmailNotifier.sendEmail(AWSEmailNotifier.java:84) ~[main/:na]
... 14 common frames omitted
Ed Bukoski
Feb 18, 2016, 6:20:11 PM2/18/16
to simianar...@googlegroups.com
Not sure exactly what the issue is but this is a clue:
Invalid message subject: Blah BlahInstance i-832ca544 of ASG monkey-target is being terminated by Chaos monkey.
There is a debug message in AWSEmailNotifier.java that prints the subject line and recipient. Did you see that line in your log?
--
You received this message because you are subscribed to the Google Groups "Simian Army Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to simianarmy-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Svetlin Slavov
Feb 23, 2016, 10:15:50 AM2/23/16
to Simian Army Users
Turns out I had not escaped the spacebars properly. It works fine now.
Satish Kumar Anandan
Mar 19, 2016, 2:28:52 AM3/19/16
to Simian Army Users
HI Slav,
With the new updated code of the Simian Army we are able to see conformity monkey only working and not chaos monkey. Can you please share if any other specific settings which needs to be done for making the chaos monkey working.
Satish.
Reply all
Reply to author
Forward
0 new messages