jettyRun Hangs/Chaos Monkey Is Unable To Terminate an Instance

[luthi...@gmail.com]

Sorry if these are newbie questions, but I'm new to Gradle/jettyRun as well as the Netflix Simian Army scripts.

Question 1: Is it normal for SimianArmy to hang at 75% during the jettyRun build? I've been Ctrl + Z interrupting it, killing the java process and rerunning it, but this doesn't seem right.

Question 2: My Chaos Monkey is unable to terminate instances-this is the error message I get:

ERROR BasicChaosEmailNotifier - [BasicChaosEmailNotifier.java:178] Property simianarmy.chaos.notification.sourceEmail is not set or its value is not a valid email.

2015-03-26 21:49:20.486 - ERROR BasicChaosMonkey - [BasicChaosMonkey.java:201] failed to terminate instance i-fab74231

java.lang.RuntimeException: Property simianarmy.chaos.notification.sourceEmail is not set or its value is not a valid email.

at com.netflix.simianarmy.basic.chaos.BasicChaosEmailNotifier.getSourceAddress(BasicChaosEmailNotifier.java:179)

at com.netflix.simianarmy.aws.AWSEmailNotifier.sendEmail(AWSEmailNotifier.java:77)

at com.netflix.simianarmy.basic.chaos.BasicChaosEmailNotifier.buildAndSendEmail(BasicChaosEmailNotifier.java:133)

at com.netflix.simianarmy.basic.chaos.BasicChaosEmailNotifier.sendTerminationGlobalNotification(BasicChaosEmailNotifier.java:76)

at com.netflix.simianarmy.basic.chaos.BasicChaosMonkey.sendTerminationNotification(BasicChaosMonkey.java:445)

at com.netflix.simianarmy.basic.chaos.BasicChaosMonkey.terminateInstance(BasicChaosMonkey.java:374)

at com.netflix.simianarmy.basic.chaos.BasicChaosMonkey.doMonkeyBusiness(BasicChaosMonkey.java:127)

at com.netflix.simianarmy.Monkey.run(Monkey.java:134)

at com.netflix.simianarmy.Monkey$1.run(Monkey.java:155)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)

I have since gone back and edited the Chaos.properties config file to add my email-is it possible that not having the notifier email set was what was stopping it from terminating an instance? I've restarted the jettyRun several times, and no more instance terminations, and I read the configuration Wiki and how it said Chaos monkey will only terminate an instance once a day.  I would like to query the REST API to create an on-demand instance termination like it says to here but I get a 404 error when I try to query for recent Chaos events, so I doubt a termination request query would work.

Does anyone know how I can get Chaos Monkey terminating instances/the jettyRun to finish building/able to query the REST API?

Here is background info:

-I have SimianArmy installed on a c4.2xlarge EC2 instance using Amazon's standard RHEL 7.1 AMI image. 

-First gradle build went fine, I edited each monkey's scripts according to what the Quick Start Guide on Github said to. 

-Then I went to do the second build (./gradlew jettyRun), and it goes through listing INFO events like BasicSimianArmyContext - , etc...but then it  hangs at "> Building 75% > :jettyRun > Running at http://localhost:8080/simianarmy"

-No errors immediately proceeding this, just the info message (INFO Monkey - [Monkey.java:138] Reporting what I did...)

-The localhost:8080/simianarmy link doesn't show anything except a WebInfo link, which I click and it gives a "not found" error.

- Doing curl http://localhost:8080/api/v1/chaos in a new terminal window while the jettyRun is still going results in a 404 error.

-Access/secret key credentials are set in the .aws/config file, I'm using Java JDK 6 and the emails for Janitor Monkey and Conformity Monkey are going out.

-Credentials used for the monkeys are for a user with full admin access.

I've killed and respun up multiple boxes and worked through the Quick Start guide with them multiple times, but I still keep running into this same problem. :(

[bolleypally vijay kumar]

Hi luthi,

 

Im also new to simian army and i faced few problems related to email notification i gone through the code and found few tips 

My suggestion might help you to see an instance is terminated by chaos monkey 

 Simple just comment ‘sendTerminationNotification’ temporarly in terminateInstance method (com.netflix.simianarmy.basic.chaos.BasicChaosMonkey.java) and run to see instance terminated by chaos monkey

 

To solve your problam

1. Configure simianarmy.chaos.<groupType>.<groupName>.notification.enabled = true and ‘simianarmy.chaos.<groupType>.<groupName>.ownerEmail’ and ‘simianarmy.chaos.notification.sourceEmail’ and start debugging the code in ‘com.netflix.simianarmy.basic.chaos.BasicChaosMonkey.java’ this might take time but may solve your problam

 

-vijay