problem spinning up 100 nodes on ec2

[Emanuel Buzek]

Hi all,

I am trying to launch instances on EC2 with jclouds. My code works well for handful of instances, but when I try to launch ~100 nodes, I am running into  issues.

In my aws console, I can see that all instances started (running), but some of them are missing their name (metadata not applied correctly.) This could be like 30% of all instances.

This trace is in my logs several times, probably for each semi-failed instances:

65) AWSResponseException on node us-east-1/i-9c9b4fef:

org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 503, error: AWSError{requestId='a7f3e8b1-018c-45ff-a8ad-9d509e88288b', requestToken='null', code='RequestLimitExceeded', message='Request limit exceeded.', context='{Response=, Errors=}'}

at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:78)

at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:71)

at org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.shouldContinue(BaseHttpCommandExecutorService.java:197)

at org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:167)

at org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:135)

And later on, it seems like jclouds cannot apply the metadata for failed instances:

99) IllegalStateException on node us-east-1/i-0aaa7e79:

java.lang.IllegalStateException: api response for node(us-east-1/i-0aaa7e79) was null, so we can't customize

at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:139)

at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:60)

at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

at java.util.concurrent.FutureTask.run(FutureTask.java:138)

The line in my code (which takes several minutes to return and throws RunNodesException) is this:

 computeService.createNodesInGroup

          (groupName, numInstances,

           computeService.templateBuilder()

                         .hardwareId(hardwareId)

                         .imageId(imageId)

                         .locationId(region)

                         .options(templateOpts)

                         .build());

I am using jclouds 1.5.0 - do you think that upgrade might help?

Thank you,

Emanuel Buzek

[Adrian Cole]

Yes it will definitely help a tag support was refactored in 1.5.5 or so.  Current is 1.5.7

--
You received this message because you are subscribed to the Google Groups "jclouds" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jclouds+u...@googlegroups.com.
To post to this group, send email to jcl...@googlegroups.com.
Visit this group at http://groups.google.com/group/jclouds?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Emanuel Buzek]

Thanks Adrian,

I will try to update and will let you know if this doesn't fix the issue.

[Emanuel Buzek]

Hi,

it seems like the problem is caused by hitting the AWS API request limit (2000/hour/account by default), because jclouds makes multiple API requests per instance launched.

a) has this been improved (batched) in newer versions of jclouds?

b) how many requests exactly does jclouds issue per instance launched?

thank you,

emanuel

[Adrian Cole]

Hi, Emanuel.

Version 1.6 has work in progress on this. The start is a new class
called PresentInstances, which polls in aggregate. There's still
refactoring to do to poll in other places in aggregate, hence 1.6.0
still being in alpha. I've slowly chipped away at this, but can make
more time to complete it while releases are going out this weekend.

Some weeks ago, I summarized related issues in a comment here:
https://github.com/jclouds/jclouds/issues/1214

-A