EC2 error "Request limit exceeded" with createNodesInGroup

[Paolo Di Tommaso]

Dear all, 

I'm struggling with the following EC2 error when starting a cluster with 50 nodes.  This error does not arise with smaller cluster (2-10 nodes).

Do you ever experienced this kind of problem? Any idea how to solve? 

[org.jclouds.http.handlers.BackoffLimitedRetryHandler] Cannot retry after server error, command has exceeded retry limit 5: [method=AWSInstanceAsyncClient.describeInstancesInRegion, request=POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1]
[jclouds.compute] << problem applying options to node(us-east-1/i-ec4e4688):
org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 503, error: AWSError{requestId='a545b001-c11d-4d38-adfe-89070a16c13e', requestToken='null', code='RequestLimitExceeded', message='Request limit exceeded.', context='{Response=, Errors=}'}

Thank you,

Paolo

[Andrei Savu]

This is a known issue when trying to start a large number of nodes using a single method call. 

There is some work happening on this: http://www.jclouds.org/documentation/reference/pool-design

My suggestions is to split this group in smaller ones over a period of time (e.g. 3 x 15 nodes at a time). 

--
You received this message because you are subscribed to the Google Groups "jclouds" group.
To view this discussion on the web visit https://groups.google.com/d/msg/jclouds/-/OKlt70OEZKMJ.
To post to this group, send email to jcl...@googlegroups.com.
To unsubscribe from this group, send email to jclouds+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/jclouds?hl=en.

[Paolo Di Tommaso]

Hi Andrei, 

Thanks for your tip. But I was digging in the JClouds sources, and I have found this constants: 

jclouds.max-retries

jclouds.retries-delay-start (Default value for this property is 50 milliseconds)

Do you think would help increasing the "retries-delay-start" to something bigger, let's say 500ms - 1s ? 

Best,

Paolo

[Andrei Savu]

On Fri, Mar 30, 2012 at 1:22 PM, Paolo Di Tommaso <paolo.d...@gmail.com> wrote:

Do you think would help increasing the "retries-delay-start" to something bigger, let's say 500ms - 1s ? 

It could work but you are still going to see errors in the log files. I would go for an alternative approach (multiple batches). 

[Adrian Cole]

Best way to solve this would be to make a aggregate version of our state check predicates. 

Currently, we launch in parallel, but the state checkers run individual http requests per node.  We can fix this, and it would significantly ..like SIGNIFICANTLY reduce the requests to ec2.

If interested, lets chat on jclouds-dev and/or irc.

-A

[Pablo Di Tommaso]

Hi Adrian, 

I'm definitely interested in your proposal. I would avoid the burden to split the creation and script execution in many different chunks, also because JClouds is designed to handle groups of nodes. 

But to reduce the number of concurrent requests would not be enough to restrict the size of the user thread pool ? 

Cheers,

Paolo

 

On Friday, March 30, 2012 5:04:01 PM UTC+2, Adrian Cole wrote:

Best way to solve this would be to make a aggregate version of our state check predicates. 

Currently, we launch in parallel, but the state checkers run individual http requests per node.  We can fix this, and it would significantly ..like SIGNIFICANTLY reduce the requests to ec2.

If interested, lets chat on jclouds-dev and/or irc.

-A