Re: EC2 error "Request limit exceeded" with createNodesInGroup

[Adrian Cole]

Best way to solve this would be to make a aggregate version of our state check predicates. 

Currently, we launch in parallel, but the state checkers run individual http requests per node.  We can fix this, and it would significantly ..like SIGNIFICANTLY reduce the requests to ec2.

If interested, lets chat on jclouds-dev and/or irc.

-A

On Mar 30, 2012 3:01 AM, "Paolo Di Tommaso" <paolo.d...@gmail.com> wrote:

Dear all, 

I'm struggling with the following EC2 error when starting a cluster with 50 nodes.  This error does not arise with smaller cluster (2-10 nodes).

Do you ever experienced this kind of problem? Any idea how to solve? 

[org.jclouds.http.handlers.BackoffLimitedRetryHandler] Cannot retry after server error, command has exceeded retry limit 5: [method=AWSInstanceAsyncClient.describeInstancesInRegion, request=POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1]
[jclouds.compute] << problem applying options to node(us-east-1/i-ec4e4688):
org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 503, error: AWSError{requestId='a545b001-c11d-4d38-adfe-89070a16c13e', requestToken='null', code='RequestLimitExceeded', message='Request limit exceeded.', context='{Response=, Errors=}'}

Thank you,

Paolo

--
You received this message because you are subscribed to the Google Groups "jclouds" group.
To view this discussion on the web visit https://groups.google.com/d/msg/jclouds/-/OKlt70OEZKMJ.
To post to this group, send email to jcl...@googlegroups.com.
To unsubscribe from this group, send email to jclouds+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/jclouds?hl=en.

[Pablo Di Tommaso]

Hi Adrian, 

I'm definitely interested in your proposal. I would avoid the burden to split the creation and script execution in many different chunks, also because JClouds is designed to handle groups of nodes. 

But to reduce the number of concurrent requests would not be enough to restrict the size of the user thread pool ? 

Cheers,

Paolo

 

On Friday, March 30, 2012 5:04:01 PM UTC+2, Adrian Cole wrote:

Best way to solve this would be to make a aggregate version of our state check predicates. 

Currently, we launch in parallel, but the state checkers run individual http requests per node.  We can fix this, and it would significantly ..like SIGNIFICANTLY reduce the requests to ec2.

If interested, lets chat on jclouds-dev and/or irc.

-A

[Adrian Cole]

taking jclouds off this thread as it will get scary fast :)

So, this is more an efficiency concern. For example, we can get the
state of all 50 in a single http request. That's gonna be more
scalable than just slowing down our http executor.

Firstly, take over: http://code.google.com/p/jclouds/issues/detail?id=792

Then, I'd do this.

look at: EC2CreateNodesInGroupThenAddToSet

You'll see the initial thing we have to do is ensure that we can read
our writes, ex. that the instances we ran are visible to the api.
This is handled in InstancePresent.

First step would be to attempt to make a version of this which uses a
single network call, and returns an updated list of runningInstances
when all are visible.

Ex. make this:

Predicate<AtomicReference<Iterable<RunningInstance>>>

This would be similar to AtomicNodeRunning, except update the list
based on the multi-arg version of describeInstancesInRegion. When all
instances are api visible, update the atomic reference and return
true.

This should be testable offline using an ExpectTest, but right now, it
is likely quicker just updating EC2CreateNodesInGroupThenAddToSetTest,
and bearing with mocks.

ttu8r!
-A

> --
> You received this message because you are subscribed to the Google Groups
> "jclouds" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/jclouds/-/B85v1U39oB8J.

[Chris Strand]

Hey Adrian/Paolo,

I may have misunderstood your last post Adrian, but it seems like the suggested approach to fixing this is geared towards EC2 only? This of course makes sense in the context of this thread and the referenced issue, but there are many other providers who have limits on requests hitting their API.

It may be possible to bring down the number of requests jclouds sends in general rather than just for one provider?

Chris

PS: I'm only chipping in because exceeding the limits of any provider is a worry for me since it will play havoc with automatically meeting increased demand!

You received this message because you are subscribed to the Google Groups "jclouds-dev" group.
To post to this group, send email to jclou...@googlegroups.com.
To unsubscribe from this group, send email to jclouds-dev...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/jclouds-dev?hl=en.

[Adrian Cole]

Hey, Chris.

eye-to-eye, man!

My suggestion was to start with one piece in the ec2 impl, mainly as a
warm-up. Next stage after that would be to repeat in the general
abstraction.

You interested in helping out w/paolo?

-A

[Andrew Phillips]

>> I may have misunderstood your last post Adrian, but it seems like the
>> suggested approach to fixing this is geared towards EC2 only? This of course
>> makes sense in the context of this thread and the referenced issue, but
>> there are many other providers who have limits on requests hitting their
>> API.
>>
>> It may be possible to bring down the number of requests jclouds sends in
>> general rather than just for one provider?

I can certainly see the use case here but am not sure there is a
simple way of doing this "generically" without having some more
knowledge of what precisely the constraint is.

As Adrian pointed out earlier in the thread, any kind of global
throttle (e.g. on the HTTP executor risks becoming a straightjacket
that also limits jclouds' abilities to make "internal" requests (e.g.
in the TweetStore demo app the HTTP executor is also used from Quartz
to call back to the application).

I suspect we will need some mechanism for more configurable
limitations but am curious to see how we can do this without
introducing too much global state.

ap

[Chris Strand]

Hey,

You interested in helping out w/paolo?

-A

Sure, I'll help out where I can - I may need some hand holding on IRC to do it properly though! As Andrew suggested we should work out a decent solution first. Each provider will have different limitations (if any) - but what I had in mind was simply taking a look around to see if jclouds does more API calls than really necessary anywhere. For example when I was taking a look at Issue 863 I noticed that all nodes seem to be listed when doing operations that I wouldn't initially expect to require that.

Chris

[Pablo Di Tommaso]

Guys, 

Sorry if I'm not taking care of this issue, but I can find the time to work on it. 

I hope the be able in the next days. 

Best,

Paolo

[Adrian Cole]

I await in anticipation :)

Thanks, Paolo!

-A

> --
> You received this message because you are subscribed to the Google Groups
> "jclouds-dev" group.

> To view this discussion on the web visit

> https://groups.google.com/d/msg/jclouds-dev/-/n1CtL5yGhasJ.