pool provider design

[Adrian Cole]

Hi, guys.

I'm following up on the request originally made by the whirr and
arquillian guys last year to have a pool provider which would help
reduce the impact of node startup time on deploying code and services.
This has come up recently again from Cloudsoft and Neotys, as they
encounter scaling limitations deploying their services on clouds.
Currently, we only test scale of a few nodes, but realistic use cases
are often 20+. To this end, I'd like to kick off a design that will
help address the pooling concern coupled with how to launch batches in
ways that make sense on a per-cloud controller and/or per-offering
basis. This will include discussions around throttling, discussed in
earlier threads, and throttling will end up a separate design.

I've asked each supported cloud provider to expose details about how
best to use their cloud for this purpose, and these results will be
published as an appendix to the design section.

http://code.google.com/p/jclouds/wiki/PoolDesign

Please watch this space and collaborate on the requirements and design!

Cheers,
-Adrian

[Adrian Cole]

Hi, All.

Hugo and I have started adding notes here: http://code.google.com/p/jclouds/wiki/PoolDesign

I'm working on a rough cut of the design, conceding that we have a few things conflated.  A lot of the discussions have been about a few things that support pools rather than the pool design itself.  For example, throttling and cloud-specific scaling strategies support efficient pools, but aren't key tenants of pools themselves.  If you are interested in live progression on this, join irc freenode #jclouds or watch/edit the wiki.

Cheers,

-Adrian

[Andrew Phillips]

> I'm working on a rough cut of the design, conceding that we have a few
> things conflated. A lot of the discussions have been about a few things
> that support pools rather than the pool design itself.

One additional point I remember from the discussions was the
distinction between the goal of efficiently spinning up multiple nodes
(for which pooling is one possible solution) and the goal of being
able to manage a set of long(er)-lived resources - i.e. longer than
the lifetime of the jclouds session that starts them.

Is that something to take to the wiki, or would that be a separate page?

ap

[Adrian Cole]

Hi, Andrew.

The Pool design wiki should really focus on the pool itself, not so much efficiently scaling it.  I haven't yet, but will carve out a couple related wikis:

throttling: we need to design how to manage api throttling as it is a cross/cutting concern across all apis including blobstore and compute

similar to large blob design we need large group design, ex. use the inputs from service providers to better manage strategies to launch high counts of nodes.  For example, in rackspace, we need to space requests every 6 seconds and get feedback from request limits as to whether or not to try.

pool design should ideally focus on the pool semantics and how to manage transitions (cleaning, assigning state, etc), scaling of the pool itself should be handled with cloud providers who are efficient per the above 2 designs.  This is started here, but I haven't finished documenting current thoughts: http://code.google.com/p/jclouds/wiki/PoolDesign

Does this make sense?

-a

--
You received this message because you are subscribed to the Google Groups "jclouds-dev" group.
To post to this group, send email to jclou...@googlegroups.com.
To unsubscribe from this group, send email to jclouds-dev...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/jclouds-dev?hl=en.

[Andrew Phillips]

> pool design should ideally focus on the pool semantics and how to manage
> transitions (cleaning, assigning state, etc), scaling of the pool itself
> should be handled with cloud providers who are efficient per the above 2
> designs. This is started here, but I haven't finished documenting current
> thoughts: http://code.google.com/p/jclouds/wiki/PoolDesign
>
> Does this make sense?

Crystal! My comment stems only from the fact that a lot of the initial
feedback from providers on the page addresses the "how to start
multiple nodes" question more than pooling.
The focus moves towards pPooling as you've outlined it here as you
move down the page ;-)

ap