variable list_sizes

[Paul Querna]

Hi,

We have seen a couple providers who don't have set plans, and their
sizes can be completely arbitrary. (ie, 128mb to 2gb of ram, in 1mb
increments).

I have been pondering how to best expose this flexibility, without
making it too complicated.

Currently we have list_sizes as the main way to get a NodeSize object
to use when calling create_node. What we do today for providers like
VPS.net, is a range(), and create potentially hundreds of NodeSize
objects. This seems less than optimal if you wanted to make a UI
listing all of the available sizes for example.

I think the best was to do it would be to add a new optional method to
the driver interface. (Alternative names welcome)
'variable_node_size', would take the same kwargs (and optionally
provider specific ones) as the NodeSize interface, if the provider
can't create a NodeSize as requested, throw an exception.

For all of the other drivers, they would just not implement this method.

So, using it would be something like

ns = None
if hasattr(driver, 'variable_node_size'):
try:
ns = driver.variable_node_size(ram=568, disk=50)
except:
# node size isn't available

if not ns:
ns = driver.list_sizes()[0] # fall back to your normal node size selection

Thoughts?

Thanks,

Paul

[Alex Polvi]

On Thu, Oct 22, 2009 at 11:27 AM, Paul Querna <pa...@querna.org> wrote:
>
> Hi,
>
> We have seen a couple providers who don't have set plans, and their
> sizes can be completely arbitrary. (ie, 128mb to 2gb of ram, in 1mb
> increments).

Being more specific, VPS.net and vCloud both have variable sizes.
vCloud has fixed RAM, but variable disk (you can pick how many GB you
want). VPS.net allows you to get different sizes in fixed units,
adding 256M of RAM and 10G of disk per unit.

> I have been pondering how to best expose this flexibility, without
> making it too complicated.
>
> Currently we have list_sizes as the main way to get a NodeSize object
> to use when calling create_node.  What we do today for providers like
> VPS.net, is a range(), and create potentially hundreds of NodeSize
> objects.  This seems less than optimal if you wanted to make a UI
> listing all of the available sizes for example.
>
> I think the best was to do it would be to add a new optional method to
> the driver interface. (Alternative names welcome)
> 'variable_node_size', would take the same kwargs (and optionally
> provider specific ones) as the NodeSize interface, if the provider
> can't create a NodeSize as requested, throw an exception.
>
> For all of the other drivers, they would just not implement this method.
>
> So, using it would be something like
>
> ns = None
> if hasattr(driver, 'variable_node_size'):
>  try:
>    ns = driver.variable_node_size(ram=568, disk=50)
>  except:
>     # node size isn't available
>
> if not ns:
>  ns = driver.list_sizes()[0] # fall back to your normal node size selection

I think the use case is going to be someone wants an equivalent sized
node on a different provider. So maybe the interface should just be
driver.get_size(ram=568, disk=50), which is supported on any provider.
The providers that have discrete sizes, we just round up. The NodeSize
object has price in it, so you should be able to inspect and make sure
that the machine satisfies your requirement. This way we would still
support list_sizes, which has all the possible sizes on that provider.
For VPS.net and vCloud, it would be a really long list that gets
generated.

# no matter what the driver, ns will have at least 568M of RAM and 50G of disk
try:
ns = driver.get_size(ram=568, disk=50)
except NodeSizeNotFound:
.....

No matter what the solution, I think it should be supported on all the
drivers, so we do not need to do weird hasattr stuff.

-Alex

--
co-founder, cloudkick.com
twitter.com/cloudkick
541 231 0624

[Paul Querna]

On Thu, Oct 22, 2009 at 11:46 AM, Alex Polvi <po...@cloudkick.com> wrote:
>> I have been pondering how to best expose this flexibility, without
>> making it too complicated.
>>
>> Currently we have list_sizes as the main way to get a NodeSize object
>> to use when calling create_node.  What we do today for providers like
>> VPS.net, is a range(), and create potentially hundreds of NodeSize
>> objects.  This seems less than optimal if you wanted to make a UI
>> listing all of the available sizes for example.
>>
>> I think the best was to do it would be to add a new optional method to
>> the driver interface. (Alternative names welcome)
>> 'variable_node_size', would take the same kwargs (and optionally
>> provider specific ones) as the NodeSize interface, if the provider
>> can't create a NodeSize as requested, throw an exception.

.....

> I think the use case is going to be someone wants an equivalent sized
> node on a different provider. So maybe the interface should just be
> driver.get_size(ram=568, disk=50), which is supported on any provider.
> The providers that have discrete sizes, we just round up. The NodeSize
> object has price in it, so you should be able to inspect and make sure
> that the machine satisfies your requirement. This way we would still
> support list_sizes, which has all the possible sizes on that provider.
> For VPS.net and vCloud, it would be a really long list that gets
> generated.
>
> # no matter what the driver, ns will have at least 568M of RAM and 50G of disk
> try:
>  ns = driver.get_size(ram=568, disk=50)
> except NodeSizeNotFound:
>  .....
>
> No matter what the solution, I think it should be supported on all the
> drivers, so we do not need to do weird hasattr stuff.

Nice, I really like the idea of a get_size method. The default
implementation can just call list_sizes(), and find the nearest match,
and just always round up.

And since you can inspect the NodeSize after the call, you can always
see exactly what you are getting.

I guess if the call is something like disk=500, and the provider has a
max disk size of 100gb, we would just throw an exception, as its not
possible to get a node that big? (the other option is to round 'down'
in that case, but I think it becomes a messy interface then -- always
rounding up to the next available size makes lots of sense to me)

Thanks,

Paul

[Ivan]

This is from Peter (my boss at rimuhosting) as he is not a member of
the list yet.

At RimuHosting we operate from different data centers. Different data
centers have different costs, host server types and, thus, pricing.
We also have different offerings like low contention (vs. regular
contention) servers in some locations. What disk/mem is available to
a running VPS depends on what host server it is running on. And in
some cases the current host server also affects their pricing (e.g. to
get 'good' pricing for a VPS with lots of memory they may need to move
from a host server with a little memory to a host server with lots of
memory.

So for us to give pricing for a new VPS we would need to know the
location, the memory/disk/mem size, and whether they want a low
contention server. We offer a permutation of most of those options as
our pricing plans which we offer in list_sizes. In addition we can
support 'upgrades' for memory, disk and bandwidth. On some plans
those upgrades are 'infinitely' variable. On other plans, like the
low-contention VPS the memory needs to go up in chunks (in order for
the small number of VPSs on a host to 'fit' neatly).

So: how about changing list_sizes so that it supports a few more
options. Let us be able to provide a filter on the returned sizes.

Add INode.location_id

Add INodeSize.location_id

Add:
SizeQuery
- minimum_memory
- minimum_disk
- minimum_bandwidth
- location_id
- size_id
- node_id (get pricing relative to the location of the node)
- ... etc. e.g. you could add maximum_memory, or color.
# All values optional, if not provided then they won't restrict the
search

Change NodeDriver.list_sizes (optionalSizeQuery)
# So that we still just have list_sizes, but now the driver can filter
its output.

And add NodeDriver.resize_node(node, size) and Node.resize(size)
# resizes the node to the provided size. Throws an exception if it
cannot do resize that node to that size and price.

Optionally add NodeDriver.list_locations() and Location.location_id,
location_name.

Example usage: you would be able to query each driver and get the
lowest price for a VPS with 4GB of mem and disk.

[Alex Polvi]

Ivan, Peter,

Wow, thanks for the ideas! I have been thinking a lot about this.
Adding the location to the NodeSize attribute makes a lot of sense.
The QuerySize stuff I think is a bit too out of scope right now, we
could add that once we get the lower level stuff finished. As for the
location in the NodeSize object, here is a proposed approach:

http://github.com/cloudkick/libcloud/issues#issue/5

I have been debating between country codes or using straight lat/long.
Country codes seem like the simplest way to go. Another option would
be to come up with our own syntax for determining location (like EC2
has us-east, eu-west).

Let me know what you think!

-Alex

[Ivan]

Its a good start. We might need a list_locations() when trying to get
a custom NodeSize back though.

Personally I would like to see the locations maybe start with the
country code, but then be able to have any string appended as a
provider specific extension. Basically the appended string should be a
identifier that provides more infomation than just a country
location.
E.g US-Dallas, US-Dallas-Low-Contention (different product at the same
datacenter). UK-London, AU-DC1, AU-DC2 (eg 2 datacenters in AUS, but
don't want to say exactly where, which I think a provider should be
able to do).

I think they need to be flexible, but still have some level of useful
information in them.

-Ivan