boto for Nimbus Cloud

[Jan-Philip Gehrcke]

You perhaps know the Nimbus cloud (or "the Workspace Service"
http://workspace.globus.org).

Currently, I am working on a Google Summer of Code project for Nimbus
and want to control the cloud using a programmatic approach. Nimbus
has implemented an EC2-frontend. This supports parts of EC2's SOAP API
using X.509 authentication.

Last year I collected experience using boto together with AWS.
Development based on Python/boto/AWS really made fun and was very
effective. Now, I would like to confer this to the work with Nimbus.
But the problem is clear: Boto only supports the Query API and Nimbus
only supports the SOAP API.

Now I have 3 options in my mind:
1) Extend boto with SOAP API
2) Extend Nimbus with Query API
3) Don't use boto together with Nimbus

What do you think about these three options? Which way would be the
fastest to realize? What's the particular effort / the complexity? I
don't have an overview regarding option 3.. any suggestions are
appreciated!

These are my first thoughts (after only a bit of investigation):
1) Handling of certificates and SOAP as new request-"backend" would
have to be developed for boto. I'm not totally sure, but I think
handling of the responses wouldn't have to be touched, which perhaps
would keep the overall changes small. Implementing SOAP would
eventually require importing an external SOAP-providing module.

2) Nimbus' internal API is the so-called "Resource Management API" (RM
API). Currently there exist 2 web service frontends to this RM API:
The "partial EC2 SOAP API frontend" and a "WSRF frontend". A wrapper
around this RM API would have to be developed, providing another
frontend: the "partial EC2 Query API frontend". Eventually a problem:
The whole Grid thing uses authentication with certificates.
Introducing SecretKey/AccessKey would introduce a new security model.

3) Do you know of any other programmatic clients providing the SOAP
API? Currently I cannot chum up with the idea of e.g. developing with
Java :-( I like Python/boto so much :-) Or do you have some other
solution in your mind I did not mention until now?

Thank you for sharing your opinions!

Jan-Philip Gehrcke

[Jan-Philip Gehrcke]

You would help me a lot by commenting on
--> tasks/effort/complexity for extending boto with SOAP API and X.509
authentication <--
All the other questions are a bit off-topic. Sorry for this long mail
btw :)

Good night,

Jan-Philip

[Chris Moyer]

I would strongly suggest extending Nimbus to use the REST API, since
there's almost no chance that boto will ever support the SOAP API.

--
Chris Moyer

[mitch]

Hi -

See my comments below:

On Jun 14, 10:48 am, Jan-Philip Gehrcke <jgehr...@googlemail.com>
wrote:

> Now I have 3 options in my mind:
> 1) Extend boto with SOAP API

The actual requests and responses are very similar between SOAP and
Query. I suppose it is possible that a SOAP-based Connection object
could be built that provided it's own implementation of make_request
that handled the SOAP request/response. I will admit that I know very
little about SOAP and have an even smaller amount of interest in
learning anything more about it so I probably wouldn't be much help.
However, I would expect it would be pretty easy just to try it with a
single request/response pair and see how it goes.

> 2) Extend Nimbus with Query API

I like the sound of that. Of course, I'm not the person who has to
extend Nimbus 8^)

> 3) Don't use boto together with Nimbus

Not using boto would be bad enough, not using Python would be
unthinkable! I'm not really sure how good the support for SOAP is in
Python. Anyone else have any experience there? On the Java side,
Dave Kavanagh provides the typica library but he's also using the
Query API. I suspect you would have to look at some of the C#/.Net
implementations to find much SOAP support. Shudder.

[Mihai Ibanescu]

On Mon, Jun 15, 2009 at 7:23 PM, mitch<Mitch....@gmail.com> wrote:
>
> Hi -
>
> See my comments below:
>
> On Jun 14, 10:48 am, Jan-Philip Gehrcke <jgehr...@googlemail.com>
> wrote:
>
>> Now I have 3 options in my mind:
>> 1) Extend boto with SOAP API
>
> The actual requests and responses are very similar between SOAP and
> Query.  I suppose it is possible that a SOAP-based Connection object
> could be built that provided it's own implementation of make_request
> that handled the SOAP request/response.  I will admit that I know very
> little about SOAP and have an even smaller amount of interest in
> learning anything more about it so I probably wouldn't be much help.
> However, I would expect it would be pretty easy just to try it with a
> single request/response pair and see how it goes.

There is another gain in extending Boto to use the SOAP API. As I have
found out recently, the length of the URL is limited by Amazon to 16k.
This includes all URL-encoded fields. In some of my tests, launch
requests with user data exceeding 4k were failing (user data is base64
encoded too).

This is a shame, since the SOAP API lets you use 16k of user data.

Mihai

[Neil Soman]

There's also the EC2 Java library:

http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1674

[Jan-Philip Gehrcke]

Neil, I could not find a definite statement about which API the EC2
Java lib uses. But in each code sample I see secretkey/accesskey
definition. So, in my eyes it can't use the SOAP API?!

I hope it is correct that secretkey/accesskey can only be used with
the Query API. Using the EC2 API tools, you can authenticate using
your certificate. At this point, I hope this means that they must have
implemented the SOAP API, right? Using these assumptions, I concluded
that Nimbus actually only implemented SOAP API, because you have to
use the EC2 API tools and your certificate :-) Would change the whole
discussion now, if that conclusion was wrong..

The idea of getting Nimbus extended sounds great to me, too. But
currently I really cannot estimate how much work this will be. I have
to discuss this with the Nimbus people.

Mitch, thank you your comments and for encouraging me just to try
SOAPing with boto :-)

Mihai, you can always design your system in a way so that you don't
need masses of user-data. It's not that big disadvantage being limited
to ~ 8K.

Jan-Philip



On 16 Jun., 01:38, Neil Soman <neilso...@gmail.com> wrote:

> http://developer.amazonwebservices.com/connect/entry.jspa?externalID=...

[Neil Soman]

On Tue, Jun 16, 2009 at 2:02 AM, Jan-Philip
Gehrcke<jgeh...@googlemail.com> wrote:
>
> Neil, I could not find a definite statement about which API the EC2
> Java lib uses. But in each code sample I see secretkey/accesskey
> definition. So, in my eyes it can't use the SOAP API?!
>

Oh, does it? Sorry, my bad. I thought perhaps that was the Java
library behind the EC2 tools.

You could always run WSDL2Java on a WSDL and generate stubs for a
client app and go from there. Of course, you are constrained to Java
(or C!).