new dependencies for initial Google Compute Engine support

[Shahms King]

boto-dev community,

I have picked up the ball on adding Google Compute Engine support to
boto and wanted to gauge reaction to adding new dependencies on a few
libraries, specifically:

oauth2client
google-api-python-client
httplib2

The httplib2 dependency is particularly unfortunate, but is required
by the other two. While it is my plan to remove the direct httplib2
dependency, at this point I opted for expediency over purity and
wanted to discuss any potential objections up front.

--Shahms

[Gregory Taylor]

I have raised this concern before in talking about maintenance and bloat, but will raise it again in light of the brewing Boto 3. Are the Google services really a good fit for mainline boto (and vice-versa?). Are we lumping Google into boto just because it is a Python cloud module that is popular, or are there good reasons for keeping them in the same codebase? Is the additional overhead in core boto worth it?

I can only speak for myself, but a more trim, focused module is much more attractive as a user and as a maintainer. IMO, the Google stuff would be best in an external module (that could have its own dependencies and behaviors). Perhaps Boto 3 could have some base plumbing that the Google stuff could still import and use/override, but I'm not sure even that makes sense given the different trajectories and roadmaps of Google vs. AWS.

We support other things, like Eucalyptus, but at least in the past there was some API compatibility between it and AWS (there may still be, I don't know). Even though I still think an external package has its merits for Euca, the historic overlap is a reason to keep it mainlined.

This may be a case of me being woefully ignorant of some sweeping overlap between AWS and the Google services, so I am all ears to being properly educated.

--Shahms

--

---
You received this message because you are subscribed to the Google Groups "boto-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to boto-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Shahms King]

On Monday, March 18, 2013 11:31:52 AM UTC-7, Gregory Taylor wrote:

I have raised this concern before in talking about maintenance and bloat, but will raise it again in light of the brewing Boto 3. Are the Google services really a good fit for mainline boto (and vice-versa?). Are we lumping Google into boto just because it is a Python cloud module that is popular, or are there good reasons for keeping them in the same codebase? Is the additional overhead in core boto worth it?

I'm not sufficiently familiar with the intentions of the "brewing Boto 3" to comment, but poking through the botocore git branch a bit, it appears as though that code would be more suitable for Google Compute Engine, rather than less.

 

I can only speak for myself, but a more trim, focused module is much more attractive as a user and as a maintainer. IMO, the Google stuff would be best in an external module (that could have its own dependencies and behaviors). Perhaps Boto 3 could have some base plumbing that the Google stuff could still import and use/override, but I'm not sure even that makes sense given the different trajectories and roadmaps of Google vs. AWS.

I suppose it depends on what kind of user we're talking about here.  Certainly if a user has no desire or intention of supporting/using Google Compute Engine, then such support is unnecessary.  But if a user is already familiar with boto and wants to support Google Compute Engine, having boto provide an way to do this is a big win.  Could you elaborate on your "different trajectories" comment a bit? Are you referring to API transport technologies? Underlying features? Or something else? 

 

We support other things, like Eucalyptus, but at least in the past there was some API compatibility between it and AWS (there may still be, I don't know). Even though I still think an external package has its merits for Euca, the historic overlap is a reason to keep it mainlined.

This may be a case of me being woefully ignorant of some sweeping overlap between AWS and the Google services, so I am all ears to being properly educated.

Again, it depends what level we're talking about.  The JSON/REST API of GCE certainly doesn't have much overlap with the XML/Query based API of EC2, but they have significant conceptual overlap in that they both present a similar model for customers desiring to run their own code on VM instances in the respective clouds.

On balance, I think adding support for additional cloud service provides is a compelling benefit for boto users.

--Shahms

[Gregory Taylor]

* Bundling Google stuff isn't necessarily a win for the users or developers. For example, notice how woefully undocumented the Google modules are: http://boto.readthedocs.org/en/latest/ The storage module has some API docs, but that's it. How about unit tests? Ah, completely un-tested. If we are honest to ourselves, the Google APIs are playing second or third fiddle, and always have. Not to say that it wasn't very generous for Google and others to contribute the code, they are admirable contributions.

* The Google-specific module developers have to get gun-shy about using third party dependencies in any quantity, since the AWS users don't need them. Your first email is a good illustration of this. This is particularly the case if you ever needed to use a module with compiled extension.

* The vast majority of the "core" boto developer time is spent on AWS stuff. The Google-specific code, documentation, and tests are always going to be behind the AWS-specific things. The Google-specific stuff is just along for the ride.

* The underlying APIs are so different that it seems like we're just bolting on an additional set of services (and its respective maintenance burden) just to have a bunch of "cloud stuff" all in one package (which isn't necessarily a good thing).

I mention all of this with no ill will, but I do think the Google-related APIs would be best importing botocore from their own separate Python package.

The Google stuff could live in its own package/repo. The overhead for boto mainline would be less, and the Google stuff could grow its dependency list however it likes (better for developers). All of the Google-related documentation could be in one location, rather than mixing it into the AWS stuff (less confusing for the users). The only difference from the user's perspective is a different name to import, and a more concise collection of documentation.

Since we are seeing work on the next generation of boto, now would be the time to at least discuss this. I think there are a lot of perks to splitting this out from a user *and* developer perspective.

[Shahms King]

Gregory, thanks for your feedback. Documentation and testing concerns
aside (which are problems regardless of where the code lives), it
sounds like, given that the work on v3 is progressing and this is the
general direction for the boto community, it does seem fair to all
involved that we abandon the current effort to add GCE support into
boto/boto.

As you mention, it seems like this is a good time to discuss the
direction the boto community would prefer to go with respect to
supporting other cloud platform providers. From my brief foray into
the botocore source, it seems like a more suitable base for supporting
non-AWS platforms. I suppose the question for the boto community now
is would you prefer to include that support from within boto directly
or simply provide a solid foundation on which others can build that
support?

(I also happen to agree with you on the dependency front, I just
wanted to test the waters before devoting the time towards removing
them and I'm glad that I did so.)

--Shahms

> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "boto-dev" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/boto-dev/fNM7AFM1Ors/unsubscribe?hl=en.
> To unsubscribe from this group and all its topics, send an email to

> boto-dev+u...@googlegroups.com.
>
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

--
:-P

[Gregory Taylor]

It'd certainly be an interesting exercise to see how far you could get building on top of botocore as it stands. 

Also, for the sake of clarity I'll point out that I can't make a call either way on this. It's ultimately Mitch's call. Take my viewpoint on this with a grain of salt, because I am ultimately just some dude on the mailing list :)

--

---

You received this message because you are subscribed to the Google Groups "boto-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to boto-dev+u...@googlegroups.com.

[Chris Moyer]

I would love to see other cloud providers in boto3, but it would be good if extra dependencies would only be added if using those other providers. 

That being said, I know that most devs on this list already have their hands full just with AWS support, so someone would need to take on the role of maintaining any GCE support. I think that's always the biggest worry when adding support for new things...

And no offense to Mitch, but boto has grown far beyond just one man now 8^). Everyone's input is helpful and I don't think any decision with this high of an impact should be made by just one person.

--------------------------------

Sent from my iPhone

Chris Moyer

[Cory Binnersley]

Hi boto-dev community,

I am interested in integrating and supporting the Google Compute Engine service within the botocore framework.

I've successfully written a first-pass iteration of integrating GCE support into botocore, and would like to discuss the necessary implementation details. Furthermore, once I had Google Compute Engine service running in botocore, I was able to trivially get other Google Cloud services running such as Cloud Storage. I see great potential in this, and was hoping to get feedback on whether or not this would fit into botocore's goals.

The only extra dependency required is pycrypto 2.6

-Cory Binnersley

[Chris Moyer]

That's excellent! I thought pycrypto was already a requirement in boto, but if not I really don't see the harm in adding it. Would it be useful for other services too? 

--------------------------------

Sent from my iPhone

Chris Moyer

--

[Cory Binnersley]

I've sent in a pull request which integrates both Google Compute Engine and Google Cloud Storage into botocore.

I'm not sure how useful pycrypto will be to other services, but it is required for authentication using Service Accounts. I've added in support for an alternative authentication using Installed Applications, which does not require pycrypto, and will fill in how to use this method in the documentation in the near future (there is a setup script which has yet to be released).

I've also included some simple examples for interacting with GCE/GCS using botocore in the documentation.

I hope to get some feedback on the implementation/documentation, and am open to any suggestions/comments/changes you all may have.

Thanks!

-Cory Binnersley

[Gregory Taylor]

This is very good work, and I applaud your ambition and generosity. You've even got some documentation and tests, which is awesome.

However, I fear that we're trying to make boto do too much. As I previously mentioned, I think the ideal case scenario is that this goes in a separate package that sits on top of botocore, where it can be maintained separately, accumulate its own dependencies as needed, and not be as entangled with the core.

The bigger the foundational/core stuff gets, the harder it will be to keep in our heads and fully understand. I'm not sure where others stand, but I'd be striving to keep the core as small as humanly possible for the sake of maintainability and ease of future expansion/improvement.

With all of that said, perhaps we could discuss some of the pros and cons for putting additional services in separate packages (where they'd sit "on top of" botocore)? 

Here is what I could think of quickly:

Pros:

• Prevents spaghetti code. Google Compute (or Rackspace, or whatever else) code would be self-contained.
• Reduces the size of botocore, which is a maintainability win.
• Allows the addition of arbitrary third party modules without impacting the entire boto userbase.
• Google Compute (and others) could release outside of the botocore release schedule, allowing them to address issues and release fixes faster or slower than boto.
• At the same token, we wouldn't need to hold up a boto release if an issue appeared last minute, or needed further stabilizing in GC.

Cons:

• We'd need to be more careful about either preserving backwards compatibility for foundational stuff in botocore, or figuring out how to version it. We'd need to be careful about version X.Y of botocore supporting version X.Y of boto-gc. More of a cloud framework model.
• Building on the previous, our service-specific unit tests may need to test against multiple major versions of boto (tox could probably help with this).
• We'd need to make sure to give external services ample mention in the docs, as far as pointing them at the service-specific site/docs/github. Perhaps simply using boto somewhere in the name would help, as well (boto-googlecompute? botogle?)

I definitely don't want to de-value Cory's work, it's very good. I'd just like to at least hear some discussion about where to put this (and future non-AWS services). I fear that we could end up trying to do everything, but do few things well.

--

Greg Taylor

(864) 888-7964

http://gc-taylor.com

[Cory Binnersley]

Thanks Greg,

I agree that keeping the core of botocore to a small size is beneficial in multiple aspects. Adding external services into external packages seems like a great idea, and I think it would be useful to discuss the changes necessary to support this.

Currently, the biggest change I foresee has to do with the credential system. Since botocore only supports a very limited subset of potential authentication methods used by cloud providers, botocore would either have to accept additional credential methods within the main branch, or support some sort of "credential injection" or similar ability. This could also apply to other things, such as endpoints.

If supporting custom endpoints might be useful to providers, then removing the coupling between service.type, endpoints, and parameters (see parameters.py:build_parameter_rest) would be required. Not sure of the best approach to do this, or if this (adding endpoints) is even necessary, but this decoupling is something to keep in mind if it is.

Maybe a possible solution is to have botocore define a "BotocoreModule" class or something similar, which allows service providers to define a credential mechanism/custom logic. Users could then do something along the following lines:

#####

import botocore

import botocore.extensions.google

botocore.session.register(botocore.extensions.google.Compute)

#####

This would import and register the services/logic defined by the Compute class into botocore, and users could continue using botocore as per usual. Obviously, the details of this would be tricky, and need discussion. Similarly, this may not be user-friendly enough, and other approaches may be more suitable.

Again, thanks for the feedback. I look forward to continuing our discussions.

-Cory Binnersley

[Chris Moyer]

What if botocore was just a small abstraction layer, and each cloud provider was an extension, including AWS. Then all that's in botocore would be shared code, anything specific to one system or another would be isolated but still integrated. That seems like a good way to offer extension support too for contributors and other platforms. 

--------------------------------

Sent from my iPhone

Chris Moyer

[Gregory Taylor]

Would definitely be worth thinking about. Perhaps we could discuss what kind of things would be common to Amazon and Google Compute to get the conversation going.

I don't have enough familiarity with Google Compute's auth models or API to be any kind of help pointing out overlap, unfortunately.

[Shahms King]

GCE and the majority of other Google APIs use OAuth2 for authentication/authorization and REST+JSON to expose the actual API.  There are some slight differences from the endpoint perspective, but not many.  The main difference is that (at least in the case of GCE) it's a resource-primary, rather than function-primary, API.  From a support perspective it's relatively minor and translating between the two schemas pretty straightforward (but Cory should have more information there, being the one who did the actual work).

I think it would be nice if botocore also provided utilities for accessing shared configuration for credentials and the like as right now the Auth and Credential classes are pretty tightly coupled, except one is easy to extend (Auth) while the other somewhat less so.

--Shahms

You received this message because you are subscribed to a topic in the Google Groups "boto-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/boto-dev/fNM7AFM1Ors/unsubscribe.
To unsubscribe from this group and all its topics, send an email to boto-dev+u...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
:-P

[Cory Binnersley]

I think it would be useful both to us as developers, as well as users, to get a better understanding of what botocore really is.

If one stripped away the current AWS material and looked at what remained in botocore, one would see something very close the following: a framework which supports communication with RESTful APIs defined in JSON model files.

Currently, both Amazon and Google provide RESTful interfaces to the majority (if not all) of their cloud service offerings. Botocore provides a convenient abstraction layer on top of HTTP, including support for adding authentication to the requests. This removes the burden from developers of having to deal directly with the HTTP layer themselves, but at the same time still exposes the low level functionality needed to build higher level APIs on top of it. One big advantage botocore has is the separation of services into JSON models. Ideally, this would mean that botocore could support any RESTful API defined in one of these models.

As mentioned already, there are a few parts of botocore that require some more abstraction for this to be achieved.

To get to this “core” which I described earlier, AWS specific code would need to be decoupled from botocore, and put into it’s own separate package/module, as an example of those that GCE and other providers would need to provide. This includes hardcoded paths and region code found in the Service class. I’m unsure of the exact scope of work required to do this, but it is certainly doable if a developer was willing.

I suppose we should maybe take a further step back at this point and get some overall clarification. Is the expectation that over time boto users should begin considering moving from boto-v2 to boto-v3? Is "boto-v3" a to-be-written layer on top of botocore, or is it expected that users will code their applications directly against botocore?

-Cory

[Michael Shamberger]

Cory,

I did not find your pull request to botocore?   I am interested in seeing this code.

Mike

[Eric Johnson]

Hey Mike,

The code was parked at https://github.com/GoogleCloudPlatform/botocore but hasn't been touched since the summer when the thread died off.  I haven't re-verified, but I think there was an open pull request out there too.

Eric

--