Re: [boto-users] Helping with Python 3 support

[Chris Moyer]

IMHO your approach of forking off to a separate project sounds preferable. I think there's a lot to be said about making a specific version of boto (boto3?) that's designed just for python3, and keeping the existing boto a python2 friendly library. I certainly think it should be part of the boto group on github though, but not sure what Mitch's plans are.

And thank you for being interested in contributing to boto! Projects like this don't work without the supportive community of people behind it that are willing to chip in and make it better.

On Mon, Jan 28, 2013 at 12:41 PM, Aaron Maxwell <am...@redsymbol.net> wrote:

Hi Mitch, (and anyone else interested),

I'm Aaron Maxwell, and have been coding primarily in Python 3 for production systems for almost 2 years.  I've also been using boto for another few projects (in python 2.7) in 2012, so am pretty familiar with the library.

First, THANK YOU very much for creating boto. It's been massively beneficial to my work.

I'm now looking at utilizing boto in the Py3k codebase. I could make a personal py3k-only port by just forking it, applying 2to3, and make other changes as I need (ignoring those parts I don't). But I'd much rather contribute directly to your own efforts, Mitch, of making boto officially support Python 3.

When you have a moment, can you please clarify the type of help you'd like? What types of pull requests, specifically, would you like to receive?

Thanks,
Aaron

--
You received this message because you are subscribed to the Google Groups "boto-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to boto-users+...@googlegroups.com.
To post to this group, send email to boto-...@googlegroups.com.
Visit this group at http://groups.google.com/group/boto-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Chris Moyer

[Aaron Maxwell]

Hey Chris,

My pleasure for contributing - looks like you've contributed a lot more, so I should be thanking you.  Boto has helped me a lot, and I'm just trying to give back, which I'm sure you can relate to!

Re: having separate botos for pythons 2 and 3, I have to respectfully disagree. Having been involved in several py3k ports, that approach seems to lead to success only if the maintainers decide to more or less drop Python 2 support. (I'm pretty sure that is not the right thing for boto at this time.)  Otherwise the code bases start to diverge, dev resources are split trying to add features to each, and one of the two eventually gets abandoned.

At this point, in 2013, it's very likely boto can be made to support both 2 and 3 with a single codebase.  See my next post for details.

But, we'll see what Mitch says :-)

Thanks,
Aaron

[Aaron Maxwell]

Hi Mitch,

From your twitter, looks like you are traveling right now, so no rush - please reply only at your convenience.  I went ahead and made a fork:

https://github.com/redsymbol/boto/

As much as possible, I'm packaging everything specific to the port in a folder named "py3kport/".  Basically development documentation and utilities.

In particular, there is a file named py3kport/README.txt describing what I believe is the best development strategy.  There is a Python hack night here in San Francisco next Thursday (Feb 7th), and I'm planning to recruit some help there if I can. [0] The README is geared towards getting an interested contributor up and running.  The whole point is to create pull requests you want to accept and merge, so please advise if anything needs to be different.

Some questions:

 - What's the minimum Python version you want to support? Pretty please say either 2.6 or 2.7 ;-)

 - Is it okay if we utilize the six module? Looks like it's already used in credentials.py.

 - I forked from "develop". That good? It's not any trouble yet if I need to re-fork from a different branch, but it will be eventually.

Thanks,
Aaron

[0] If anyone on this list is in SF and wants to pitch in, drop me a line! amax at redsymbol dot net.

[Jeff Bauer]

It would be great to see support for S3 in this fork, since S3 was
the first (and still most broadly) deployed AWS service. If this
fork gets some traction, I'd be happy to test the S3 service
components.

Jeff Bauer
Rubicon, Inc.

[Aaron Maxwell]

Hey Mitch,

Just want to check in, do you have any feedback? If I'm going to rally contributors at the hack night this Thursday, I'll need to get started contacting people on Monday or Tuesday. Is this something you would like as a contribution?

A question I didn't ask: the py3kport/README.txt outlines a strategy based on 2to3.  Another good approach would be to have a single code base, that runs in both (say) Python 2.6 and 3.3, using the six module to help.  I'd recommend one of these two approaches, but it truly doesn't matter to me which is used, so let me know what you prefer (or we can discuss it too if you like).

Thanks,
Aaron

[dd]

This is a link to the most recent status update on Python 3 for Boto

https://groups.google.com/forum/?fromgroups=#!searchin/boto-users/%22python$203%22/boto-users/qUzNix_noig/89jXnYTdDLIJ

It's frustrating but Python 3 does not seem to be getting off the ground with Boto.

We switched to using Apache Libcloud which provides Python access to some AWS services including S3, and supports Python 3.

[Aaron Maxwell]

Well, I think Mitch is focused more on boto 3 right now, which will support Py3k from the start.

Re libcloud, I love  how it abstracts across different cloud frameworks.  But I'm still biased towards boto: AWS is very, very rich, and boto is the only python library that comes close to enabling so many of its features. (libcloud doesn't support VPC, for example, and possibly never will.)

[Toby Burress]

So, my fork (https://github.com/kurin/boto) now passes all unit tests in pythons 2.6, 2.7, and 3.3.  Including the new set of httpretty tests.

The utilities in bin/ still need working on, but I've tested it unextensively in ec2 (spun up a couple of instances, copied some ssh keys around) and s3 (created buckets and keys; stored some data) and it seemed to work.  I'd love to get some actual usage tests.

[Aaron Maxwell]

Just to be clear to everyone: Toby did all the real work on this. (Thanks, Toby!)  https://github.com/kurin/boto should be considered the most advanced py3k fork now.

Aaron

[Mitchell Garnaat]

Hi -

First, let me say that is an impressive accomplishment.  We attempted an in-place update of boto to support both Python 2.x and 3.x before and we failed.  Perhaps it was partly because we were also trying to convert to requests at the same time.  I think we bit off more than we could chew.  So, congratulations to all involved.  Great job!

I know I've been kind of MIA on this discussion.  You shouldn't interpret that to mean that it isn't important to me.  It's vitally important.  I'm just oversubscribed right now.  My short-term focus has been mainly on aws-cli and botocore.  We really want to get aws-cli to a 1.0 release as soon as we can.  While command line tools appeal to a somewhat different audience than boto, I think there is a lot of overlap and I'm really excited that the new, universal CLI tools for AWS are being developed in Python.  It's great for the community and great for our customers.

As far as boto is concerned, here are my thoughts.

The current boto project is thriving.  The package has been downloaded from PyPI over 4.5 million times and that number is growing by 8-10K a day.  Of course, that doesn't mean that thousands of new individuals are downloading it every day.  Most of those downloads are automated installs on EC2 instances as part of deployments (I'm speculating here, no data to back that up).  But that number is a good indicator of how many people are using and depending on the current boto code base.

So, the prime directive is to do no harm.  We want to continue to support and extend the current boto codebase.  If we can get Python 3.x support into the current code base without breaking 2.x support, I am 100% in support of that.

Having said all of that, I do have concerns about the current boto codebase independent of whether it supports Python 3.x or not.  The current code base is hand-coded.  Every change in a service and every new service requires a considerable amount of new code to be written.  Sometimes that code is written by different people (thanks for those contributions, BTW) and all of this code has been written over the course of almost seven years.  The first version of boto supported three services, S3, EC2, and SQS.  And there was no concept of regions because there was only one endpont for each service.  Today we support 23 Amazon services across 8 regions.  Plus Google Storage.  The pace of innovation on the service side, however, has put a strain on boto.  It's very difficult to keep up when significant amounts of hand-coding are required just to get basic access to a new service or an update of an existing service.

That's where botocore comes in.  The botocore project is a complete rethinking of the low-level layer of boto.  Internal to AWS, we have developed tools that transform the canonical definition of a service into an intermediate JSON format that we can distribute.  These JSON service descriptions are now being used by many of the AWS SDK, like Ruby, PHP, Javascript, etc.  The botocore package is essentially an engine that can interpret these JSON data service descriptions and dynamically create a low-level Python interface to the services.  The benefits are that:

• We can factor out the low-level Python code in such a way that all common services use the same code.  So, all Query-style services use the exact same QueryEndpoint class to marshal parameters to the service.  All services that use SigV4 for authentication use the exact same SigV4Auth class to sign requests.  And all classes with XML responses use the exact same Response class to parse the XML and transform it to native Python data structures.  The code base is dramatically smaller and easier to maintain.
• New services and changes in existing services can be incorporated immediately simply by updating the JSON service description.  No coding required.
• Since we are developing a new code base, we can design it to be compatible with Python 2.6-3.x from the start.  And, we can take advantage of other great Python packages like Kenneth Reitz' requests package rather than rolling our own HTTP layer based on httplib.

The aws-cli package is the first attempt to build something on top of botocore and I have to say it has been going really well.  We are very excited about it.  The next project will be to build a new boto package, let's call it boto3, on top of the same botocore base.  The goal of boto3 would be to supply higher-level interfaces on top of botocore.  For example, we may want to have a compatibility layer that tries to emulate the existing boto interfaces for popular services like S3 and EC2.  Or, people in the community may want to develop some completely new abstractions layer on top of the botocore layer.  But one thing boto3 will not be is 100% compatible with boto2.  There will be changes, just as there were changes from Python 2.x to Python 3.x.

So, boto2 and boto3 will be separate projects.  We will support both but our hope would be that eventually, most people would make the move to boto3.  But we will not abandon the boto 2.x code base.  It will continue to be supported as long as people are using it.

This is a pretty long note.  I thought about providing a tl;dr but I really would like everyone to read the whole thing.  And then let me know what you think.  The success of boto is almost completely a result of the community that has grown around it.  It's your project as much as it is mine.  If you have concerns, express them.  If you like the plan, let me know.

Mitch

On Wed, Feb 20, 2013 at 7:54 PM, Toby Burress <ku...@delete.org> wrote:

So, my fork (https://github.com/kurin/boto) now passes all unit tests in pythons 2.6, 2.7, and 3.3.  Including the new set of httpretty tests.

The utilities in bin/ still need working on, but I've tested it unextensively in ec2 (spun up a couple of instances, copied some ssh keys around) and s3 (created buckets and keys; stored some data) and it seemed to work.  I'd love to get some actual usage tests.

--

[Toby Burress]

Well, I haven't *completely* converted the code.  As I said, the integration tests and binaries still need some work.  But if you think this is good, should I send you a pull request?  Or would you rather developer effort be focused on botocore?

[dd]

Sounds encouraging and like there might be an actual plan in place, that's a good thing. Please forgive me being skeptical but "big rewrite projects" tend to take a long time to deliver, if they deliver at all. Also given how stretched you sound Mitch it sounds like Amazon isn't spending the money on having enough developers to give AWS Python/boto the resource it needs.  The short question being "when can we save stuff to S3 with Python 3" - I'm guessing the answer might be "no time soon"?

[dd]

Sounds like Amazon should hire this Toby Burress guy and get him coding the new version.

On Friday, February 22, 2013 4:31:46 AM UTC+11, Mitchell Garnaat wrote:

[Setra]

I use libcloud as a base library (since my clients don't like vendor-locking issues) but I need to use boto when creating Vendor specific features.

Cloning Boto might be great but it would also be nice to port some of its features to libcloud.

[Toby Burress]

The Python 3 code has a different requirements list than the Python 2 code.

There is now a branch for python3 at https://github.com/boto/boto/tree/py3kport.  The Python 3 requirements are in py3kport/requirements.txt-3.3, which you can install via `pip install -r`.

If you run `python setup.py install` with a python3.3 binary, it *should* correctly convert and install the code.  You need a fairly recent version of virtualenv (and the distribute library that comes with it; 0.6.31 works).

Please open bug reports if you run into any issues.  Thanks!

On Sunday, March 3, 2013 8:53:06 PM UTC-5, Colton Provias wrote:

It's more of when there is a large enough demand for Python 3 support.  On my side, I've tried Toby's work and I'm just hitting snags with required libraries and their compatibility issues (such as nose).  So in the meantime, I'm using my own small library for S3 access but am patiently awaiting a Python 3 version of Boto.

[James Light]

Awesome!

--