python 3.x

[maxw]

As far as I can tell, the python protocol buffer code is not yet
compatible with python 3.x. I found a few messages in this discussion
forum indicating that some people had worked on this, but I saw no
pointer to actual code. Are you aware of plans or projects (internal
to google or otherwise) supporting python 3.x? If not, are there
known reasons blocking this?

Thank you in advance!

[Gregory P. Smith]

We're not working on porting it to Python 3.x internally right now as we are not yet ready to move our own codebase to Python 3. It does need to happen and I'd like to see it done this year but I have not taken a close look at it to see what it will take to do the port and I don't believe anyone is currently lined up to work on it.

At first glance I'd break the task down into the following steps:

1) Port the python code under protobuf/python to be clean for conversion to 3.x using 2to3 and fix it up so that unittests pass on both python 2.4 and 3.1; Start with the inner most modules, doing one at a time. I'm not sure how difficult this will be as 2.4 compatibility needs to be maintained. We -might- actually need to fork these into a protobuf/python3 tree if keeping the code both sane while still compatible with 2.4 is too messy.

2) Fix up the python code generator protobuf/src/google/protobuf/compiler/python to support generating code for 3.x.  I'd probably start by taking the easy route: adding a --python3_out to protoc and treading python 3.x as a new language. It is possible to have it generate code that passes 2to3 and works but that could be too much pain; writing code dealing with unicode/str/bytes differences that works on < 2.6 that also works right when passed through 2to3 can be a bit too "fun."

3) After the above is done and pure python 3.x protobufs are working and passing all tests, porting the C++ wrapping extension module under python/google/protobuf/pyext should be the easiest part.

Any attempts you or anyone else can make on the above would be appreciated.

I suspect it might be a lot easier if we can move the minimum supported python version up to 2.6.

-Greg  (g...@google.com)

[Roberto Alsina]

On Tuesday, July 10, 2012 8:13:54 PM UTC-3, Gregory P. Smith wrote:

[resending, initial send didn't make it to the list]

On Tuesday, July 10, 2012 8:06:34 AM UTC-7, Roberto Alsina wrote:

Hello, as part of porting one a product to Python 3, we are willing to port protobuf which is one of our dependencies.

Would a python 2.6 / 3.3 codebase be acceptable for merging upstream? Or would it have to support 2.4?

I ask because the effor to achieve both is quite different.

 

I'd aim for 2.6 / 3.2 rather than 3.3 because 3.2 is the python 3 version available as a package in recent stable linux distros.  We're primarily using 2.6 (soon 2.7) at Google.  People with a need for support of Python versions earlier than 2.6 should be able to use an older release of the protobuf compiler.

I will talk with the devs. If it's significantly less effort to aim for 3.3, we may go with that, if it's not, then 3.2

 

Obviously I haven't spent any time on porting it to Python 3 yet.  Hit me up for code reviews or discussion as you see fit.  We'll be needing this as well but some other work for our 3.x transition has been a higher priority for me so far so getting to this has been further down my list.

My rough thoughts on python 3.x support for protobuf:

I'd be awesome if the protobuf Python libraries & tests used by the generated code could be safe in both 2.6 and 3.2 without the need for conversion using 2to3.  But... that can get messy depending on the code.  If that gets messy, at least make sure that it convert cleanly with 2to3.

The protoc generated code has more options: work in both, work when passed through 2to3, or generate 2.x and 3.x specific versions of the code based on a protoc command line flag.

I am aiming for generating version specific code via option.

 

I prefer anything that avoids a 2to3 step when possible.  A build system already needs to know which version of python it is targeting so it seems reasonable to pass a protoc command line flag but I'm not wedded to that idea.

After reading what I wrote above and comparing it to my post to this list last year quoted below, the main thing that has changed is that I don't care about 2.4 anymore. :)

Which is good :-)

 

-gps

PS in order to accept patches upstream we'll need a contributor license agreement signed.  Quoting previous messages asking for that:

"""
http://code.google.com/legal/individual-cla-v1.0.html -- If you own copyright on your patch.  (This can be signed via a simple web form at the bottom of the page.)
http://code.google.com/legal/corporate-cla-v1.0.html -- If your employer does.

That should not be a problem. 

[Charles Law]

I did some work on this a few months back.  If you search the mailing list for python3 (no space) you can find the thread.

The work I did is pretty rough - it was mostly a proof on concept.  I have a repo that generates python3 files with the suffix pb2_py3.  My code passes all the unittests and I was able to read/write data to Riak, and verified the data wrote correctly using the official python 2 protobufs.

If/when I get the time, I'd like to get the same code working for Python 2 and 3.  The main hurdle is the use of metaclasses since 2to3 doesn't convert those correctly.  I have seen there are some tricks to get this working using the metaclass's constructor (takes in name, bases & a dict), but I haven't experimented with this yet and don't want the code generators to generate a dict to pass in.  After that I think using from __future__ import unicode_literals will cover most of the leftover tricky bits, but it's hard to say without diving in.

On Wednesday, February 6, 2013 6:17:58 AM UTC-8, Wojciech Danilo wrote:

Hi! I\m also very interested in prtobuf support inPy3. Are there any newsin this topic?

[Charles Law]

I have the python 2.6+ & 3.2+ compatible code done.  Search the mailing list for python3 (no space) for more details.