gevent py3

[Benoit Chesneau]

Hi guys,

I've ported recently gunicorn to python 3.2/3.3 and wanted to port
async workers as well. Is there any stable branch or mostly I should
test these days? Do you envision to release the 1.0 with the python 3
compatibility?

By the way if any gevent dev is at pycon I would be happy to discuss
about that with you :)

Please, let me know.

- benoît

[Ralf Schmitt]

Benoit Chesneau <bche...@gmail.com> writes:

> Hi guys,
>
> I've ported recently gunicorn to python 3.2/3.3 and wanted to port
> async workers as well. Is there any stable branch or mostly I should
> test these days? Do you envision to release the 1.0 with the python 3
> compatibility?

I don't think that's possible. There's still lots of stuff to be done
and the 1.0 release should happen real soon now AFAIK.

--
Cheers
Ralf

[Benoit Chesneau]

Is there any open branch for python 3 support anyway? Would be happy to help.

- benoit

[Sean McQuillan]

Hi guys,

I saw this thread a few days back and wanted to see if anyone else was going to make the PyCon sprints?

Cheers,

Sean

[Benoit Chesneau]

On Sat, Mar 10, 2012 at 8:54 AM, Benoit Chesneau <bche...@gmail.com> wrote:

> Is there any open branch for python 3 support anyway? Would be happy to help.
>

So? Any branch I can use for a start?

- benoît

[Sean McQuillan]

Hey Benoit,

I'm going to work on hacking out python 3 support here at pycon.  You on irc?

Sean

[Denis Bilenko]

There's no branch open specifically for python3, you should just fork trunk.
 

- benoît

[Benoit Chesneau]

On Mon, Mar 12, 2012 at 10:33 AM, Sean McQuillan <se...@twilio.com> wrote:
> Hey Benoit,
>
> I'm going to work on hacking out python 3 support here at pycon.  You on
> irc?
>
> Sean
>

I am the pycon too in the circus room. I need to focus on circus right
now, but can probably help sometimes in the day.

- benoît

[Benoit Chesneau]

OK.

- benoît

[Damien Churchill]

On 12 March 2012 17:37, Denis Bilenko <denis....@gmail.com> wrote:
>
> There's no branch open specifically for python3, you should just fork trunk.
>

You could use
https://bitbucket.org/anil_g/gevent as a reference though, it was
using 2to3 which will be the cleanest solution.

[Ralf Schmitt]

Damien Churchill <dam...@gmail.com> writes:

please do not use 2to3. we're aiming for a single source distribution
(with some help of six in gevent/six.py)

--
Cheers
Ralf

[Damien Churchill]

I initially went down a similar route but ran into lots of fun
regarding strings (things excepting unicode instead of bytestring for
example), after a quick look over six it doesn't seem like six will
help with that.. I could be mistaken however.

[Ralf Schmitt]

Damien Churchill <dam...@gmail.com> writes:

> I initially went down a similar route but ran into lots of fun
> regarding strings (things excepting unicode instead of bytestring for
> example), after a quick look over six it doesn't seem like six will
> help with that.. I could be mistaken however.

it has some helper functions:
http://packages.python.org/six/#binary-and-text-data

[Damien Churchill]

Yeah looks similar to what I ended up doing which got quite ugly. It's
probably possible though, was quite a while ago I looked at it all and
I imagine there is a more elegant solution than I came up with.

[Damien Churchill]

Just curious, but what's the thinking behind wanting to use six over 2to3?

[Ralf Schmitt]

Damien Churchill <dam...@gmail.com> writes:

>
> Just curious, but what's the thinking behind wanting to use six over 2to3?

It's about 'single source' vs 2to3.

On the pro side there is:

- no separate compilation step
- being able to work from a hg checkout with python 3
- being able to map python 3 tracebacks to source code
- no need to worry about 2to3 bugs (I've seen that in bottle, they had
to change code in order to make it work with 2to3 on certain
installations)

On the cons side:
- ugly code (especially the ex = sys.exc_info()[1] stuff)

--
Cheers
Ralf

[Damien Churchill]

*shudders*

I tend to agree with the pro points, and if the string mess can be
resolved then it would end up nicer to use, maybe not so much to read,
but that's not a huge deal.

[Jim Fulton]

Some other important cons:

- You have to use values and items rather than itervalues and
iteritems, which may be a significant performance hit, depending
on your app.

- To add insult to injury, you also have to wrap values and
items calls in list calls if you want to make sure you
have copy semantics in Python 3.

Still, I don't like 2to3 except for getting started on a single-
source port.

Jim

--
Jim Fulton
http://www.linkedin.com/in/jimfulton

[Ralf Schmitt]

Jim Fulton <j...@zope.com> writes:

>
> Some other important cons:
>
> - You have to use values and items rather than itervalues and
> iteritems, which may be a significant performance hit, depending
> on your app.

six helps a bit with that.it has six.iteritems, six.itervalues for
that. instead of calling d.iteritems() you would call six.iteritems(d).

>
> - To add insult to injury, you also have to wrap values and
> items calls in list calls if you want to make sure you
> have copy semantics in Python 3.

six doesn't help here (no keys/values function).

I'm getting sad thinking about python 3. Some of the changes introduced
just seem stupid.

--
Cheers
Ralf

[Matthias Urlichs]

Hi,

Jim Fulton <j...@zope.com> writes:
> Some other important cons:
>

> - To add insult to injury, you also have to wrap values and
> items calls in list calls if you want to make sure you
> have copy semantics in Python 3.
>

So you use list(six.iterkeys(DICT)). Or add an appropriate keys()
wrapper to six. Or submit a py3 patch that adds stable iterators
to dictionaries. Or write a dict wrapper which supports it.

While I agree that code which works unchanged between py2 and py3 is not as
straightforward to write as we all would like, ultimately I'd rather use
that than rely on 2to3.

It could have been much worse. Perl6, for instance, also initially planned
as a somewhat-incremental approach, is a wholly different language WRT perl5.

--
-- Matthias Urlichs

[Colin Marc]

I was playing around with this a bit today. It seems like, besides syntax, the biggest problem is different signatures between 2 and 3. For example, socket.makefile in 3.3 has a very different signature than 2.6, and uses io.* objects internally instead of _fileobject (which has been removed). Another example is the ssl library, which has new parameters and uses SSLContext.wrap_socket to wrap sockets instead of ssl.wrap_socket(). The list goes on.

What are your thoughts on how to handle this? It seems like you'd either have to do big "if six.PY3: ... else: ..." blocks, or define completely different modules and then select which one to import based on version.

[Benoit Chesneau]

On Mon, Mar 26, 2012 at 3:23 PM, Colin Marc <coli...@gmail.com> wrote:
> I was playing around with this a bit today. It seems like, besides syntax,
> the biggest problem is different signatures between 2 and 3. For example,
> socket.makefile in 3.3 has a very different signature than 2.6, and uses
> io.* objects internally instead of _fileobject (which has been removed).
> Another example is the ssl library, which has new parameters and uses
> SSLContext.wrap_socket to wrap sockets instead of ssl.wrap_socket(). The
> list goes on.
>
> What are your thoughts on how to handle this? It seems like you'd either
> have to do big "if six.PY3: ... else: ..." blocks, or define completely
> different modules and then select which one to import based on version.
>

I would provide a patch for the SocketRaw object instead for the
socket part. It's possible to provide a comptible api for 2.7/2.6.

- benoit

[Denis Bilenko]

On Tue, Mar 27, 2012 at 2:23 AM, Colin Marc <coli...@gmail.com> wrote:

What are your thoughts on how to handle this? It seems like you'd either have to do big "if six.PY3: ... else: ..." blocks, or define completely different modules and then select which one to import based on version.

I think the difference between socket modules is so big, that it warrants putting different implementation into different modules.

That is, I'd do this:
- rename gevent/socket.py gevent/socket2.py
- implement gevent/socket3.py for Python3
- add socket.py that imports one or another depending on sys.version

[Colin Marc]

What about a more inheritance-based monkeypatching strategy? It seems like most of the differences between 2 and 3 in the socket and ssl libraries are gevent-unspecific; the send* and recv* methods didn't really change. 

Basically:

- The gevent socket class would inherit from the python socket class instead of wrapping the C socket class. If whitelisting methods is the issue, that could be done with __getattribute__.

- The gevent SSLSocket class would inherit from the python SSLSocket class, which would make the inheritance tree look like this:

gevent.ssl.SSLSocket -> ssl.SSLSocket -> gevent.socket.socket (monkeypatched) -> socket.socket

This would make the gevent.ssl module tiny, but you'd have to monkey patch socket to use it. That said, you could either do something like (pseudocode):

monkeypatch socket

import ssl as _real_ssl

un-monkeypatch socket

Or, you could do monkeypatching by default, which is cleaner but has its own issues, obviously. That's not a huge barrier, though. The advantage would be that most of the differences with regards to different versions of python would be hidden, and the gevent modules would be a lot less code.

Thoughts? I'm willing to try implementing this strategy as part of my porting work, if there's interest. 

[Colin Marc]

In case my rambling wasn't clear, here is some code which demonstrates what I'm talking about: https://gist.github.com/2268248. I've tested it with 3.2.

[AnilG]

Perhaps I'm a tiny bit late on this thread but I just wanted to throw
in some comments in case it helps, and to repeat my previous offers:
if someone can point me at a particular task they want done for gevent
py3k I'd like to help. Not being familiar with gevent code and still a
newbie with respect to gevent itself I'm probably better put to
something laborious, rather than strategic.

2TO3 / SIX:

I know Denis doesn't like it and maybe I don't understand, but:
I just thought 2to3 would be better because it results in fixed final
code.
The decisions on which code to use depending on Py 2/3 are made once
at build time. No run time overhead.
The alternative is to make the code check for Py 2/3 usage at every
point a code difference is required; at run time!
I don't understand how this can be viable? This has got to be
vulnerable and slow in comparison?

I thought six was a compatibility layer? Again, a run time overhead?

With 2to3 as well, much work on conversion has already been done, you
just run the converter and it fixes; no manual effort.

http://python3porting.com/toc.html
http://packages.python.org/distribute/python3.html
http://docs.python.org/library/2to3.html
http://pypi.python.org/pypi/six

UNICODE STRINGS

With regard to strings in Py3k, I can't claim to see the full picture,
but trusting the Python team it feels like unicode should be
consistent and elegant in comparison to Py2 now.

When I looked at gevent socket.py with respect to this I felt there
are two cases:

1. Most if not all strings need to be Py3k unicode strings.
2. Actual data for carrying over the socket is never inspected and
comes in as bytes (and goes out as bytes).

It seems to me the following kinds of strings need to be normal Py3k
unicode strings:

* doc strings
* error messages and exception values e.g. raise error(EBADF, 'Bad
file descriptor')
* method names e.g. __implements__ and __imports__
* comparisons to unicode from other modules e.g. sys.platform ==
'win32'
* attribute names e.g. getattr(switch, '__self__', None)
* __repr__ and __str__ values

In socket.py I can't see any strings that might need to be byte
strings except perhaps in bind_and_listen and getfqdn,
but even here the standard socket module uses Py3k unicode strings for
hostnames.
b'' is tolerated (presumably tested for False) but b'hostname' is not.

File "/usr/local/lib/python3.1/socket.py", line 292, in getfqdn
hostname, aliases, ipaddrs = gethostbyaddr(name)
TypeError: must be string, not bytes

In gevent socket.py sending bytes works, and sending a unicode string
throws TypeError as it should:

Traceback (most recent call last):
File "tmc.py", line 17, in <module>
s.send(data) # send test string
File "/usr/local/lib/python3.1/site-packages/gevent-1.0dev-py3.1-
freebsd-8.2-RELEASE-i386.egg/gevent/socket.py", line 542, in send
return sock.send(data, flags)
TypeError: must be bytes or buffer, not str

ARES UNICODE

I ran into questions about Ares.
If I remember there may be a case for decoding/encoding of strings
with respect to Ares, because data on the socket needs to be
inspected?
This then means that an encoding needs to be known but the function
prototypes don't provide for one.
I presume the probable solution is to fix a required/guaranteed
encoding, presumably utf-8.

My thoughts, in constructive spirit, in case it helps. Give me gevent
Py3k work if you have any spare.

[Colin Marc]

Hi There! Work is ongoing at https://bitbucket.org/colinmarc/gevent3000. We've gotten pretty far creating separate socket and ssl modules for python 3, as Denis suggested above. If you have any insight into the C-ares/utf-8 issue, I'm currently stuck on that and would welcome the help. =)

[AnilG]

Thanks for the direction, Marc, and sorry about the delay in reply. I'm working towards getting some time for this and will reply when I have news.