Officially supported Python versions

[Alek Storm]

The README file looks out of date. Which versions of Python are officially supported at this point? I think it's 2.5, 2.6, 2.7, and 3.2, which the README should be updated to reflect. Are we planning on 3.3 support when it's finalized in August, and can we definitively say we'll never support 3.0 or 3.1?

Alek

[Ben Darnell]

Yeah, I just noticed the badly-outdated README file right after the
2.3 release; I need to update that. The "prerequisites" section on
tornadoweb.org is more complete and up to date. The supported
versions are 2.5+ for the 2.x line and 3.2+ for the 3.x line, with the
intention of supporting each new release as it comes out (I've already
added the 3.3 alphas to my standard test routines). I don't think it
makes sense to go back and fill in 3.0 or 3.1 support (see
https://github.com/facebook/tornado/pull/500).

-Ben

[Alek Storm]

SGTM

[Alek Storm]

What are the supported interpreters? Unofficially, we've talked about:

* CPython (of course)

* Jython

* PyPy?

* IronPython?

Also, will Windows support ever be official?

Alek

[Ben Darnell]

I'd say pypy is officially supported (I'd still be cautious about
using pypy in prod, but that caution is directed at pypy in general
rather than tornado's support for it).

It's not hard to get tornado mostly working on jython if there is
enough interest, although I think it would be unlikely to graduate to
fully-supported status any time soon. Ironpython has some fundamental
problems with its handling of unicode/bytes issues, so I don't think
it's suitable for use with tornado without substantial changes on
their side.

Officially supporting windows would require at least an IOLoop
implementation using windows' equivalent of epoll and kqueue.

-Ben

[Alek Storm]

On Sun, Jun 17, 2012 at 12:13 PM, Ben Darnell <b...@bendarnell.com> wrote:

I'd say pypy is officially supported (I'd still be cautious about
using pypy in prod, but that caution is directed at pypy in general
rather than tornado's support for it).

It's not hard to get tornado mostly working on jython if there is
enough interest, although I think it would be unlikely to graduate to
fully-supported status any time soon.  Ironpython has some fundamental
problems with its handling of unicode/bytes issues, so I don't think
it's suitable for use with tornado without substantial changes on
their side.

I'm more sympathetic to IronPython's cause; let me see if I can get it working with minimal changes in Tornado (Guido officially blessed its str==unicode implementation). Could mean a couple more users, who knows?

Officially supporting windows would require at least an IOLoop
implementation using windows' equivalent of epoll and kqueue.

WFM on Windows with Python 2.7 - am I doing something special? Looks like Windows does indeed have its own select.select() implementation (http://docs.python.org/library/select.html#select.select); it just doesn't accept file objects.

Alek

[Ben Darnell]

On Mon, Jun 18, 2012 at 11:55 AM, Alek Storm <alek....@gmail.com> wrote:
> On Sun, Jun 17, 2012 at 12:13 PM, Ben Darnell <b...@bendarnell.com> wrote:
>>
>> I'd say pypy is officially supported (I'd still be cautious about
>> using pypy in prod, but that caution is directed at pypy in general
>> rather than tornado's support for it).
>>
>> It's not hard to get tornado mostly working on jython if there is
>> enough interest, although I think it would be unlikely to graduate to
>> fully-supported status any time soon.  Ironpython has some fundamental
>> problems with its handling of unicode/bytes issues, so I don't think
>> it's suitable for use with tornado without substantial changes on
>> their side.
>
>
> I'm more sympathetic to IronPython's cause; let me see if I can get it
> working with minimal changes in Tornado (Guido officially blessed its
> str==unicode implementation). Could mean a couple more users, who knows?

str==unicode is fine (jython does the same thing, and it's why a lot
of the python 3 changes in tornado are conditioned on "str is unicode"
rather than sys.version[0] == 3). The problem is that on ironpython,
type(u"".encode("utf8")) == unicode, i.e. there's no way that I can
see to distinguish programmatically between an encoded byte string and
a unicode string that happens to contain only latin1 characters.
There is a bytes type, but it's not always used where it should be
(for example, socket.send only accepts type str).

Anyway, if you'd like to take a look at this and see if there's a
solution that I've missed, here are the secrets to get you past the
first few stumbling blocks. Start with a jython-compatible branch and
drop this at the top of runtests.py:
+import sys
+if sys.platform == 'cli':
+ # ironpython's signal module is broken under mono
+ sys.modules['signal'] = object()
+ # doctests needs this
+ sys.__displayhook__ = sys.displayhook

and run with "mono ipy.exe -X:Frames"

>
>> Officially supporting windows would require at least an IOLoop
>> implementation using windows' equivalent of epoll and kqueue.
>
>
> WFM on Windows with Python 2.7 - am I doing something special? Looks like
> Windows does indeed have its own select.select() implementation
> (http://docs.python.org/library/select.html#select.select); it just doesn't
> accept file objects.

Tornado works on windows now, but doesn't scale very well - the limit
on the number of file descriptors for select() is lower than for unix
(although I think it's become somewhat reasonable in modern versions),
and performance would presumably be better with a windows-specific
implementation. More importantly, this would be a sign that someone
has looked into a windows implementation more deeply than just seeing
that the tests pass and nothing seems obviously broken, which is a key
part of the distinction between "working" and "supported".

-Ben

[Alek Storm]

On Mon, Jun 18, 2012 at 11:51 PM, Ben Darnell <b...@bendarnell.com> wrote:

On Mon, Jun 18, 2012 at 11:55 AM, Alek Storm <alek....@gmail.com> wrote:
> On Sun, Jun 17, 2012 at 12:13 PM, Ben Darnell <b...@bendarnell.com> wrote:
>>
>> I'd say pypy is officially supported (I'd still be cautious about
>> using pypy in prod, but that caution is directed at pypy in general
>> rather than tornado's support for it).
>>
>> It's not hard to get tornado mostly working on jython if there is
>> enough interest, although I think it would be unlikely to graduate to
>> fully-supported status any time soon.  Ironpython has some fundamental
>> problems with its handling of unicode/bytes issues, so I don't think
>> it's suitable for use with tornado without substantial changes on
>> their side.
>
>
> I'm more sympathetic to IronPython's cause; let me see if I can get it
> working with minimal changes in Tornado (Guido officially blessed its
> str==unicode implementation). Could mean a couple more users, who knows?

str==unicode is fine (jython does the same thing, and it's why a lot
of the python 3 changes in tornado are conditioned on "str is unicode"
rather than sys.version[0] == 3).  The problem is that on ironpython,
type(u"".encode("utf8")) == unicode, i.e. there's no way that I can
see to distinguish programmatically between an encoded byte string and
a unicode string that happens to contain only latin1 characters.
There is a bytes type, but it's not always used where it should be
(for example, socket.send only accepts type str).

I often see Jython and IronPython mentioned in the same breath as having str==unicode, but from firing up a Jython shell, I think its behavior has changed in at least 2.5. It's very annoying to deal with, but users can guarantee that strings passed into Tornado will not be accidentally encoded twice by calling, e.g., bytes(u'example'.encode('utf8')) to get a `bytes` instance. Similarly, we will guarantee that strings returned from Tornado are of the correct type.

Anyway, if you'd like to take a look at this and see if there's a
solution that I've missed, here are the secrets to get you past the
first few stumbling blocks.  Start with a jython-compatible branch and
drop this at the top of runtests.py:
+import sys
+if sys.platform == 'cli':
+    # ironpython's signal module is broken under mono
+    sys.modules['signal'] = object()
+    # doctests needs this
+    sys.__displayhook__ = sys.displayhook

and run with "mono ipy.exe -X:Frames"

Thanks!

>> Officially supporting windows would require at least an IOLoop
>> implementation using windows' equivalent of epoll and kqueue.
>
>
> WFM on Windows with Python 2.7 - am I doing something special? Looks like
> Windows does indeed have its own select.select() implementation
> (http://docs.python.org/library/select.html#select.select); it just doesn't
> accept file objects.

Tornado works on windows now, but doesn't scale very well - the limit
on the number of file descriptors for select() is lower than for unix
(although I think it's become somewhat reasonable in modern versions),
and performance would presumably be better with a windows-specific
implementation.  More importantly, this would be a sign that someone
has looked into a windows implementation more deeply than just seeing
that the tests pass and nothing seems obviously broken, which is a key
part of the distinction between "working" and "supported".

What would the criteria be for Windows gaining "supported" status, specifically?

[Ben Darnell]

On Fri, Jun 22, 2012 at 1:52 AM, Alek Storm <alek....@gmail.com> wrote:
>> str==unicode is fine (jython does the same thing, and it's why a lot
>> of the python 3 changes in tornado are conditioned on "str is unicode"
>> rather than sys.version[0] == 3).  The problem is that on ironpython,
>> type(u"".encode("utf8")) == unicode, i.e. there's no way that I can
>> see to distinguish programmatically between an encoded byte string and
>> a unicode string that happens to contain only latin1 characters.
>> There is a bytes type, but it's not always used where it should be
>> (for example, socket.send only accepts type str).
>
>
> I often see Jython and IronPython mentioned in the same breath as having
> str==unicode, but from firing up a Jython shell, I think its behavior has
> changed in at least 2.5. It's very annoying to deal with, but users can

Ah, interesting. I hadn't noticed that; I just assumed that tornado's
py3-compatibility had let it work with jython without any str/unicode
issues.

> guarantee that strings passed into Tornado will not be accidentally encoded
> twice by calling, e.g., bytes(u'example'.encode('utf8')) to get a `bytes`
> instance. Similarly, we will guarantee that strings returned from Tornado
> are of the correct type.

Cool, that seems to work. You should be able to get a lot further
than I did if you put that in tornado.escape.utf8() and
tornado.util.b().

>> >> Officially supporting windows would require at least an IOLoop
>> >> implementation using windows' equivalent of epoll and kqueue.
>> >
>> >
>> > WFM on Windows with Python 2.7 - am I doing something special? Looks
>> > like
>> > Windows does indeed have its own select.select() implementation
>> > (http://docs.python.org/library/select.html#select.select); it just
>> > doesn't
>> > accept file objects.
>>
>> Tornado works on windows now, but doesn't scale very well - the limit
>> on the number of file descriptors for select() is lower than for unix
>> (although I think it's become somewhat reasonable in modern versions),
>> and performance would presumably be better with a windows-specific
>> implementation.  More importantly, this would be a sign that someone
>> has looked into a windows implementation more deeply than just seeing
>> that the tests pass and nothing seems obviously broken, which is a key
>> part of the distinction between "working" and "supported".
>
>
> What would the criteria be for Windows gaining "supported" status,
> specifically?

The main thing is simply someone who wants to support it. :) If you'd
like to take on that responsibility, I think we can easily move
Windows to the "supported for development but not for production"
column (which, to be honest, is what Mac OS should be downgraded to
given the poor networking performance I've seen in Lion). To remove
the "not for production" qualification, I think we'd need need a
native IOLoop and positive reports from someone who's taken the leap
and run in production on windows despite the lack of "official"
support.

-Ben

[Alek Storm]

On Sat, Jun 23, 2012 at 7:22 PM, Ben Darnell <b...@bendarnell.com> wrote:

On Fri, Jun 22, 2012 at 1:52 AM, Alek Storm <alek....@gmail.com> wrote:
>> str==unicode is fine (jython does the same thing, and it's why a lot
>> of the python 3 changes in tornado are conditioned on "str is unicode"
>> rather than sys.version[0] == 3).  The problem is that on ironpython,
>> type(u"".encode("utf8")) == unicode, i.e. there's no way that I can
>> see to distinguish programmatically between an encoded byte string and
>> a unicode string that happens to contain only latin1 characters.
>> There is a bytes type, but it's not always used where it should be
>> (for example, socket.send only accepts type str).
>
>
> I often see Jython and IronPython mentioned in the same breath as having
> str==unicode, but from firing up a Jython shell, I think its behavior has
> changed in at least 2.5. It's very annoying to deal with, but users can

Ah, interesting.  I hadn't noticed that; I just assumed that tornado's
py3-compatibility had let it work with jython without any str/unicode
issues.

> guarantee that strings passed into Tornado will not be accidentally encoded
> twice by calling, e.g., bytes(u'example'.encode('utf8')) to get a `bytes`
> instance. Similarly, we will guarantee that strings returned from Tornado
> are of the correct type.

Cool, that seems to work.  You should be able to get a lot further
than I did if you put that in tornado.escape.utf8() and
tornado.util.b().

Yup, already did, but IronPython's unicode/bytes confusion is much more pervasive than I'd originally thought - most of the standard library only accepts and returns unicode, so I've had to add a lot of, essentially, casts. This also means that some parts of the standard library that were borrowed from CPython don't play well with the unicode-only parts, so monkey-patching is required.

There's also an external dependency on the IronPython.Zlib module (https://bitbucket.org/jdhardy/ironpythonzlib), which is actually excellent - just drop it in, call clr.AddReference(), and go.

This would all be annoying, but workable, if it weren't for IronPython's select() implementation's refusal to notify the caller of a closed connection by setting the read flag on the socket with zero bytes available - it just doesn't happen. The Python wrapper around the .NET Socket.Select() call doesn't do anything funky, so the blame would normally lie with the .NET library, but both the Microsoft and Mono implementations are broken this way. Also, the handshake in SSLIOStream appears to block indefinitely, but I haven't tracked this down yet. There are other things that are broken.

I'll file a couple bugs with the respective projects, but I'm putting my IronPython branch on hold until the more essential pieces of Tornado can actually work.

Yeah, from the reports I've seen, Windows' select() basically exists to mimic the Berkeley API, not to actually give acceptable performance. Someone would have to write an epoll-esque abstraction on top of native Windows calls (and Windows seems to have an endless array of APIs for doing so: http://stackoverflow.com/questions/67082/what-is-the-best-epoll-kqueue-select-equvalient-on-windows, http://curl.haxx.se/mail/lib-2008-08/0292.html), and I'm not exactly a Windows programmer.

Alek