Tornado 5.0 update: possible end of Python 2.7 support

[Ben Darnell]

I've just merged the last of the three key asyncio integration changes planned for Tornado 5.0:

- Use AsyncIOLoop by default

- Reconcile tornado.concurrent.Future and asyncio.Future

- Use asyncio.Task when running `async def` coroutines

Unfortunately, the Future changes proved more invasive than expected. Due to a small but crucial difference in the two Future classes (asyncio.Future knows about the event loop, t.c.Future does not), I wasn't able to make this optional: if you're on Python 3, we have to make t.c.Future an alias for asyncio.Future (and therefore it's no longer possible to configure alternative IOLoops on Python 3).

This means that we have an increasing number of areas in the code where we have one implementation on python 2 and another for python 3. While I've tried to keep them as compatible as possible, there's enough room for different behavior that if I had an app using Python 2.7 and Tornado 4.5 and wanted to upgrade both, I'd try to go to python 3 while still on tornado 4.5 instead of the other way around.

I had envisioned Tornado 5.x as the last transitional release(s) before going python 3-only, but this suggests that maybe Tornado 4.5 makes the transition as smooth as it's ever going to be and the changes in 5.0 make it the right time to go 3-only.

If we drop python 2 support in tornado 5.0, we'll be able to delete a bunch of code (including large parts of tornado.ioloop, tornado.concurrent, and tornado.platform), and generally simplify things. We'd also be able to use async/await internally (assuming we set the minimum at python 3.5), which would be a performance win. 

On the other hand, 81% of our downloads still come from python 2.7 according to pypi statistics (that's actually higher than last time I looked - 2.7 has grown more in the last two months than all python 3 versions combined). And I think one of Tornado's competitive strengths is that we span the 2/3 divide well. 

I haven't made up my mind yet about this question. But anyway, these changes are ready to get some more exposure and testing. If you're interested, please try your application with the master branch and report back if you run into any issues (and a big thanks to those of you who already had). I'll work on some release notes to point out known issues to watch out for. 

-Ben

[Pierce Lopez]

Perhaps, if Python 2.7 compatibility is dropped in Tornado 5.x, it would make sense to maintain Tornado 4.5 in parallel for a while (with bugfix-only patch releases)? That might split the userbase somewhat, but on the other hand it could keep Python 2.7 users in a good situation while clearly indicating they should find a good time to do the big transition.

At my work we're all Python 3.5 by now. But also, I tend to like the idea of stable patch releases - I like to not be rushed to validate and adapt to major upgrades. (But I think I'm reasonable - we're on python 3.5 aren't we.)

Anyway, that's my 2c: I think dropping python-2.7 support in tornado-5 would be reasonable if tornado-4.5 were maintained for a while too.

- Pierce

--
You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-tornado+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Aliane Abdenour]

maybe that 4.x will be a LTS while 5.x will continue untill 2.7 will be stopped from users, this will happen within 4 years, the same time the LTS will be updated?

[Antoine Pitrou]

On Sun, 10 Dec 2017 03:42:22 +0000
Ben Darnell <b...@bendarnell.com> wrote:
>
> On the other hand, 81% of our downloads still come from python 2.7
> according to pypi statistics (that's actually higher than last time I
> looked - 2.7 has grown more in the last two months than all python 3
> versions combined).

PyPI statistics are difficult to base a judgement on, since an unknown
part of those numbers come from automated CI services.

> And I think one of Tornado's competitive strengths is
> that we span the 2/3 divide well.

Agreed, this is one of the main reasons to go to Tornado as far as I'm
concerned (though ability to easily submit patches turned out to be
another strong point :-)).

> I haven't made up my mind yet about this question. But anyway, these
> changes are ready to get some more exposure and testing. If you're
> interested, please try your application with the master branch and report
> back if you run into any issues (and a big thanks to those of you who
> already had).

I'll run the distributed test suite again ASAP.

Regards

Antoine.

[Matthew Rocklin]

As a downstream library developer who continues to support Python 2/3 I would remain relatively unconcerned by Tornado moving to Python 3 if the user API and semantics between 4.5 and 5.x remained somewhat stable when using Python 2 features, at least for a while.  I think that it's a good thing for Tornado to evolve without the encumbrance of Python 2, but that maintaining API stability for a while would be convenient.

Also, there may be some concerns when publishing new Python 3-only libraries on PyPI when Python 2-compatible libraries exist.  I know that Jupyter went through this a while ago and had to be slightly clever to avoid situations where Python 2 users downloaded the Python 3 source code.

[Ben Darnell]

On Sun, Dec 10, 2017 at 8:33 AM Matthew Rocklin <mroc...@gmail.com> wrote:

As a downstream library developer who continues to support Python 2/3 I would remain relatively unconcerned by Tornado moving to Python 3 if the user API and semantics between 4.5 and 5.x remained somewhat stable when using Python 2 features, at least for a while.  I think that it's a good thing for Tornado to evolve without the encumbrance of Python 2, but that maintaining API stability for a while would be convenient.

I'm not planning to make any breaking changes to existing APIs, so as long as you stick to documented interfaces it should be feasible to support tornado 4.x on python 2 and tornado 5.x on python 3 simultaneously. 

Then, to use a couple of recent changes by you and Antoine as examples, you'd get the benefit of the improved IOStream.write() performance for free on tornado 5, but to use IOStream.read_into() (still pending, but let's assume it gets merged into 5.0) you'd need a version switch in your code (unless it gets backported to a 4.6 release)

 

Also, there may be some concerns when publishing new Python 3-only libraries on PyPI when Python 2-compatible libraries exist.  I know that Jupyter went through this a while ago and had to be slightly clever to avoid situations where Python 2 users downloaded the Python 3 source code.

Do you know what they did? I see a warning in setup.py[0] and they set python_requires (which is understood by recent versions of pip). Is there anything else?

[0] https://github.com/ipython/ipython/blob/cefab40bb755260ddec63efc24d8e1ea3b6fba06/setup.py#L25-L58

 

On Sunday, December 10, 2017 at 7:15:53 AM UTC-5, Antoine Pitrou wrote:

On Sun, 10 Dec 2017 03:42:22 +0000

Ben Darnell <b...@bendarnell.com> wrote:
>
> On the other hand, 81% of our downloads still come from python 2.7
> according to pypi statistics (that's actually higher than last time I
> looked - 2.7 has grown more in the last two months than all python 3
> versions combined).

PyPI statistics are difficult to base a judgement on, since an unknown
part of those numbers come from automated CI services.

True. And it's possible there's a bias in the data if python 3 projects are more likely to use modern versions of pip and other tools and set up caching properly so they're not hitting pypi every time. 

For another data point (but a much smaller sample size), Debian's popularity contest shows twice as many installations on python 2 as on python 3, and python 3 is growing rapidly: https://qa.debian.org/popcon.php?package=python-tornado

Anecdotally, I think a large majority of issues filed, stack overflow questions, etc, are still using python 2 (where that can be distinguished, usually either because the code contains print statements or a stack trace shows a python version number). 

-Ben

 

[Ben Darnell]

On Sun, Dec 10, 2017 at 4:01 AM Aliane Abdenour <alabde...@gmail.com> wrote:

maybe that 4.x will be a LTS while 5.x will continue untill 2.7 will be stopped from users, this will happen within 4 years, the same time the LTS will be updated?

The problem with an LTS release is that it requires someone to do the Support over a Long Term. The great thing about dropping support for python 2.7 is that I get to stop caring about that version :)

If 5.0 drops support for python 2, I'd be somewhat more likely to do bugfix releases to 4.5, and possibly even new features in a 4.6, but the time that I spend on tornado will be focused on the current version, so any backports would need to be driven by more community involvement.

Fortunately, Tornado 4.5 is a mature package, so even if it doesn't get additional updates I think it can serve python 2 users well for the remainder of python 2's supported lifetime. 

-Ben

 

On Dec 10, 2017 06:09, "Pierce Lopez" <pierce...@gmail.com> wrote:

Perhaps, if Python 2.7 compatibility is dropped in Tornado 5.x, it would make sense to maintain Tornado 4.5 in parallel for a while (with bugfix-only patch releases)? That might split the userbase somewhat, but on the other hand it could keep Python 2.7 users in a good situation while clearly indicating they should find a good time to do the big transition.

At my work we're all Python 3.5 by now. But also, I tend to like the idea of stable patch releases - I like to not be rushed to validate and adapt to major upgrades. (But I think I'm reasonable - we're on python 3.5 aren't we.)

Anyway, that's my 2c: I think dropping python-2.7 support in tornado-5 would be reasonable if tornado-4.5 were maintained for a while too.

- Pierce

On Sat, Dec 9, 2017 at 10:42 PM, Ben Darnell <b...@bendarnell.com> wrote:

I've just merged the last of the three key asyncio integration changes planned for Tornado 5.0:

- Use AsyncIOLoop by default

- Reconcile tornado.concurrent.Future and asyncio.Future

- Use asyncio.Task when running `async def` coroutines

Unfortunately, the Future changes proved more invasive than expected. Due to a small but crucial difference in the two Future classes (asyncio.Future knows about the event loop, t.c.Future does not), I wasn't able to make this optional: if you're on Python 3, we have to make t.c.Future an alias for asyncio.Future (and therefore it's no longer possible to configure alternative IOLoops on Python 3).

This means that we have an increasing number of areas in the code where we have one implementation on python 2 and another for python 3. While I've tried to keep them as compatible as possible, there's enough room for different behavior that if I had an app using Python 2.7 and Tornado 4.5 and wanted to upgrade both, I'd try to go to python 3 while still on tornado 4.5 instead of the other way around.

I had envisioned Tornado 5.x as the last transitional release(s) before going python 3-only, but this suggests that maybe Tornado 4.5 makes the transition as smooth as it's ever going to be and the changes in 5.0 make it the right time to go 3-only.

If we drop python 2 support in tornado 5.0, we'll be able to delete a bunch of code (including large parts of tornado.ioloop, tornado.concurrent, and tornado.platform), and generally simplify things. We'd also be able to use async/await internally (assuming we set the minimum at python 3.5), which would be a performance win. 

On the other hand, 81% of our downloads still come from python 2.7 according to pypi statistics (that's actually higher than last time I looked - 2.7 has grown more in the last two months than all python 3 versions combined). And I think one of Tornado's competitive strengths is that we span the 2/3 divide well. 

I haven't made up my mind yet about this question. But anyway, these changes are ready to get some more exposure and testing. If you're interested, please try your application with the master branch and report back if you run into any issues (and a big thanks to those of you who already had). I'll work on some release notes to point out known issues to watch out for. 

-Ben

--
You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python-tornad...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python-tornad...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python-tornad...@googlegroups.com.

[Matthew Rocklin]

> Do you know what they did? I see a warning in setup.py[0] and they set python_requires (which is understood by recent versions of pip). Is there anything else?

Judging from this talk python_requires was the main point.  It might be worth skimming through though.

https://youtu.be/2DkfPzWWC2Q?t=594

https://us.pycon.org/2017/schedule/presentation/319/

- Pierce

To unsubscribe from this group and stop receiving emails from it, send an email to python-tornado+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python-tornado+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python-tornado+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Tornado Web Server" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-tornado/pErgXjAR77s/unsubscribe.
To unsubscribe from this group and all its topics, send an email to python-tornado+unsubscribe@googlegroups.com.

[Ben Darnell]

On Sat, Dec 9, 2017 at 10:42 PM Ben Darnell <b...@bendarnell.com> wrote:

If we drop python 2 support in tornado 5.0, we'll be able to delete a bunch of code (including large parts of tornado.ioloop, tornado.concurrent, and tornado.platform), and generally simplify things. We'd also be able to use async/await internally (assuming we set the minimum at python 3.5), which would be a performance win. 

 I did a quick experiment with this and found that replacing all the gen.coroutines in tornado with async/await results in a 16% increase in requests per second for a simple benchmark. 

-Ben

[Gavin M. Roy]

I think it's time for Python 3.5+ only and welcome the change.

On Sat, Dec 9, 2017 at 10:42 PM, Ben Darnell <b...@bendarnell.com> wrote:

--

[Antoine Pitrou]

On Sun, 10 Dec 2017 13:15:40 +0100
Antoine Pitrou <soli...@pitrou.net> wrote:
>
> > I haven't made up my mind yet about this question. But anyway, these
> > changes are ready to get some more exposure and testing. If you're
> > interested, please try your application with the master branch and report
> > back if you run into any issues (and a big thanks to those of you who
> > already had).
>
> I'll run the distributed test suite again ASAP.

Things seem mostly ok here except for a single test failure (which I
suspect we'll be able to fix on our side), both on Python 2.7 and 3.6.

Regards

Antoine.

[A. Jesse Jiryu Davis]

Motor on master passes its tests with Tornado installed from master. I'll release the next Motor, version 1.2, in the next week or so. It has new features to support the new MongoDB 3.6, which we released last week.

If Tornado 5 drops Python 2, that would motivate me to drop Python 2 in Motor as well. The benefits for Motor in speed and simplicity if I used "async" and "await" throughout would probably be even better than they are for Tornado itself. Motor users tend to use "async for" loops over MongoDB documents. I have to implement "async for" very inefficiently because the underlying I/O code must be implemented with complicated callbacks to work in Python 2 and 3.

Furthermore, since Motor supports asyncio and Tornado, I currently have to abstract away the various inconsistencies, like "yield" versus "yield from". If I used "async" and "await" everywhere, most of that trouble would go away.

Regards

Antoine.

[Stuart Longland]

On 10/12/17 13:42, Ben Darnell wrote:
> If we drop python 2 support in tornado 5.0, we'll be able to delete a
> bunch of code (including large parts of tornado.ioloop,
> tornado.concurrent, and tornado.platform), and generally simplify
> things. We'd also be able to use async/await internally (assuming we set
> the minimum at python 3.5), which would be a performance win. 

There are a lot of Linux distributions that ship with Python 3.4, not
3.5. Debian Jessie being one.

Installing Python 3.5 is non-trivial as then *every* Python package
dependency must be re-compiled instead of making use of standard Debian
repository packages.

At my workplace we not long ago made the decision to drop Debian Wheezy
support, which means we were no longer stuck with Python 2.7 and the
near useless 3.2.

While I'd like to move to Debian Stretch which ships 3.5, it isn't my
decision and I'd have to justify the effort on business grounds.
--
Stuart Longland (aka Redhatter, VK4MSL)

I haven't lost my mind...
...it's backed up on a tape somewhere.

[Antoine Pitrou]

On Mon, 18 Dec 2017 08:32:20 +1000
Stuart Longland

<stu...@longlandclan.id.au> wrote:
> On 10/12/17 13:42, Ben Darnell wrote:
> > If we drop python 2 support in tornado 5.0, we'll be able to delete a
> > bunch of code (including large parts of tornado.ioloop,
> > tornado.concurrent, and tornado.platform), and generally simplify
> > things. We'd also be able to use async/await internally (assuming we set
> > the minimum at python 3.5), which would be a performance win. 
>
> There are a lot of Linux distributions that ship with Python 3.4, not
> 3.5. Debian Jessie being one.
>
> Installing Python 3.5 is non-trivial as then *every* Python package
> dependency must be re-compiled instead of making use of standard Debian
> repository packages.

Or you can use existing binary distributions such as Anaconda (or
Miniconda + conda-forge).

Regards

Antoine.

[Stuart Longland]

… no we can't because those require x86 or Power64 CPUs and a lot of our
fleet runs ARMv5.

[Matthew Rocklin]

One of the conda forge/anaconda maintainers maintains a distribution of ARM packages. https://github.com/jjhelmus/berryconda 

I think that there is also a talk about this at last year's as scipy conference if you want to look on YouTube. 

--
You received this message because you are subscribed to a topic in the Google Groups "Tornado Web Server" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-tornado/pErgXjAR77s/unsubscribe.
To unsubscribe from this group and all its topics, send an email to python-tornado+unsubscribe@googlegroups.com.

[Stuart Longland]

On 22/12/17 09:43, Matthew Rocklin wrote:
> One of the conda forge/anaconda maintainers maintains a distribution of
> ARM packages. https://github.com/jjhelmus/berryconda 

Yeah, if you look: it's ARMv6 and ARMv7… not ARMv5. ARMv6/v7 has a
hardware floating-point available; ARMv5 does not. These binaries will
not work.

Sadly, as much as I'd like to go to something faster, there aren't many
machines in the sub-AU$200 price bracket that come in a DIN-rail mount
case, with RS-232 and 485 ports, a DC-DC power supply with a 9-30V input
power rating and hardware able to operate at 85°C.

The BeagleBone Industrial comes close, but is sold as a bare board with
bare UART ports. The devices we use now are built around Freescale
i.MX286s with 128MB RAM… and we have a largish fleet of them.

There's a dozen of these polling M-Bus and Modbus devices in the
residential towers at Barangaroo. They've been there for a couple of
years now, and will be there for years to come. Supporting them will be
partly my job for the next few years.

Tornado is a fantastic library for writing tools that perform all kinds
of industrial data acquisition functions. For sure, we can just pin it
at Tornado 4.5 and lock that in long-term, but then again, why shut
people out because they're not running the absolute bleeding edge?

[Antoine Pitrou]

On Fri, 22 Dec 2017 21:02:22 +1000
Stuart Longland

<stu...@longlandclan.id.au> wrote:
> On 22/12/17 09:43, Matthew Rocklin wrote:
> > One of the conda forge/anaconda maintainers maintains a distribution of
> > ARM packages. https://github.com/jjhelmus/berryconda 
>
> Yeah, if you look: it's ARMv6 and ARMv7… not ARMv5. ARMv6/v7 has a
> hardware floating-point available; ARMv5 does not. These binaries will
> not work.
>
> Sadly, as much as I'd like to go to something faster, there aren't many
> machines in the sub-AU$200 price bracket that come in a DIN-rail mount
> case, with RS-232 and 485 ports, a DC-DC power supply with a 9-30V input
> power rating and hardware able to operate at 85°C.

That's an interesting use case. In any case, you're right, there are
not many binary distributions which beat Debian's cross-platform
support :-)

Best regards

Antoine.

[Ben Darnell]

On Fri, Dec 22, 2017 at 6:02 AM Stuart Longland <stu...@longlandclan.id.au> wrote:

Tornado is a fantastic library for writing tools that perform all kinds
of industrial data acquisition functions.  For sure, we can just pin it
at Tornado 4.5 and lock that in long-term, but then again, why shut
people out because they're not running the absolute bleeding edge?

You're not getting "shut out" of anything but more of the bleeding edge: if Python 3.5, released in 2015, is "absolute bleeding edge", then surely Tornado 5.0, to be released in 2018 is even more so.  And "shut out" is an overstatement - you always have the option to build Python 3.5+ from source if you decide you want to upgrade to a newer 3.5-only version of Tornado while you're still on debian jessie. Until then, staying on Tornado 4.5 is a perfectly reasonable option (or even Tornado 3.2.2, which is the version packaged in jessie. If you're using 4.5 on jessie you're already going beyond debian packages for at least some things).

As for "why", the advantages of Python 3.5 for asynchronous applications (the async/await keywords) are pretty significant, and I'd like to be able to take advantage of them in Tornado itself. Requiring 3.5 will leave some users behind, but there are far fewer people stuck on 3.4 than on 2.7, so when we make the decision to drop 2.7, it's easy to set the next minimum at 3.5 (where it will probably stay for a long time). 

That said, I'm now leaning towards keeping 2.7 support in Tornado 5.0. Then in 5.1 I'll add a bunch of DeprecationWarnings, and finally remove stuff in 6.0. This will probably go beyond dropping python 2.7 - things like StackContext and older callback-based interfaces are likely to be removed as well. 

-Ben