Tornado 5.0 Status

[Ben Darnell]

As we discussed previously (in this thread: https://groups.google.com/d/msg/python-tornado/uqf0DjxILHc/aMSC0481DwAJ), I'm working on improving integration between Tornado and asyncio for 5.0. This initially involved two major changes:

1. IOLoop uses the asyncio event loop by default when asyncio is available

2. Native (async def) coroutines use the asyncio task runner instead of tornado.gen.coroutine

The first part, using the asyncio event loop by default, is done on the master branch now. This involved some changes to the instance() and current() methods to make them match asyncio.get_event_loop() (instance() is now deprecated). This may break some applications that depended on instance() to access the global singleton, but I expect most applications to be fine with this change. For now, it is possible to opt out of the asyncio-by-default change with `IOLoop.configure('tornado.ioloop.PollIOLoop')` (but keep reading).

The coroutine integration is trickier. The first obstacle has to do with the two types tornado.concurrent.Future and asyncio.Future. The asyncio task runner doesn't understand anything but asyncio's own Future class, so we must convert between the two often (which is hard to do efficiently, especially since both classes have magic in their destructors to detect when they go unused). It might be better just use asyncio.Future everywhere (when available). This involves several small changes to things like the timing of callbacks, and one big one: asyncio Futures require an asyncio event loop at construction time, and that event loop must be running when they resolve (I consider this a design flaw in asyncio, but it's too late now). We can make this work, but only if we remove support for other event loops from python 3 completely, and require that asyncio be used whenever it is present (this will be disruptive to pyzmq). I think this is probably the right move but I'm still working through some of the edge cases here.

The other big question has to do with stack_context and legacy callback-based interfaces. The asyncio task runner doesn't know anything about this, so stack_contexts will not be in effect during native coroutines. This doesn't make any difference for ExceptionStackContext, but it would prevent other (thread-local-like) uses. I considered ripping out stack_context completely for this release, but I think there's still probably enough people out there depending on it (often without realizing it! the failure mode of removing stack_context would be subtle) that it's not a good idea. However, to move towards a stack_context-free future (which will improve performance), I'll be deprecating the `callback` arguments throughout the codebase in favor of coroutine-style usage. 

-Ben

[A. Jesse Jiryu Davis]

These are bold, excellent moves. I've just retested Motor against Tornado's HEAD with Python 2.7 and 3.6, and the tests still pass. I'm astonished, given the scope of Tornado changes and how intricately Motor integrates with Tornado's and asyncio's Futures, coroutines, and event loops. So far it seems everything still works.

[Ben Darnell]

If you're feeling adventurous, try the bdarnell/asyncio-future branch (https://github.com/bdarnell/tornado/tree/asyncio-future). That's where the risky changes are at the moment.

--
You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-tornad...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[A. Jesse Jiryu Davis]

I've tried it:

pip install --force-reinstall git+https://github.com/bdarnell/tornado.git@asyncio-future

Again with Python 2.7 and 3.6, all Motor's tests pass. I've confirmed that my tests of the Motor-Tornado integration now use Tornado's AsyncIOLoop. (Motor's tests with Tornado inherit from tornado.testing.AsyncTestCase, whose get_new_ioloop() uses the default loop, which is tornado.platform.asyncio.AsyncIOLoop now.) This is good news for both of us. It looks like I managed to implement Motor without relying on Tornado internals, and you're switching your internals without affecting the API. Interfaces work. =)

To unsubscribe from this group and stop receiving emails from it, send an email to python-tornado+unsubscribe@googlegroups.com.

[Japhy Bartlett]

A really big thank you in general, making things work smoothly with the native async/await is a huge huge help for people who are newish/junior to python3 and tornado, and certainly not the most fun thing I can imagine working on.  

I would push lightly for deprecating callbacks later than sooner, because they are not the worst crutch for people who haven't wrapped their heads around promises (yet).  For someone coming from JavaScript especially, who already groks async but is bad at python, callbacks are a familiar interface.

Just can't say enough though, thank you for choosing this to focus on!

[Ben Darnell]

On Sat, Jun 10, 2017 at 11:02 PM Japhy Bartlett <ja...@pearachute.com> wrote:

I would push lightly for deprecating callbacks later than sooner, because they are not the worst crutch for people who haven't wrapped their heads around promises (yet).  For someone coming from JavaScript especially, who already groks async but is bad at python, callbacks are a familiar interface.

The rise of javascript and increasing familiarity with callback-based programming is, I think, the best argument for preserving the callback-based interfaces. This raises a question: Javascript doesn't have anything like stack_context (as far as I know), so how do people handle errors there? Do you just have to put a try/catch in all your callbacks and propagate errors manually? Or is there some other javascript-inspired error-handling idiom we could provide for callbacks that would be less invasive than stack_context?  

-Ben

[Sam carman]

The tornado 5.0 will be for give up python2.7?

在 2017年6月11日星期日 UTC+8上午1:27:59，Ben Darnell写道：

[Japhy Bartlett]

Yeah usually error handling in JS is extremely tedious (if done at all, especially in the browser), and this is one of the most compelling reasons to start using promises.  The stack trace for a callback would begin with the callback function, without any real context on where it was called from.

The status quo for error handling is so much worse than Python.  

--

[Ben Darnell]

On Sun, Jun 11, 2017 at 7:26 PM Sam carman <pypsuit...@gmail.com> wrote:

The tornado 5.0 will be for give up python2.7?

No, Tornado 5.0 will continue to support Python 2.7. One of my goals for this release is to provide a clean migration path for applications running on Tornado on Python 2.7 to Tornado+asyncio on Python 3. 

-Ben

 

--
You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python-tornad...@googlegroups.com.

[Stuart Longland]

On 12/06/17 10:19, Ben Darnell wrote:
> On Sun, Jun 11, 2017 at 7:26 PM Sam carman <pypsuit...@gmail.com
> <mailto:pypsuit...@gmail.com>> wrote:
>
> The tornado 5.0 will be for give up python2.7?
>
>
> No, Tornado 5.0 will continue to support Python 2.7. One of my goals for
> this release is to provide a clean migration path for applications
> running on Tornado on Python 2.7 to Tornado+asyncio on Python 3.

Out of interest, is there a planned sunset date for support of Python 2.7?

We're using Tornado's IOLoop quite extensively on Python 2.7 in
production, and while Python 3.x is on the road map (and I do want to
move away from Python 2.7), we have to support devices that are
currently running Debian Wheezy which ships Python 2.7 and 3.2.

If anyone is interested, this is one project where we're using Tornado:
http://www.vrt.com.au/industry/projects/lendlease-and-living-utilities-lendlease-barangaroo

There, the Tornado IOLoop forms the core of our communications drivers
talking to M-Bus thermal meters (using libmbus), Modbus devices
(pymodbus) and EDMI Atlas energy meters (using pyserial). Back-end
reporting is done over AMQP.

The devices are ARM-based industrial computers which shipped with Debian
Wheezy. They *can* run Jessie, but at the time, Jessie was still very
new. The boxes basically have the compute power of a desktop computer
from the late 90s: 450MHz ARMv5 CPU, 128MB RAM. Tornado on Python 2.7
works well.

Python 3.2 is nearly useless for compatibility with 2.7 code, and I'm
holding out until these devices are eventually updated to something
newer. Naturally though, I need to make a business case for that. :-)

The move though to transition towards asyncio is a good one. The dev
team working on Tornado have done an excellent job over the years, we've
never had a breaking change since we started with Tornado v4, every
release has *just worked*.
--
Stuart Longland (aka Redhatter, VK4MSL)

I haven't lost my mind...
...it's backed up on a tape somewhere.

[Ben Darnell]

On Sun, Jun 11, 2017 at 8:37 PM Stuart Longland <stu...@longlandclan.id.au> wrote:

On 12/06/17 10:19, Ben Darnell wrote:
> On Sun, Jun 11, 2017 at 7:26 PM Sam carman <pypsuit...@gmail.com
> <mailto:pypsuit...@gmail.com>> wrote:
>
>     The tornado 5.0 will be for give up python2.7?
>
>
> No, Tornado 5.0 will continue to support Python 2.7. One of my goals for
> this release is to provide a clean migration path for applications
> running on Tornado on Python 2.7 to Tornado+asyncio on Python 3.

Out of interest, is there a planned sunset date for support of Python 2.7?

Nothing definite yet. This will be driven in part by usage statistics - Python 2.7 accounted for 77% of Tornado downloads from pypi in the month of May, and this number hasn't changed much over the last year. I'm kind of surprised that we haven't seen more movement towards python 3 (although maybe that will accelerate with ipython going py3-only). 

At some point, though, we're going to want the performance (and ease of development) benefits that come with native coroutines, so we'll want to make the switch regardless of the numbers. I think it's unlikely that we'll be actively supporting python 2 by the time it reaches its official end-of-life in 2020 (unless my crazy idea to use cython to get native coroutines on py2 works out).

So to sum up I'd say we're probably more than a year away from dropping python 2, but less than 3 years. 

Also, since I've just looked at the stats, here are the different versions of Python 3. Based on this, Tornado 5.0 will support Python 3.4+

3.3: 0.2%

3.4: 6.8%

3.5: 8.5%

3.6: 4.9%

 

Python 3.2 is nearly useless for compatibility with 2.7 code, and I'm
holding out until these devices are eventually updated to something
newer.  Naturally though, I need to make a business case for that. :-)

Can't you ship your own python interpreter instead of waiting for them to get upgraded to a whole new operating system? That's also something you'd have to make a business case for, of course, but maybe easier than a whole-system upgrade. 

 

-Ben

[Stuart Longland]

On 12/06/17 11:38, Ben Darnell wrote:
> Python 3.2 is nearly useless for compatibility with 2.7 code, and I'm
> holding out until these devices are eventually updated to something
> newer. Naturally though, I need to make a business case for that. :-)
>
>
> Can't you ship your own python interpreter instead of waiting for them
> to get upgraded to a whole new operating system? That's also something
> you'd have to make a business case for, of course, but maybe easier than
> a whole-system upgrade.

This is an option… and I have considered it.

We basically ship our software as a pile of Debian packages, and where
practical, try to use the standard packages in Debian repositories.
Some, like Tornado, we package up ourselves (thanks to stdeb). (I can
throw our .deb packages for Tornado up somewhere if there's interest.
We track the 'stable' branch.)

Doing it this way, each is individually shipped, with its own license
file, and it is blindingly obvious who owns what file and what license
it is covered by. Standard Debian tools will tell you who owns a given
file, and you then can look for the licensing metadata for that package.
apt looks after dependencies, it all just works.

If we ship our own Python package, we then have to re-package *all* of
the Python dependencies we use. Very time consuming to reconfigure CI
to do that.

Solutions like virtualenv exist, but then everything is shipped as one
big mass, which then becomes a pain to maintain, and I don't feel has
the authors' best interests at heart.

The amount of effort involved, it is easier to send someone down to
Sydney with a stack of MicroSD cards and just swap the OSes out. It'd
be done in a day. :-)

Debian Wheezy will soon be out of support, once the next version is out
(Stretch?). Once that happens, there's a good business case to update
these to Jessie or Stretch, which means Python 3.4 is available to us.

That said, it's good to see that backward compatibility is still
provided, and will be for some time. Of all the open source projects
I've seen and how they handle API stability, Tornado is one I hold in
high regard as it has been painless from day one.

[Antoine Pitrou]

On Sat, 10 Jun 2017 17:27:42 +0000
Ben Darnell <b...@bendarnell.com> wrote:
> As we discussed previously (in this thread:
> https://groups.google.com/d/msg/python-tornado/uqf0DjxILHc/aMSC0481DwAJ),
> I'm working on improving integration between Tornado and asyncio for 5.0.
> This initially involved two major changes:
>
> 1. IOLoop uses the asyncio event loop by default when asyncio is available
> 2. Native (async def) coroutines use the asyncio task runner instead of
> tornado.gen.coroutine

I gave it a quick run on Dask distributed
(https://github.com/dask/distributed/). From the start it's hitting two
pain points that we'll need to work around:
- the loop object doesn't have a `_running` attribute anymore (granted,
this was a private attribute)
- `PeriodicCallback` lost its `io_loop` argument

The latter could be solved by shipping our own PeriodicCallback
implementation. The former will be trickier to solve.

Regards

Antoine.

[Ben Darnell]

On Mon, Jun 12, 2017 at 9:58 AM Antoine Pitrou <soli...@pitrou.net> wrote:

On Sat, 10 Jun 2017 17:27:42 +0000
Ben Darnell <b...@bendarnell.com> wrote:
> As we discussed previously (in this thread:
> https://groups.google.com/d/msg/python-tornado/uqf0DjxILHc/aMSC0481DwAJ),
> I'm working on improving integration between Tornado and asyncio for 5.0.
> This initially involved two major changes:
>
> 1. IOLoop uses the asyncio event loop by default when asyncio is available
> 2. Native (async def) coroutines use the asyncio task runner instead of
> tornado.gen.coroutine

I gave it a quick run on Dask distributed
(https://github.com/dask/distributed/).  From the start it's hitting two
pain points that we'll need to work around:
- the loop object doesn't have a `_running` attribute anymore (granted,
  this was a private attribute)

I think that you probably don't want to do checks like this. There's discussion of this in https://github.com/tornadoweb/tornado/issues/2070

 

- `PeriodicCallback` lost its `io_loop` argument

The latter could be solved by shipping our own PeriodicCallback
implementation. 

Couldn't you just stop passing that argument? Were you ever passing a value other than IOLoop.current()? Why?

-Ben

 

The former will be trickier to solve. 

Regards

Antoine.

[Antoine Pitrou]

On Mon, 12 Jun 2017 16:19:58 +0000
Ben Darnell <b...@bendarnell.com> wrote:
>
> I think that you probably don't want to do checks like this. There's
> discussion of this in https://github.com/tornadoweb/tornado/issues/2070

Well, checking whether a given IOLoop object is running is a primitive
already exposed by other event loop-driven frameworks such as asyncio
and Twisted.

See https://github.com/tornadoweb/tornado/pull/2085

> > The latter could be solved by shipping our own PeriodicCallback
> > implementation.
>
> Couldn't you just stop passing that argument? Were you ever passing a value
> other than IOLoop.current()? Why?

We're initializing a PeriodicCallback from potentially a different
thread than the thread the event loop will be running in.

Regards

Antoine.

[MinRK]

On Sat, Jun 10, 2017 at 7:27 PM, Ben Darnell <b...@bendarnell.com> wrote:

As we discussed previously (in this thread: https://groups.google.com/d/msg/python-tornado/uqf0DjxILHc/aMSC0481DwAJ), I'm working on improving integration between Tornado and asyncio for 5.0. This initially involved two major changes:

1. IOLoop uses the asyncio event loop by default when asyncio is available

2. Native (async def) coroutines use the asyncio task runner instead of tornado.gen.coroutine

The first part, using the asyncio event loop by default, is done on the master branch now. This involved some changes to the instance() and current() methods to make them match asyncio.get_event_loop() (instance() is now deprecated). This may break some applications that depended on instance() to access the global singleton, but I expect most applications to be fine with this change. For now, it is possible to opt out of the asyncio-by-default change with `IOLoop.configure('tornado.ioloop.PollIOLoop')` (but keep reading).

The coroutine integration is trickier. The first obstacle has to do with the two types tornado.concurrent.Future and asyncio.Future. The asyncio task runner doesn't understand anything but asyncio's own Future class, so we must convert between the two often (which is hard to do efficiently, especially since both classes have magic in their destructors to detect when they go unused). It might be better just use asyncio.Future everywhere (when available). This involves several small changes to things like the timing of callbacks, and one big one: asyncio Futures require an asyncio event loop at construction time, and that event loop must be running when they resolve (I consider this a design flaw in asyncio, but it's too late now). We can make this work, but only if we remove support for other event loops from python 3 completely, and require that asyncio be used whenever it is present (this will be disruptive to pyzmq). I think this is probably the right move but I'm still working through some of the edge cases here.

FWIW, I’ve given up on waiting for level-triggered FDs from libzmq, and am working on removing the need for injecting zmq_poll from pyzmq, so hopefully pyzmq 17 won’t be affected by these changes and should be able to sit on any asyncio/tornado implementation. (this is partly motivated by your 5.0 plans, so thanks for keeping us updated!).

The other big question has to do with stack_context and legacy callback-based interfaces. The asyncio task runner doesn't know anything about this, so stack_contexts will not be in effect during native coroutines. This doesn't make any difference for ExceptionStackContext, but it would prevent other (thread-local-like) uses. I considered ripping out stack_context completely for this release, but I think there's still probably enough people out there depending on it (often without realizing it! the failure mode of removing stack_context would be subtle) that it's not a good idea. However, to move towards a stack_context-free future (which will improve performance), I'll be deprecating the `callback` arguments throughout the codebase in favor of coroutine-style usage. 

-Ben

--
You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python-tornado+unsubscribe@googlegroups.com.

​

[Ben Darnell]

On Sun, Jun 11, 2017 at 7:28 PM Japhy Bartlett <ja...@pearachute.com> wrote:

Yeah usually error handling in JS is extremely tedious (if done at all, especially in the browser), and this is one of the most compelling reasons to start using promises.  The stack trace for a callback would begin with the callback function, without any real context on where it was called from.

The status quo for error handling is so much worse than Python.  

This is exactly my concern. I don't want to support a callback-based interface if it just encourages error-prone patterns. The consequences of poor exception handling are more severe in a server-side context than in a browser context, since the server could leak memory or sockets over many requests. 

I think it would be a mistake to remove stack_context while the callback-based APIs are still in use; this would just lead to problems for everybody who was (perhaps unknowingly) relying on the exception safety net that stack_context provides. At some point we're going to need to stop paying the performance price for supporting a programming style that is no longer encouraged.

I think that for Tornado 5.0, I'll just deprecate most of the callback-oriented interfaces without removing them. But I may remove callback arguments completely from some lower-level interfaces like IOStream (and maybe the locks and queues modules) so that they don't have to pay the stack_context price unnecessarily. 

-Ben

 

On Sun, Jun 11, 2017 at 10:28 AM, Ben Darnell <b...@bendarnell.com> wrote:

On Sat, Jun 10, 2017 at 11:02 PM Japhy Bartlett <ja...@pearachute.com> wrote:

I would push lightly for deprecating callbacks later than sooner, because they are not the worst crutch for people who haven't wrapped their heads around promises (yet).  For someone coming from JavaScript especially, who already groks async but is bad at python, callbacks are a familiar interface.

The rise of javascript and increasing familiarity with callback-based programming is, I think, the best argument for preserving the callback-based interfaces. This raises a question: Javascript doesn't have anything like stack_context (as far as I know), so how do people handle errors there? Do you just have to put a try/catch in all your callbacks and propagate errors manually? Or is there some other javascript-inspired error-handling idiom we could provide for callbacks that would be less invasive than stack_context?  

-Ben

--

You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python-tornad...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python-tornad...@googlegroups.com.