Tulip third-party projects and Trollius

[Victor Stinner]

Hi,

Tulip uses "yield from" whereas this instruction is new in Python 3.3
and cannot be used on Python 2, so Trollius uses "yield" instead. The
problem is that third-party projects like aiohttp uses "yield from".

On the OpenStack mailing list, the following question was asked:
"How useful is trollius if we can't use other thirdparty libraries
written for asyncio?"

Would it be possible to support CPython 3.4 / Tulip (Python 3.3) /
Trollius (Python 2.6/2.7, but works also on Python 3) in the same code
base without duplicating almost all the code?

Trollius is currently a branch of Tulip. It's not easy to maintain it
(but I do maintain it, it's possible ;-)). I'm not sure that each
third-party project would like its own Trollius branch.

I don't know if it would be possible to replace "yield from" with
"yield" at runtime, for example when a module is imported. The Python
2 parser cannot be used for that, because "yield from" raises a syntax
error.

It may be possible to write a script replacing "yield from " with
"yield " directly in the source code, something like 2to3, but I don't
like this option.

Any idea?

Note: porting Python 3 code requires a little bit more work than just
replacing "yield from" with "yield". For example, Python 2 doesn't
support definition of functions with keyword-only parameters, doesn't
support super() (with no parameter) and use old-style class by default
(your classes have to explicitly inherit from object).

Victor

[Yury Selivanov]

Victor,

On 2/20/2014, 6:33 AM, Victor Stinner wrote:
> Hi,
>
> Tulip uses "yield from" whereas this instruction is new in Python 3.3
> and cannot be used on Python 2, so Trollius uses "yield" instead. The
> problem is that third-party projects like aiohttp uses "yield from".
>
> On the OpenStack mailing list, the following question was asked:
> "How useful is trollius if we can't use other thirdparty libraries
> written for asyncio?"

Guido once proposed to use 'yield From (fut)' syntax in Trollius.
I think that it was a really great proposition.

Pros:

- Easy to port projects from trollius (python2) to asyncio (python3).
I have a huge experience of writing source code analyzers, and I think
that it will be fairly easy to develop an automatic conversion tool,
that will automatically update 'yield From ()' syntax to 'yield from'.
It should be possible to reuse 2to3 infrastructure (trollius fixer).

Even without using a tool you can simply run 'replace all' command
in your editor.

- Relatively easy to do the reverse port of projects on asyncio to
trollius. Again, automated tool, that goes through the code, and
updates 'yield from' with 'yield From ()' in @coroutines. This one
will be a bit harder to write, but again, entirely possible.

- Once you have your library from asyncio ported to the trollius,
you can still see all places where 'yield from' was. Hence -- easy
debugging.

Cons:

- I don't see any, really. 'From' wrappers will have a small
performance cost, but I'm sure it will be negligible.



Yury

[Antoine Pitrou]

On Thu, 20 Feb 2014 11:30:21 -0500
Yury Selivanov <yseliv...@gmail.com>
wrote:

>
> Cons:
>
> - I don't see any, really. 'From' wrappers will have a small
> performance cost, but I'm sure it will be negligible.

From what I understand, you still can't write a library that works with
both trollius and asyncio.

A possibility for a 2.x- and 3.x-compatible library is to use callback
style rather than coroutine style. It may even allow you to be
compatible with Tornado.

Regards

Antoine.

[Yury Selivanov]

Right, you still can't do it in a straightforward way.
But you can write a tool to convert python3 'yield from'
to something that Trollius can work with (be it plain 'yield'
or 'yield From'). Writing such tool will require:

- parsing the python code (doable)

- writing an AST analyzer to pick functions defined with
@coroutine (doable)

- writing an AST transformer to replace 'yield from'
with 'yield' or 'yield From' (very doable, once you have
two first items)

- dumping the transformed code back

[I'm not sure if 3to2 can be reused for this].

My proposal is two-fold:

1. A tool (set of tools) to convert native asyncio code
in Trollius code;

2. Using 'yield From' in Trollius to ease migration to
python3 and asyncio in the future; and to ease debugging of
code ported from asyncio to Trollius.

Yury

[Guido van Rossum]

The source transformation route has not been very popular with Python 2 vs. 3 porting projects.

I wonder if it would be possible (using asyncio internals) to write a From() function that lets you write "yield From(x)" instead of "yield from x" in Tulip?

--
--Guido van Rossum (python.org/~guido)

[Yury Selivanov]

On 2/20/2014, 12:54 PM, Guido van Rossum wrote:
> The source transformation route has not been very popular with Python 2 vs.
> 3 porting projects.

It's not popular, but nevertheless, many use it just fine.

> I wonder if it would be possible (using asyncio internals) to write a
> From() function that lets you write "yield From(x)" instead of "yield from
> x" in Tulip?

I think it's doable. Although, I'm not sure we should include
this in asyncio, as we either should encourage people to use
'yield From' in asyncio and python3, or we'll end up with half
code using one form, and half using another.

Yury

[Guido van Rossum]

It should not be part of asyncio, but a separate library on PyPI, like six.

[Luca Sbardella]

On 20 February 2014 11:33, Victor Stinner <victor....@gmail.com> wrote:

Hi,

Tulip uses "yield from" whereas this instruction is new in Python 3.3
and cannot be used on Python 2, so Trollius uses "yield" instead. The
problem is that third-party projects like aiohttp uses "yield from".

On the OpenStack mailing list, the following question was asked:
"How useful is trollius if we can't use other thirdparty libraries
written for asyncio?"

In pulsar 0.8 (not released yet! requires asyncio/trollius) I use a specialised asyncio Task class which is tolerant to yielded generators.

The client can do both yield or yield from and it works fine (yield ~30% slower on some basic benchmarks)

Look for the Task class and async function in here for details

https://github.com/quantmind/pulsar/blob/yieldfrom/pulsar/async/futures.py

 

Would it be possible to support CPython 3.4 / Tulip (Python 3.3) /
Trollius (Python 2.6/2.7, but works also on Python 3) in the same code
base without duplicating almost all the code?

maybe setting a task_factory in the event loop?

Pulsar creates tasks from the eventloop task_factory callable if it exists, otherwise reverts to the standard asyncio Task.

In this way switching task class can be done via the current asyncio api by setting a new event loop policy.

Trollius is currently a branch of Tulip. It's not easy to maintain it
(but I do maintain it, it's possible ;-)). I'm not sure that each
third-party project would like its own Trollius branch.

I thought about that but sensationally I came to the right decision, no.

Asyncio does exactly what I needed and thanks to trollius I was able to remove the whole async internals in pulsar and concentrate on the apps.

Your effort is very much appreciated.

 

I don't know if it would be possible to replace "yield from" with
"yield" at runtime, for example when a module is imported. The Python
2 parser cannot be used for that, because "yield from" raises a syntax
error.

It may be possible to write a script replacing "yield from " with
"yield " directly in the source code, something like 2to3, but I don't
like this option.

 

ditto