Creation of the new Trollius project

[Victor Stinner]

Hi,

I created a new Trollius project, port of the Tulip project on Python 2.7:
https://bitbucket.org/haypo/trollius/

Trollius is very close to my previous "tulip_py2" project which was
more a proof of concept, but the code is a little bit cleaner. The
Mercurial repository is now a clone of Tulip repository, all changes
are in a new "trollius" branch.

In my opinion, changes should first be made in CPython, then reported
to Tulip and finally to Trollius (CPython asyncio => Tulip =>
Trollius). I may be possible to modify first Tulip (Tulip => CPython
and Tulip => Trollius), but it requires to compare more carefully
changes (ex: using meld tool).

Some Trollius unit tests are still failing, and I didn't touch
examples yet (examples don't work).

Differences between Trollius and Tulip:

* Python 2 has no keyword-only parameters
* Trollius coroutines use yield whereas Tulip coroutines must use yield-from
* Trollius coroutines must use "raise Return(value)", whereas Tulip simply
uses "return value" thanks to Python 3.3 improvements
* If the concurrent.futures module is missing, BaseEventLoop.run_in_executor()
raises NotImplementedError and asyncio.wrap_future() is not available

See commits to see all differences between Trollius and Tulip:
https://bitbucket.org/haypo/trollius/commits/all

Changes between tulip_v2 and Trollius:

- Trollius repository is a clone of Tulip repository, so it will be
easier to update it (it is possible to use "hg merge")
- Trollius has an implementation of time.monotonic() working on
Windows, Mac OS X, Linux, FreeBSD, OpenBSD, Solaris
- test_support.py is now in asyncio/ directory
- no more concurrent.futures backport: I copied concurrent.futures
classes and constants to not depend on it. I plan to reuse the
existing concurrent.futures backport for EventLoop.run_in_executor()
(which currently raises a NotImplementError if the module is missing).
- selectors.SelectSelector also uses wrap_error() to get an InterruptedError

Victor

[Tobias Oberstein]

Hi Victor,

Is it supposed to work on PyPy?


Traceback (most recent call last):
File "app_main.py", line 72, in run_toplevel
File "server.py", line 19, in <module>
from autobahn.asyncio.websocket import WebSocketServerProtocol, \
File
"/home/oberstet/scm/AutobahnPython/autobahn/autobahn/asyncio/websocket.py",
line 26, in <module>
import asyncio
File
"/home/oberstet/pypy-2.2.1-linux64/site-packages/asyncio/__init__.py",
line 9, in <module>
from . import selectors
File
"/home/oberstet/pypy-2.2.1-linux64/site-packages/asyncio/selectors.py",
line 13, in <module>
from asyncio.backport import wrap_error
File
"/home/oberstet/pypy-2.2.1-linux64/site-packages/asyncio/backport.py",
line 30, in <module>
__builtins__['InterruptedError'] = InterruptedError
TypeError: 'module' object does not support item assignment


Probably this ("wont fix" issue): https://bugs.pypy.org/issue876


This is on

oberstet@vbox-ubuntu1310:~/scm/AutobahnPython/examples/asyncio/websocket/echo$
pypy -V
Python 2.7.3 (87aa9de10f9c, Nov 24 2013, 18:48:13)
[PyPy 2.2.1 with GCC 4.6.3]
oberstet@vbox-ubuntu1310:~/scm/AutobahnPython/examples/asyncio/websocket/echo$
uname -a
Linux vbox-ubuntu1310 3.11.0-15-generic #23-Ubuntu SMP Mon Dec 9
18:17:04 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

[Tobias Oberstein]

Hi Victor,

I've added support for Trollius in Autobahn 0.7.3

https://github.com/tavendo/AutobahnPython#python-support

pip install autobahn[asyncio]

will - as a dependency - install Trollius on Python 2, Tulip on Python
3.3 and nothing on Python 3.4+.

Awesome!

Cheers,
/Tobias

[Tobias Oberstein]

> * Trollius coroutines must use "raise Return(value)", whereas Tulip simply

Could you explain why that particular construct of "returning by
raising" was chosen?

It works, but looks strange ..

https://github.com/tavendo/AutobahnPython/blob/master/examples/asyncio/websocket/slowsquare/server_py2.py#L34

/Tobias

[Guido van Rossum]

There are essentially two alternatives: yield a magic value, or raise
a magic exception. I don't like yield, because it overloads the other
semantic of yield, which are already so severely overloaded. So I
prefer the magic exception. To raise an exception, you can either use
a raise statement with a special exception, or a special function that
raises it. The semantics are the same. I prefer the raise statement,
because it makes it clear that control flows out of the function here
(and any code following it is dead). Also, Emacs automatically dedents
after a raise statement, which is what you want. Finally, in Python
3.3+, return from a generator is actually implemented by raising
StopIteration with the return value as argument. So it's semantically
very close.

--
--Guido van Rossum (python.org/~guido)

[Victor Stinner]

Hi,

2014/1/7 Tobias Oberstein <tobias.o...@gmail.com>:

> Is it supposed to work on PyPy?

I didn't test, but it would be nice to support PyPy as well.

> __builtins__['InterruptedError'] = InterruptedError
> TypeError: 'module' object does not support item assignment

I plan to remove these hacks. I wrote them like get to quickly get a
working proof of concept and limit changes with Tulip.

Victor

[Victor Stinner]

2014/1/7 Tobias Oberstein <tobias.o...@gmail.com>:

> Hi Victor,
>
> I've added support for Trollius in Autobahn 0.7.3
>
> https://github.com/tavendo/AutobahnPython#python-support

Great! Do you have specific code for Trollius? If yes, where is it please?

I plan to support Python 2.6, there is a work-in-progress branch for this.

Victor

[Tobias Oberstein]

Am 07.01.2014 16:55, schrieb Guido van Rossum:
> There are essentially two alternatives: yield a magic value, or raise
> a magic exception. I don't like yield, because it overloads the other
> semantic of yield, which are already so severely overloaded. So I
> prefer the magic exception. To raise an exception, you can either use
> a raise statement with a special exception, or a special function that
> raises it. The semantics are the same. I prefer the raise statement,
> because it makes it clear that control flows out of the function here
> (and any code following it is dead). Also, Emacs automatically dedents

Not sure I fully get it:

Twisted's `inlineCallback` use regular raise for exceptions, but a
special `returnValue` function for returning

https://github.com/tavendo/AutobahnPython/blob/master/examples/twisted/websocket/slowsquare/server.py#L36

Is that a third alternative? [not a rhetorical question! ;)]

> after a raise statement, which is what you want. Finally, in Python
> 3.3+, return from a generator is actually implemented by raising
> StopIteration with the return value as argument. So it's semantically
> very close.

That's very interesting. Need to think more about it ..

Thanks for this detailed explanation!

/Tobias

[Tobias Oberstein]

Am 07.01.2014 17:18, schrieb Victor Stinner:
> 2014/1/7 Tobias Oberstein <tobias.o...@gmail.com>:
>> Hi Victor,
>>
>> I've added support for Trollius in Autobahn 0.7.3
>>
>> https://github.com/tavendo/AutobahnPython#python-support
>
> Great! Do you have specific code for Trollius? If yes, where is it please?

Nope, no specific code.

What I did is change the Autobahn/asyncio integration code to purely
rely on futures and callbacks, getting rid of "yield from" altogether.

See the changes to the first file here:

https://github.com/tavendo/AutobahnPython/commit/88289ca520a3b5c60115bf661a795342c965802f#diff-9d48c45d53c0d55b48dfecddaf2434d5

>
> I plan to support Python 2.6, there is a work-in-progress branch for this.

Cool!

/Tobias

>
> Victor
>

[Guido van Rossum]

On Tue, Jan 7, 2014 at 6:45 AM, Tobias Oberstein
<tobias.o...@gmail.com> wrote:
> Am 07.01.2014 16:55, schrieb Guido van Rossum:
>
>> There are essentially two alternatives: yield a magic value, or raise
>> a magic exception. I don't like yield, because it overloads the other
>
>> semantic of yield, which are already so severely overloaded. So I
>> prefer the magic exception. To raise an exception, you can either use
>> a raise statement with a special exception, or a special function that
>> raises it. The semantics are the same. I prefer the raise statement,
>> because it makes it clear that control flows out of the function here
>> (and any code following it is dead). Also, Emacs automatically dedents
>
> Not sure I fully get it:
>
> Twisted's `inlineCallback` use regular raise for exceptions, but a special
> `returnValue` function for returning
>
> https://github.com/tavendo/AutobahnPython/blob/master/examples/twisted/websocket/slowsquare/server.py#L36
>
> Is that a third alternative? [not a rhetorical question! ;)]

No, that function raises an exception. The disadvantage (in my mind)
over just using a raise statement is that the control flow is clear to
even the dumbest "static analysis" code (like Emacs' python-mode).

>> after a raise statement, which is what you want. Finally, in Python
>> 3.3+, return from a generator is actually implemented by raising
>> StopIteration with the return value as argument. So it's semantically
>> very close.
>
>
> That's very interesting. Need to think more about it ..
>
> Thanks for this detailed explanation!
>
> /Tobias
>
>
>>
>> On Tue, Jan 7, 2014 at 4:37 AM, Tobias Oberstein
>> <tobias.o...@gmail.com> wrote:
>>>>
>>>> * Trollius coroutines must use "raise Return(value)", whereas Tulip
>>>> simply
>>>
>>>
>>>
>>> Could you explain why that particular construct of "returning by raising"
>>> was chosen?
>>>
>>> It works, but looks strange ..
>>>
>>>
>>> https://github.com/tavendo/AutobahnPython/blob/master/examples/asyncio/websocket/slowsquare/server_py2.py#L34
>>>
>>> /Tobias
>>
>>
>>
>>
>

[Tobias Oberstein]

>> Not sure I fully get it:
>>
>> Twisted's `inlineCallback` use regular raise for exceptions, but a special
>> `returnValue` function for returning
>>
>> https://github.com/tavendo/AutobahnPython/blob/master/examples/twisted/websocket/slowsquare/server.py#L36
>>
>> Is that a third alternative? [not a rhetorical question! ;)]
>
> No, that function raises an exception. The disadvantage (in my mind)
> over just using a raise statement is that the control flow is clear to
> even the dumbest "static analysis" code (like Emacs' python-mode).

Ahh. Yes.

http://twistedmatrix.com/trac/browser/tags/releases/twisted-13.2.0/twisted/internet/defer.py#L1063

[Sorry, I should have had a look into `returnValue` myself ..]

It's just sugered up in Twisted .. and one could simply do

def returnValue(value):
raise asyncio.Result(value)

on top of Trollius.

But I agree on the control flow thing.

Thanks again,

/Tobias

[Victor Stinner]

Hi,

2014/1/7 Victor Stinner <victor....@gmail.com>:

I removed the hacks on __builtins__ and ssl modules. Could you please
try Autobahn with PyPy using Trollius?

Trollius now also has an experimental support of Python 2.6 and 3.2.

(It works also on Python 3.3 and 3.4.)

Victor

[Tobias Oberstein]

>> I plan to remove these hacks. I wrote them like get to quickly get a
>> working proof of concept and limit changes with Tulip.
>
> I removed the hacks on __builtins__ and ssl modules. Could you please
> try Autobahn with PyPy using Trollius?

Works!

Tested on Linux:

oberstet@vbox-ubuntu1310:~/scm/AutobahnPython/examples/asyncio/websocket/echo$
pypy -V
Python 2.7.3 (87aa9de10f9c, Nov 24 2013, 18:48:13)
[PyPy 2.2.1 with GCC 4.6.3]

and FreeBSD

[oberstet@tvd_build_txpypy
~/scm/AutobahnPython/examples/asyncio/websocket/echo]$ pypy -V
Python 2.7.3 (4369d6c2378e, Nov 19 2013, 23:00:19)
[PyPy 2.3.0-alpha0 with GCC 4.2.1 Compatible FreeBSD Clang 3.3
(tags/RELEASE_33/final 183502)]

Awesome!!

I've updated https://github.com/tavendo/AutobahnPython#python-support

>
> Trollius now also has an experimental support of Python 2.6 and 3.2.

Here is PyPy3 (which is tracking Python 3.2.3):

oberstet@vbox-ubuntu1310:~/scm/trollius$
~/pypy3-2.1-beta1-linux64/bin/pypy setup.py install

Traceback (most recent call last):

File "setup.py", line 35, in <module>
if sys.version < (2, 7):
TypeError: unorderable types: str < tuple

oberstet@vbox-ubuntu1310:~/scm/trollius$ hg tip
Änderung: 958:6fe9d3161e71
Zweig: trollius
Marke: tip
Nutzer: schlamar <marc.s...@gmail.com>
Datum: Wed Jan 08 10:19:04 2014 +0100
Zusammenfassung: Some fixes for IocpEventLoop.

>
> (It works also on Python 3.3 and 3.4.)

Ok, but I guess there is no real reason to use Trollius on those, right?

3.3 would use https://pypi.python.org/pypi/asyncio/
3.4 has asyncio built in

/Tobias

>
> Victor
>

[Victor Stinner]

Hi,

2014/1/8 Tobias Oberstein <tobias.o...@gmail.com>:

>> I removed the hacks on __builtins__ and ssl modules. Could you please
>> try Autobahn with PyPy using Trollius?
>
> Works!

Cool.

> I've updated https://github.com/tavendo/AutobahnPython#python-support

Python 2.6 is not supported using Trollius?

> Here is PyPy3 (which is tracking Python 3.2.3):

> (...)

> File "setup.py", line 35, in <module>
> if sys.version < (2, 7):
> TypeError: unorderable types: str < tuple

Oops, it's a typo. Fixed.

>> (It works also on Python 3.3 and 3.4.)
>
> Ok, but I guess there is no real reason to use Trollius on those, right?

Using "yield from" in Python 2 raises a SyntaxError. It's not so easy
to write a same code base for Trollius and Tulip. Supporting Python
3.3+ allows to:

(a) port a Python 2 project on Trollius,
(b) port it on Python 3,
(c) and later replace easily Trollius with Tulip.

For a huge project like OpenStack (+2.5 millions line of Python code!)
it's not posible to do the 3 steps at once. Changes must be
incremental and small.

Victor

[Tobias Oberstein]

>> I've updated https://github.com/tavendo/AutobahnPython#python-support
>
> Python 2.6 is not supported using Trollius?

I built Python 2.6 from sources .. but it fails

oberstet@vbox-ubuntu1310:~/scm/trollius$ ~/python269/bin/python
Python 2.6.9 (unknown, Jan 8 2014, 11:31:08)
[GCC 4.8.1] on linux3
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio

Traceback (most recent call last):

File "<stdin>", line 1, in <module>
File "asyncio/__init__.py", line 23, in <module>
from .backport_ssl import *
File "asyncio/backport_ssl.py", line 1, in <module>
import ssl
File "/home/oberstet/python269/lib/python2.6/ssl.py", line 60, in
<module>
import _ssl # if we can't import it, let the error
propagate
ImportError: No module named _ssl
>>>

I don't have the time to patch Py 2.6 to build on a recent host with
OpenSSL 1.0, which seems to require multiple steps:

http://askubuntu.com/questions/21547/what-are-the-packages-libraries-i-should-install-before-compiling-python-from-so

Is SSL a strict requirement of Trollius?

>
>> Here is PyPy3 (which is tracking Python 3.2.3):
>> (...)
>> File "setup.py", line 35, in <module>
>> if sys.version < (2, 7):
>> TypeError: unorderable types: str < tuple
>
> Oops, it's a typo. Fixed.

Autobahn/Trollius now works on:

oberstet@vbox-ubuntu1310:~/scm/AutobahnPython/examples/asyncio/websocket/echo$
~/pypy3-2.1-beta1-linux64/bin/pypy -V
Python 3.2.3 (d63636b30cc0, Jul 30 2013, 07:02:17)
[PyPy 2.1.0-beta1 with GCC 4.6.3]

The only remaining issue with Autobahn is the use of u"" literals

oberstet@vbox-ubuntu1310:~/scm/AutobahnPython/examples/asyncio/websocket/echo$
~/pypy3-2.1-beta1-linux64/bin/pypy
Python 3.2.3 (d63636b30cc0, Jul 30 2013, 07:02:17)
[PyPy 2.1.0-beta1 with GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
And now for something completely different: ``the future has just begun''
>>>> s = u"kjshdf"
File "<stdin>", line 1
s = u"kjshdf"
^
SyntaxError: invalid syntax
>>>>

I thought the 'u' would just be ignored on Py 3 (any version)?

>
>>> (It works also on Python 3.3 and 3.4.)
>>
>> Ok, but I guess there is no real reason to use Trollius on those, right?
>
> Using "yield from" in Python 2 raises a SyntaxError. It's not so easy
> to write a same code base for Trollius and Tulip. Supporting Python
> 3.3+ allows to:
>
> (a) port a Python 2 project on Trollius,
> (b) port it on Python 3,
> (c) and later replace easily Trollius with Tulip.

Makes sense.

>
> For a huge project like OpenStack (+2.5 millions line of Python code!)
> it's not posible to do the 3 steps at once. Changes must be
> incremental and small.

That's huge. I didn't expect OpenStack being such large;)

/Tobias

>
> Victor
>

[Aymeric Augustin]

2014/1/8 Tobias Oberstein <tobias.o...@gmail.com>

I thought the 'u' would just be ignored on Py 3 (any version)?

PEP 414 was implemented in Python 3.3.

--
Aymeric.

[Tobias Oberstein]

Am 08.01.2014 12:00, schrieb Aymeric Augustin:
>
> 2014/1/8 Tobias Oberstein <tobias.o...@gmail.com
> <mailto:tobias.o...@gmail.com>>

>
> I thought the 'u' would just be ignored on Py 3 (any version)?
>
>
> PEP 414 was implemented in Python 3.3.

Shit. I mean thanks;)

This complicates things ..

/Tobias

>
> --
> Aymeric.

[Victor Stinner]

2014/1/8 Tobias Oberstein <tobias.o...@gmail.com>:

>> I thought the 'u' would just be ignored on Py 3 (any version)?
>>
>> PEP 414 was implemented in Python 3.3.
>
> Shit. I mean thanks;)

Use six.u() function which works on all Python versions.
http://pythonhosted.org/six/#six.u

Victor

[Victor Stinner]

2014/1/8 Tobias Oberstein <tobias.o...@gmail.com>:

>>>> import asyncio
> Traceback (most recent call last):

> ...

> ImportError: No module named _ssl

Oh, I never tried Python without SSL support. It now fixed in the
development version.

> I don't have the time to patch Py 2.6 to build on a recent host with OpenSSL
> 1.0, which seems to require multiple steps:
>
> http://askubuntu.com/questions/21547/what-are-the-packages-libraries-i-should-install-before-compiling-python-from-so

Ah?

Using the the 2.6 branch of the Mercurial version of Python, it just
compiled fine on my Fedora 19 which provides OpenSSL 1.0.1e.

Maybe you just need to install "openssl-dev" ? Try "apt-get builddep
python" command to install all dependencies to compile Python.

Victor