Gevent and PyPy

Showing 1-36 of 36 messages
Gevent and PyPy galfy 3/19/12 2:18 AM
With every new version PyPy is becoming faster. If you believe the benchmarks [http://speed.pypy.org/] at the moment PyPy is 5x faster than the CPython. That makes me "crave" for a combination pypy+gevent. 
And this raises questions. 
What is needed in order to be able to run gevent under PyPy? 
Are there any efforts from third parties to make it possible?
Are there any workarounds, like CPyExt [http://morepypy.blogspot.de/2010/04/using-cpython-extension-modules-with.html] that make this happen?

Regards,
 Gal
Re: [gevent] Gevent and PyPy Damien Churchill 3/19/12 6:27 AM
On 19 March 2012 09:18, galfy <galfyo...@googlemail.com> wrote:
> What is needed in order to be able to run gevent under PyPy?

gevent would have to have support for running using PyPys stackless mode.

Re: [gevent] Gevent and PyPy Floris Bruynooghe 3/19/12 6:54 AM

Actually pypy has it's own version of the greenlet module in their
default build afaik (built on slightly different primitives).  IIRC
Armin finished that not long after europython last year.  There'd
still be an amount of porting to be done though, since gevent includes
more C code then just greenlet, you might get started easier using
eventlet which has a few hubs to choose from, some of which only rely
on the stdlib.  On plain CPython that can be expected to be a little
slower but never tried PyPy myself.

Regards,
Floris

--
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

Re: Gevent and PyPy Hong Minhee 3/19/12 2:29 PM
As Floris Bruynooghe mentioned, PyPy alread provide its own greenlet module.  Another needed one is Cython.  To make gevent to run under PyPy, Cython firstly has to be ported to PyPy or gevent has to use ctypes instead of (or as a counterpart of) Cython.
Re: [gevent] Re: Gevent and PyPy Damien Churchill 3/19/12 4:12 PM
On 19 March 2012 21:29, Hong Minhee <hong....@gmail.com> wrote:
> As Floris Bruynooghe mentioned, PyPy alread provide its own greenlet
> module.  Another needed one is Cython.  To make gevent to run under PyPy,
> Cython firstly has to be ported to PyPy or gevent has to use ctypes instead
> of (or as a counterpart of) Cython.
>

Or the cython modules are re-written in python using ctypes for use in PyPy.

Re: [gevent] Re: Gevent and PyPy Vasile Ermicioi 3/20/12 12:28 AM
at the moment PyPy is 5x faster than the CPython. That makes me "crave" for a combination pypy+gevent

I think greenlet module of pypy is not jit-able, that is, it may be slower than with cpython 
Re: [gevent] Re: Gevent and PyPy Antonin AMAND 3/21/12 12:48 AM
http://morepypy.blogspot.fr/2010/04/using-cpython-extension-modules-with.html

It seems there is a Python C level API compatibility layer for pypy.
Since Cython produces traditional C extension code it would make it
possible to run the gevent C code produced by cython on pypy.

Or am I missing something ?

Re: [gevent] Re: Gevent and PyPy Antonin AMAND 3/21/12 1:05 AM
Actually this article [1] discourage from running gevent on top of cpyext/pypy:

"cpyext is slow, and always will be slow. It has to emulate far too
many details of the CPython object model that don't exist on PyPy
(e.g., reference counting). Since people are using NumPy primarily for
speed this would mean that even if we could have a working NumPy, no
one would want to use it. Also, as soon as the execution crosses the
cpyext boundary, it becomes invisible to the JIT, which means the JIT
has to assume the worst and deoptimize stuff away."

[1] http://morepypy.blogspot.fr/2011/05/numpy-follow-up.html

Re: [gevent] Re: Gevent and PyPy André Cruz 4/26/12 4:32 AM
So, there are no efforts in progress to make gevent usable under PyPy?

More importantly, is it expected that PyPy would provide a significant performance boost?

Best regards,
André

Re: [gevent] Re: Gevent and PyPy Vasile Ermicioi 4/26/12 9:01 AM
Armin Rigo is working on STM, 
Re: Re: [gevent] Gevent and PyPy André Cruz 4/26/12 9:07 AM
On Apr 26, 2012, at 17:01 , Vasile Ermicioi wrote:

Armin Rigo is working on STM, 

And STM is needed for gevent support?

André

Re: [gevent] Re: Gevent and PyPy Matt Joiner 4/26/12 9:12 AM

gevent attempts to work around concurrency problems in Python by reducing the dependence on native threading. STM will likely reduce single threaded performance, and by extension gevent.

There's also a good chance that C extensions won't work well, and that PyPy's greenlet interface will be a low priority because of how it relates to the goals of introducing STM.

Re: [gevent] Re: Gevent and PyPy Randall Leeds 4/26/12 11:46 AM
On Thu, Apr 26, 2012 at 09:12, Matt Joiner <anac...@gmail.com> wrote:
> gevent attempts to work around concurrency problems in Python by reducing
> the dependence on native threading. STM will likely reduce single threaded
> performance, and by extension gevent.
>
> There's also a good chance that C extensions won't work well, and that
> PyPy's greenlet interface will be a low priority because of how it relates
> to the goals of introducing STM.

I'm not sure I see that. There's been quite a lot of activity on the
pypy-dev list discussing ways to update PyPy's cpyext and bring some
things better in line with Cython and CPython. I can't say I've
followed it all super closely, but from my understanding making Cython
output compile with PyPy is the direction to go and the pypy-dev list
is where you want to go to talk about it.

A lot of the current discussion revolves around making C calls faster,
with fewer layers of indirection (the GC makes this difficult).
Unfortunately, calling C code from PyPy isn't brilliantly fast right
now, so it's not clear that gevent+pypy would be much better than a
pypy wsgi server without gevent.

At least... that's what I get from the discussions I'm seeing.

-Randall
Re: Re: [gevent] Gevent and PyPy Vasile Ermicioi 4/26/12 12:03 PM

And STM is needed for gevent support?


no, I gave the link because I think it responds to your other question ( is it expected that PyPy would provide a significant performance boost )

"Notably, TM could benefit any event-based system that is written to dispatch events serially (Twisted-based, most GUI toolkit, Stackless, gevent, and so on). The events would internally be processed in parallel, while maintaining the illusion of serial execution, with all the corresponding benefits of safety. This should be possible with minimal changes to the event dispatchers. This approach has been described by the Automatic Mutual Exclusion work at Microsoft Research, but not been implemented anywhere (to the best of our knowledge)."
Re: Re: [gevent] Gevent and PyPy Matt Joiner 4/26/12 10:09 PM

Indeed that sounds quite handy.

Re: Gevent and PyPy Sean Talts 5/9/12 9:24 AM
It seems like we should be able to implement the gevent api in pure
python somewhat easily (since pypy already has greenlets and event
loops).  Has anyone done this?  Any reasons why it might be a bad
idea?

-Sean

On Apr 27, 1:09 am, Matt Joiner <anacro...@gmail.com> wrote:
> Indeed that sounds quite handy.
> On Apr 27, 2012 3:03 AM, "Vasile Ermicioi" <elff...@gmail.com> wrote:
>
>
>
>
>
>
>
>
>
> >> And STM is needed for gevent support?
>
> > no, I gave the link because I think it responds to your other question ( is
> > it expected that PyPy would provide a significant performance boost )
>
> > "Notably, TM could benefit any event-based system that is written to
> > dispatch events serially (Twisted-based, most GUI toolkit, Stackless,
> > gevent, and so on). The events would internally be processed in parallel,
> > while maintaining the illusion of serial execution, with all the
> > corresponding benefits of safety. This should be possible with minimal
> > changes to the event dispatchers. This approach has been described by the Automatic
> > Mutual Exclusion<http://research.microsoft.com/en-us/projects/ame/default.aspx> work
Re: [gevent] Re: Gevent and PyPy Floris Bruynooghe 5/9/12 11:28 AM
On 9 May 2012 17:24, Sean Talts <xit...@gmail.com> wrote:
> It seems like we should be able to implement the gevent api in pure
> python somewhat easily (since pypy already has greenlets and event
> loops).  Has anyone done this?  Any reasons why it might be a bad
> idea?

This is essentially what eventlet does.  I had a quick experiment and
simple sockets seem to work with eventlet on pypy but urllib2 does
give a strange error.  Not sure I'll spend much time investigating
why, but it's interesting anyway.
Re: [gevent] Re: Gevent and PyPy Matt Joiner 5/9/12 11:56 AM

I've done this. It's better in that the implementation is only several 100 lines. It's worse because the only reason it's useful is to work around pathological contention in the GIL introduced by very high levels of concurrency. Gevent reduces the concurrency overhead since most of it's done in C.

Re: [gevent] Re: Gevent and PyPy Sean Talts 5/9/12 1:47 PM
I've never heard of this GIL issue with coroutines... What exactly happens?  Do you know if this is still a problem in pypy?  Do you have the source for your "pygevent" somewhere online I could fork?  Thanks!
Re: [gevent] Re: Gevent and PyPy Matt Joiner 5/9/12 5:29 PM

Sorry, I explained poorly. What I'm saying is that scheduled coroutines alleviate some negative aspects of the GIL. But doing the scheduling in pure Python the overhead is very high anyway, you may as well use native threading for the improved portability and compatibility with existing code. Your mileage may vary of course.

Here's my Python 3, pure Python gevent clone: https://bitbucket.org/anacrolix/gthread The README contains an overview of my experiences with the model.

I also wrote an asynchronous framework using the new PEP 380 for Python 3.3: https://bitbucket.org/anacrolix/green380

Lastly, I wrote a pure Python script that monkey patches the stdlib and allows you to run unmodified Python code emulating threads as greenlets, which is my personal favorite: https://bitbucket.org/anacrolix/green380. python3 goco.py <usual python parameters>.

Re: [gevent] Re: Gevent and PyPy Matt Joiner 5/12/12 2:05 AM
Sorry the last link should have been https://bitbucket.org/anacrolix/goco for the pure Python framework.


On Thursday, May 10, 2012, Matt Joiner wrote:

Sorry, I explained poorly. What I'm saying is that scheduled coroutines alleviate some negative aspects of the GIL. But doing the scheduling in pure Python the overhead is very high anyway, you may as well use native threading for the improved portability and compatibility with existing code. Your mileage may vary of course.

Here's my Python 3, pure Python gevent clone: https://bitbucket.org/anacrolix/gthread The README contains an overview of my experiences with the model.

I also wrote an asynchronous framework using the new PEP 380 for Python 3.3: https://bitbucket.org/anacrolix/green380

Lastly, I wrote a pure Python script that monkey patches the stdlib and allows you to run unmodified Python code emulating threads as greenlets, which is my personal favorite: https://bitbucket.org/anacrolix/green380. python3 goco.py <usual python parameters>.

On May 10, 2012 4:47 AM, "Sean Talts" <xitrium@gmail.com> wrote:
I've never heard of this GIL issue with coroutines... What exactly happens?  Do you know if this is still a problem in pypy?  Do you have the source for your "pygevent" somewhere online I could fork?  Thanks!

On Wednesday, May 9, 2012 2:56:15 PM UTC-4, Matt Joiner wrote:

I've done this. It's better in that the implementation is only several 100 lines. It's worse because the only reason it's useful is to work around pathological contention in the GIL introduced by very high levels of concurrency. Gevent reduces the concurrency overhead since most of it's done in C.

On May 10, 2012 12:24 AM, "Sean Talts" <xitrium@gmail.com> wrote:
It seems like we should be able to implement the gevent api in pure
python somewhat easily (since pypy already has greenlets and event
loops).  Has anyone done this?  Any reasons why it might be a bad
idea?

-Sean

On Apr 27, 1:09 am, Matt Joiner <anacro...@gmail.com> wrote:
> Indeed that sounds quite handy.
> On Apr 27, 2012 3:03 AM, "Vasile Ermicioi" <elff...@gmail.com> wrote:
>
>
>
>
>
>
>
>
>
> >> And STM is needed for gevent support?
>
> > no, I gave the link because I think it responds to your other question ( is
> > it expected that PyPy would provide a significant performance boost )
>
> > "Notably, TM could benefit any event-based system that is written to
> > dispatch events serially (Twisted-based, most GUI toolkit, Stackless,
> > gevent, and so on). The events would internally be processed in parallel,
> > while maintaining the illusion of serial execution, with all the
> > corresponding benefits of safety. This should be possible with minimal
> > changes to the event dispatchers. This approach has been described by the Automatic
> > Mutual Exclusion<http://research.microsoft.com/en-us/projects/ame/default.aspx> work
> > at Microsoft Research, but not been implemented anywhere (to the best of
> > our knowledge)."
Re: Gevent and PyPy brett 6/29/12 8:29 PM
I have started on the gevent-ctypes port for PyPy.  I already removed the cython stuff, and replaced libev with a ctypes wrapper for libev.
Greenlet + JIT support is coming later on in PyPy, currently the JIT will not trace a greenlet, but at least it works.

You can get the source from here:
http://code.google.com/r/brett-gevent-ctypes/

-brett-
Re: Gevent and PyPy Adam Smith 8/21/12 4:01 PM
Can I revive this old thread to ask the status? Has pypy-1.9 (which supports jit+greenlet) improved the situation any?

There's basically nothing in the python world I want more than to have gevent-on-pypy for my worker daemons.
Re: [gevent] Re: Gevent and PyPy Benoit Chesneau 8/22/12 12:42 AM
On Wed, Aug 22, 2012 at 1:01 AM, Adam Smith
<swiss.arm...@gmail.com> wrote:
> Can I revive this old thread to ask the status? Has pypy-1.9 (which supports
> jit+greenlet) improved the situation any?
>
> There's basically nothing in the python world I want more than to have
> gevent-on-pypy for my worker daemons.
>
I think it would be easier to port eventlet to pypy rather than gevent
since pypy offer a greenlet compatible API.

- benoît
Re: [gevent] Re: Gevent and PyPy Saúl Ibarra Corretgé 8/22/12 2:27 AM
Hi,

Adam Smith wrote:
> Can I revive this old thread to ask the status? Has pypy-1.9 (which
> supports jit+greenlet) improved the situation any?
>

The PyPy guys seem to be going in the cffi direction, so I guess
gevent's core would need to be written with cffi instead of Cython.

Having that said, I don't know if there would be any performance
degradation when using CPython.

While it's not a simple task, but it can be done as a separated project:
lets say you have some cffi bindings for libev, you could create a
PyPyLoop class mirroring gevent.core.loop and then set it on
Hub.loop_class, so your core is used instead of the cython based one.
While not ideal, it may be a good start.

> There's basically nothing in the python world I want more than to have
> gevent-on-pypy for my worker daemons.
>

I'm really curious about this, are you finding any problems /
limitations? Personally I'd rather get Python 3 support than PyPy.


Regards,

--
Saúl Ibarra Corretgé
http://saghul.net/blog | http://about.me/saghul
Re: [gevent] Re: Gevent and PyPy smurf 8/22/12 4:05 AM
Hi,

Sa�l Ibarra Corretg�:
> Personally I'd rather get Python 3 support than PyPy.
>
Personally I'd rather get both, including Py3 support for PyPy.
Fortunately, people are working on it.
--
-- Matthias Urlichs
Re: [gevent] Re: Gevent and PyPy Ralf Schmitt 8/22/12 5:01 AM
Saúl Ibarra Corretgé <sag...@gmail.com> writes:

>
> The PyPy guys seem to be going in the cffi direction, so I guess
> gevent's core would need to be written with cffi instead of Cython.

cython 0.17 will have basic support for pypy [1]. that would be another
option.

[1] http://docs.cython.org/src/tutorial/pypy.html

--
cheers
ralf
Re: [gevent] Re: Gevent and PyPy Saúl Ibarra Corretgé 8/22/12 11:50 PM
Hi Ralph,

> cython 0.17 will have basic support for pypy [1]. that would be another
> option.
>
> [1] http://docs.cython.org/src/tutorial/pypy.html
>

Indeed. However, last time I checked the PyPy guys said that cpyext is
and will always be slow, that's one of the reasons why cffi was created
(IIRC).

IMHO, Gevent shouldn't get slower on CPython in order to support PyPy.

--
Saúl Ibarra Corretgé
http://saghul.net/blog | http://about.me/saghul
pypycore [was: [gevent] Re: Gevent and PyPy] Ralf Schmitt 9/15/12 2:59 PM
Saúl Ibarra Corretgé <sag...@gmail.com> writes:

>
> The PyPy guys seem to be going in the cffi direction, so I guess
> gevent's core would need to be written with cffi instead of Cython.
>
> Having that said, I don't know if there would be any performance
> degradation when using CPython.
>
> While it's not a simple task, but it can be done as a separated
> project: lets say you have some cffi bindings for libev, you could
> create a PyPyLoop class mirroring gevent.core.loop and then set it on
> Hub.loop_class, so your core is used instead of the cython based
> one. While not ideal, it may be a good start.

i've done that in: https://github.com/schmir/pypycore

it's a reimplementation of gevent.core using cffi. it's not complete at
the moment. but it already allows me to serve a hello world app with
pywsgi (with cpython).

running it on pypy currently requires the development version of
pypy. note that there seems to be a problem with pypy's greenlet
implementation. i'll have to dig a bit further into this however.

--
cheers
ralf
Re: pypycore [was: [gevent] Re: Gevent and PyPy] Saúl Ibarra Corretgé 9/18/12 1:32 PM

>
> i've done that in: https://github.com/schmir/pypycore
>
> it's a reimplementation of gevent.core using cffi. it's not complete at
> the moment. but it already allows me to serve a hello world app with
> pywsgi (with cpython).
>

I spotted that, looks great :-)

> running it on pypy currently requires the development version of
> pypy. note that there seems to be a problem with pypy's greenlet
> implementation. i'll have to dig a bit further into this however.
>

I'm also working on an alternative gevent core, based on libuv (using pyuv) and one of the things I had to "fake" was incrementing the refcount of the watchers even if no one holds a reference to them. I see you comented that line here: https://github.com/schmir/pypycore/blob/master/pypycore.py#L638 My workaround for that is to keep a list of watchers attached to the loop and add and remove them as needed. Don't you need to do something similar? Or maybe I'm misunderstanding something :-)


Cheers,

Re: pypycore [was: [gevent] Re: Gevent and PyPy] Ralf Schmitt 9/19/12 1:20 AM
Saúl Ibarra Corretgé <sag...@gmail.com> writes:

>
> I'm also working on an alternative gevent core, based on libuv (using
> pyuv) and one of the things I had to "fake" was incrementing the
> refcount of the watchers even if no one holds a reference to them. I
> see you comented that line here:

do you have any code to show us?

> https://github.com/schmir/pypycore/blob/master/pypycore.py#L638 My
> workaround for that is to keep a list of watchers attached to the loop
> and add and remove them as needed. Don't you need to do something
> similar? Or maybe I'm misunderstanding something :-)

yes, I think so. At the moment the watcher may be kept alive by a
reference from the cffi callback. but it would be collected if gc
detects the cycle. I'll have to look into it.
Re: pypycore [was: [gevent] Re: Gevent and PyPy] Saúl Ibarra Corretgé 9/19/12 3:15 PM
Ralf Schmitt wrote:
> Saúl Ibarra Corretgé<sag...@gmail.com>  writes:
>
>> I'm also working on an alternative gevent core, based on libuv (using
>> pyuv) and one of the things I had to "fake" was incrementing the
>> refcount of the watchers even if no one holds a reference to them. I
>> see you comented that line here:
>
> do you have any code to show us?
>

I'll upload it as soon as I come back from a short trip.

>> https://github.com/schmir/pypycore/blob/master/pypycore.py#L638 My
>> workaround for that is to keep a list of watchers attached to the loop
>> and add and remove them as needed. Don't you need to do something
>> similar? Or maybe I'm misunderstanding something :-)
>
> yes, I think so. At the moment the watcher may be kept alive by a
> reference from the cffi callback. but it would be collected if gc
> detects the cycle. I'll have to look into it.

I see.
Re: pypycore [was: [gevent] Re: Gevent and PyPy] Saúl Ibarra Corretgé 10/4/12 3:20 PM
Ralf Schmitt wrote:
> Saúl Ibarra Corretgé<sag...@gmail.com>  writes:
>
>> I'm also working on an alternative gevent core, based on libuv (using
>> pyuv) and one of the things I had to "fake" was incrementing the
>> refcount of the watchers even if no one holds a reference to them. I
>> see you comented that line here:
>
> do you have any code to show us?
>

Now I do :-) https://github.com/saghul/uvent
Re: pypycore [was: [gevent] Re: Gevent and PyPy] Denis Bilenko 10/5/12 5:25 AM
On Fri, Oct 5, 2012 at 12:20 AM, Saúl Ibarra Corretgé <sag...@gmail.com> wrote:
>>> I'm also working on an alternative gevent core, based on libuv (using
>>> pyuv) and one of the things I had to "fake" was incrementing the
>>> refcount of the watchers even if no one holds a reference to them. I
>>> see you comented that line here:
> Now I do :-) https://github.com/saghul/uvent

It seems that the best results would be achieved by using libuv's TCP
and UDP watchers in gevent.socket/ssl. Without it there's no advantage
in using this over bare libev.

Do you plan to do something like that? It would be a separate branch
of gevent, rather than just a new loop implementation.
Re: pypycore [was: [gevent] Re: Gevent and PyPy] Saúl Ibarra Corretgé 10/6/12 12:17 AM
Hi Denis,

The UDP and TCP handles are implemented internally more or less like the poll handle, but doing 64k reads. It may perform better since most of the work is done in C land, but I don't have numbers on that. I'll explore the possibility of doing this.

One good advantage of using libuv instead of liev is Windows support. libuv uses some obscure Windows stuff (sometimes undocumented APIs IIRC) to have a epoll like thing for the poll handles, while libev is only able to do select AFAIK.

Also, there are a ton of file operations implemented already, and in a cross-platform way. File reads and writes are not serialized yet, but there are plans to make files like streams in libuv.

Last, libuv will remove libev (yes, it currently uses it internally) and use epoll in edge triggered mode in Linux, which should also bring a performance boost.


Cheers,


Re: pypycore [was: [gevent] Re: Gevent and PyPy] Michael So 10/6/12 1:43 AM
a huge different is libuv use IOCP in Windows instead of select(). I have tried IOCP before, you can treat it like epoll or kqueue, but IOCP is actually better in someway.

libuv inherit libev and adding IOCP on windows. 
in addition, it provides some more useful features like file operations as Saúl said.

so, we u have or need in libev, libuv also has it and has it better.

it would be great to see gevent would replace libev with libuv soon. :)
More topics »