With every new version PyPy is becoming faster. If you believe the benchmarks [http://speed.pypy.org/] at the moment PyPy is 5x faster than the CPython. That makes me "crave" for a combination pypy+gevent. And this raises questions. What is needed in order to be able to run gevent under PyPy? Are there any efforts from third parties to make it possible? Are there any workarounds, like CPyExt [ http://morepypy.blogspot.de/2010/04/using-cpython-extension-modules-w...] that make this happen?
On 19 March 2012 13:27, Damien Churchill <dam...@gmail.com> wrote:
> On 19 March 2012 09:18, galfy <galfyo.pun...@googlemail.com> wrote: >> What is needed in order to be able to run gevent under PyPy?
> gevent would have to have support for running using PyPys stackless mode.
Actually pypy has it's own version of the greenlet module in their default build afaik (built on slightly different primitives). IIRC Armin finished that not long after europython last year. There'd still be an amount of porting to be done though, since gevent includes more C code then just greenlet, you might get started easier using eventlet which has a few hubs to choose from, some of which only rely on the stdlib. On plain CPython that can be expected to be a little slower but never tried PyPy myself.
As Floris Bruynooghe mentioned, PyPy alread provide its own greenletmodule. Another needed one is Cython. To make gevent to run under PyPy, Cython firstly has to be ported to PyPy or gevent has to use ctypes instead of (or as a counterpart of) Cython.
On Monday, March 19, 2012 6:18:00 PM UTC+9, galfy wrote:
> With every new version PyPy is becoming faster. If you believe the > benchmarks [http://speed.pypy.org/] at the moment PyPy is 5x faster than > the CPython. That makes me "crave" for a combination pypy+gevent. > And this raises questions. > What is needed in order to be able to run gevent under PyPy? > Are there any efforts from third parties to make it possible? > Are there any workarounds, like CPyExt [ > http://morepypy.blogspot.de/2010/04/using-cpython-extension-modules-w...] > that make this happen?
On 19 March 2012 21:29, Hong Minhee <hong.min...@gmail.com> wrote:
> As Floris Bruynooghe mentioned, PyPy alread provide its own greenlet > module. Another needed one is Cython. To make gevent to run under PyPy, > Cython firstly has to be ported to PyPy or gevent has to use ctypes instead > of (or as a counterpart of) Cython.
Or the cython modules are re-written in python using ctypes for use in PyPy.
It seems there is a Python C level API compatibility layer for pypy. Since Cython produces traditional C extension code it would make it possible to run the gevent C code produced by cython on pypy.
Actually this article [1] discourage from running gevent on top of cpyext/pypy:
"cpyext is slow, and always will be slow. It has to emulate far too many details of the CPython object model that don't exist on PyPy (e.g., reference counting). Since people are using NumPy primarily for speed this would mean that even if we could have a working NumPy, no one would want to use it. Also, as soon as the execution crosses the cpyext boundary, it becomes invisible to the JIT, which means the JIT has to assume the worst and deoptimize stuff away."
> It seems there is a Python C level API compatibility layer for pypy. > Since Cython produces traditional C extension code it would make it > possible to run the gevent C code produced by cython on pypy.
gevent attempts to work around concurrency problems in Python by reducing
the dependence on native threading. STM will likely reduce single threaded
performance, and by extension gevent.
There's also a good chance that C extensions won't work well, and that
PyPy's greenlet interface will be a low priority because of how it relates
to the goals of introducing STM.
On Apr 27, 2012 12:01 AM, "Vasile Ermicioi" <elff...@gmail.com> wrote:
On Thu, Apr 26, 2012 at 09:12, Matt Joiner <anacro...@gmail.com> wrote:
> gevent attempts to work around concurrency problems in Python by reducing
> the dependence on native threading. STM will likely reduce single threaded
> performance, and by extension gevent.
> There's also a good chance that C extensions won't work well, and that
> PyPy's greenlet interface will be a low priority because of how it relates
> to the goals of introducing STM.
I'm not sure I see that. There's been quite a lot of activity on the
pypy-dev list discussing ways to update PyPy's cpyext and bring some
things better in line with Cython and CPython. I can't say I've
followed it all super closely, but from my understanding making Cython
output compile with PyPy is the direction to go and the pypy-dev list
is where you want to go to talk about it.
A lot of the current discussion revolves around making C calls faster,
with fewer layers of indirection (the GC makes this difficult).
Unfortunately, calling C code from PyPy isn't brilliantly fast right
now, so it's not clear that gevent+pypy would be much better than a
pypy wsgi server without gevent.
At least... that's what I get from the discussions I'm seeing.
no, I gave the link because I think it responds to your other question ( is
it expected that PyPy would provide a significant performance boost )
"Notably, TM could benefit any event-based system that is written to
dispatch events serially (Twisted-based, most GUI toolkit, Stackless,
gevent, and so on). The events would internally be processed in parallel,
while maintaining the illusion of serial execution, with all the
corresponding benefits of safety. This should be possible with minimal
changes to the event dispatchers. This approach has been described by
the Automatic
Mutual Exclusion<http://research.microsoft.com/en-us/projects/ame/default.aspx>
work
at Microsoft Research, but not been implemented anywhere (to the best of
our knowledge)."
> no, I gave the link because I think it responds to your other question ( is
> it expected that PyPy would provide a significant performance boost )
> "Notably, TM could benefit any event-based system that is written to
> dispatch events serially (Twisted-based, most GUI toolkit, Stackless,
> gevent, and so on). The events would internally be processed in parallel,
> while maintaining the illusion of serial execution, with all the
> corresponding benefits of safety. This should be possible with minimal
> changes to the event dispatchers. This approach has been described by the Automatic
> Mutual Exclusion<http://research.microsoft.com/en-us/projects/ame/default.aspx> work
> at Microsoft Research, but not been implemented anywhere (to the best of
> our knowledge)."
It seems like we should be able to implement the gevent api in pure
python somewhat easily (since pypy already has greenlets and event
loops). Has anyone done this? Any reasons why it might be a bad
idea?
-Sean
On Apr 27, 1:09 am, Matt Joiner <anacro...@gmail.com> wrote:
> Indeed that sounds quite handy.
> On Apr 27, 2012 3:03 AM, "Vasile Ermicioi" <elff...@gmail.com> wrote:
> >> And STM is needed for gevent support?
> > no, I gave the link because I think it responds to your other question ( is
> > it expected that PyPy would provide a significant performance boost )
> > "Notably, TM could benefit any event-based system that is written to
> > dispatch events serially (Twisted-based, most GUI toolkit, Stackless,
> > gevent, and so on). The events would internally be processed in parallel,
> > while maintaining the illusion of serial execution, with all the
> > corresponding benefits of safety. This should be possible with minimal
> > changes to the event dispatchers. This approach has been described by the Automatic
> > Mutual Exclusion<http://research.microsoft.com/en-us/projects/ame/default.aspx> work
> > at Microsoft Research, but not been implemented anywhere (to the best of
> > our knowledge)."
On 9 May 2012 17:24, Sean Talts <xitr...@gmail.com> wrote:
> It seems like we should be able to implement the gevent api in pure
> python somewhat easily (since pypy already has greenlets and event
> loops). Has anyone done this? Any reasons why it might be a bad
> idea?
This is essentially what eventlet does. I had a quick experiment and
simple sockets seem to work with eventlet on pypy but urllib2 does
give a strange error. Not sure I'll spend much time investigating
why, but it's interesting anyway.
I've done this. It's better in that the implementation is only several 100
lines. It's worse because the only reason it's useful is to work around
pathological contention in the GIL introduced by very high levels of
concurrency. Gevent reduces the concurrency overhead since most of it's
done in C.
On May 10, 2012 12:24 AM, "Sean Talts" <xitr...@gmail.com> wrote:
> It seems like we should be able to implement the gevent api in pure
> python somewhat easily (since pypy already has greenlets and event
> loops). Has anyone done this? Any reasons why it might be a bad
> idea?
> -Sean
> On Apr 27, 1:09 am, Matt Joiner <anacro...@gmail.com> wrote:
> > Indeed that sounds quite handy.
> > On Apr 27, 2012 3:03 AM, "Vasile Ermicioi" <elff...@gmail.com> wrote:
> > >> And STM is needed for gevent support?
> > > no, I gave the link because I think it responds to your other question
> ( is
> > > it expected that PyPy would provide a significant performance boost )
> > > "Notably, TM could benefit any event-based system that is written to
> > > dispatch events serially (Twisted-based, most GUI toolkit, Stackless,
> > > gevent, and so on). The events would internally be processed in
> parallel,
> > > while maintaining the illusion of serial execution, with all the
> > > corresponding benefits of safety. This should be possible with minimal
> > > changes to the event dispatchers. This approach has been described by
> the Automatic
> > > Mutual Exclusion<
> http://research.microsoft.com/en-us/projects/ame/default.aspx> work
> > > at Microsoft Research, but not been implemented anywhere (to the best
> of
> > > our knowledge)."
I've never heard of this GIL issue with coroutines... What exactly happens? Do you know if this is still a problem in pypy? Do you have the source for your "pygevent" somewhere online I could fork? Thanks!
On Wednesday, May 9, 2012 2:56:15 PM UTC-4, Matt Joiner wrote:
> I've done this. It's better in that the implementation is only several 100 > lines. It's worse because the only reason it's useful is to work around > pathological contention in the GIL introduced by very high levels of > concurrency. Gevent reduces the concurrency overhead since most of it's > done in C. > On May 10, 2012 12:24 AM, "Sean Talts" <xitr...@gmail.com> wrote:
>> It seems like we should be able to implement the gevent api in pure >> python somewhat easily (since pypy already has greenlets and event >> loops). Has anyone done this? Any reasons why it might be a bad >> idea?
>> -Sean
>> On Apr 27, 1:09 am, Matt Joiner <anacro...@gmail.com> wrote: >> > Indeed that sounds quite handy. >> > On Apr 27, 2012 3:03 AM, "Vasile Ermicioi" <elff...@gmail.com> wrote:
>> > >> And STM is needed for gevent support?
>> > > no, I gave the link because I think it responds to your other >> question ( is >> > > it expected that PyPy would provide a significant performance boost )
>> > > "Notably, TM could benefit any event-based system that is written to >> > > dispatch events serially (Twisted-based, most GUI toolkit, Stackless, >> > > gevent, and so on). The events would internally be processed in >> parallel, >> > > while maintaining the illusion of serial execution, with all the >> > > corresponding benefits of safety. This should be possible with minimal >> > > changes to the event dispatchers. This approach has been described by >> the Automatic >> > > Mutual Exclusion< >> http://research.microsoft.com/en-us/projects/ame/default.aspx> work >> > > at Microsoft Research, but not been implemented anywhere (to the best >> of >> > > our knowledge)."
Sorry, I explained poorly. What I'm saying is that scheduled coroutines
alleviate some negative aspects of the GIL. But doing the scheduling in
pure Python the overhead is very high anyway, you may as well use native
threading for the improved portability and compatibility with existing
code. Your mileage may vary of course.
Lastly, I wrote a pure Python script that monkey patches the stdlib and
allows you to run unmodified Python code emulating threads as greenlets,
which is my personal favorite: https://bitbucket.org/anacrolix/green380.
python3 goco.py <usual python parameters>.
On May 10, 2012 4:47 AM, "Sean Talts" <xitr...@gmail.com> wrote:
> I've never heard of this GIL issue with coroutines... What exactly
> happens? Do you know if this is still a problem in pypy? Do you have the
> source for your "pygevent" somewhere online I could fork? Thanks!
> On Wednesday, May 9, 2012 2:56:15 PM UTC-4, Matt Joiner wrote:
>> I've done this. It's better in that the implementation is only several
>> 100 lines. It's worse because the only reason it's useful is to work around
>> pathological contention in the GIL introduced by very high levels of
>> concurrency. Gevent reduces the concurrency overhead since most of it's
>> done in C.
>> On May 10, 2012 12:24 AM, "Sean Talts" <xitr...@gmail.com> wrote:
>>> It seems like we should be able to implement the gevent api in pure
>>> python somewhat easily (since pypy already has greenlets and event
>>> loops). Has anyone done this? Any reasons why it might be a bad
>>> idea?
>>> -Sean
>>> On Apr 27, 1:09 am, Matt Joiner <anacro...@gmail.com> wrote:
>>> > Indeed that sounds quite handy.
>>> > On Apr 27, 2012 3:03 AM, "Vasile Ermicioi" <elff...@gmail.com> wrote:
>>> > >> And STM is needed for gevent support?
>>> > > no, I gave the link because I think it responds to your other
>>> question ( is
>>> > > it expected that PyPy would provide a significant performance boost )
>>> > > "Notably, TM could benefit any event-based system that is written to
>>> > > dispatch events serially (Twisted-based, most GUI toolkit, Stackless,
>>> > > gevent, and so on). The events would internally be processed in
>>> parallel,
>>> > > while maintaining the illusion of serial execution, with all the
>>> > > corresponding benefits of safety. This should be possible with
>>> minimal
>>> > > changes to the event dispatchers. This approach has been described
>>> by the Automatic
>>> > > Mutual Exclusion<http://research.**microsoft.com/en-us/projects/** >>> ame/default.aspx<http://research.microsoft.com/en-us/projects/ame/default.aspx>>
>>> work
>>> > > at Microsoft Research, but not been implemented anywhere (to the
>>> best of
>>> > > our knowledge)."
On Thursday, May 10, 2012, Matt Joiner wrote:
> Sorry, I explained poorly. What I'm saying is that scheduled coroutines
> alleviate some negative aspects of the GIL. But doing the scheduling in
> pure Python the overhead is very high anyway, you may as well use native
> threading for the improved portability and compatibility with existing
> code. Your mileage may vary of course.
> Here's my Python 3, pure Python gevent clone:
> https://bitbucket.org/anacrolix/gthread The README contains an overview
> of my experiences with the model.
> Lastly, I wrote a pure Python script that monkey patches the stdlib and
> allows you to run unmodified Python code emulating threads as greenlets,
> which is my personal favorite: https://bitbucket.org/anacrolix/green380.
> python3 goco.py <usual python parameters>.
> On May 10, 2012 4:47 AM, "Sean Talts" <xitr...@gmail.com<javascript:_e({}, 'cvml', 'xitr...@gmail.com');>>
> wrote:
>> I've never heard of this GIL issue with coroutines... What exactly
>> happens? Do you know if this is still a problem in pypy? Do you have the
>> source for your "pygevent" somewhere online I could fork? Thanks!
>> On Wednesday, May 9, 2012 2:56:15 PM UTC-4, Matt Joiner wrote:
>>> I've done this. It's better in that the implementation is only several
>>> 100 lines. It's worse because the only reason it's useful is to work around
>>> pathological contention in the GIL introduced by very high levels of
>>> concurrency. Gevent reduces the concurrency overhead since most of it's
>>> done in C.
>>> On May 10, 2012 12:24 AM, "Sean Talts" <xitr...@gmail.com<javascript:_e({}, 'cvml', 'xitr...@gmail.com');>>
>>> wrote:
>>>> It seems like we should be able to implement the gevent api in pure
>>>> python somewhat easily (since pypy already has greenlets and event
>>>> loops). Has anyone done this? Any reasons why it might be a bad
>>>> idea?
>>>> -Sean
>>>> On Apr 27, 1:09 am, Matt Joiner <anacro...@gmail.com> wrote:
>>>> > Indeed that sounds quite handy.
>>>> > On Apr 27, 2012 3:03 AM, "Vasile Ermicioi" <elff...@gmail.com> wrote:
>>>> > >> And STM is needed for gevent support?
>>>> > > no, I gave the link because I think it responds to your other
>>>> question ( is
>>>> > > it expected that PyPy would provide a significant performance boost
>>>> )
>>>> > > "Notably, TM could benefit any event-based system that is written to
>>>> > > dispatch events serially (Twisted-based, most GUI toolkit,
>>>> Stackless,
>>>> > > gevent, and so on). The events would internally be processed in
>>>> parallel,
>>>> > > while maintaining the illusion of serial execution, with all the
>>>> > > corresponding benefits of safety. This should be possible with
>>>> minimal
>>>> > > changes to the event dispatchers. This approach has been described
>>>> by the Automatic
>>>> > > Mutual Exclusion<http://research.**microsoft.com/en-us/projects/** >>>> ame/default.aspx<http://research.microsoft.com/en-us/projects/ame/default.aspx>>
>>>> work
>>>> > > at Microsoft Research, but not been implemented anywhere (to the
>>>> best of
>>>> > > our knowledge)."
I have started on the gevent-ctypes port for PyPy. I already removed the cython stuff, and replaced libev with a ctypes wrapper for libev. Greenlet + JIT support is coming later on in PyPy, currently the JIT will not trace a greenlet, but at least it works.
On Tuesday, March 20, 2012 5:29:50 AM UTC+8, Hong Minhee wrote:
> As Floris Bruynooghe mentioned, PyPy alread provide its own greenletmodule. Another needed one is Cython. To make gevent to run under PyPy, > Cython firstly has to be ported to PyPy or gevent has to use ctypes instead > of (or as a counterpart of) Cython.
> On Monday, March 19, 2012 6:18:00 PM UTC+9, galfy wrote:
>> With every new version PyPy is becoming faster. If you believe the >> benchmarks [http://speed.pypy.org/] at the moment PyPy is 5x faster than >> the CPython. That makes me "crave" for a combination pypy+gevent. >> And this raises questions. >> What is needed in order to be able to run gevent under PyPy? >> Are there any efforts from third parties to make it possible? >> Are there any workarounds, like CPyExt [ >> http://morepypy.blogspot.de/2010/04/using-cpython-extension-modules-w...] >> that make this happen?
On Friday, June 29, 2012 8:29:32 PM UTC-7, brett wrote:
> I have started on the gevent-ctypes port for PyPy. I already removed the > cython stuff, and replaced libev with a ctypes wrapper for libev. > Greenlet + JIT support is coming later on in PyPy, currently the JIT will > not trace a greenlet, but at least it works.
> On Tuesday, March 20, 2012 5:29:50 AM UTC+8, Hong Minhee wrote:
>> As Floris Bruynooghe mentioned, PyPy alread provide its own greenletmodule. Another needed one is Cython. To make gevent to run under PyPy, >> Cython firstly has to be ported to PyPy or gevent has to use ctypes instead >> of (or as a counterpart of) Cython.
>> On Monday, March 19, 2012 6:18:00 PM UTC+9, galfy wrote:
>>> With every new version PyPy is becoming faster. If you believe the >>> benchmarks [http://speed.pypy.org/] at the moment PyPy is 5x faster >>> than the CPython. That makes me "crave" for a combination pypy+gevent. >>> And this raises questions. >>> What is needed in order to be able to run gevent under PyPy? >>> Are there any efforts from third parties to make it possible? >>> Are there any workarounds, like CPyExt [ >>> http://morepypy.blogspot.de/2010/04/using-cpython-extension-modules-w...] >>> that make this happen?
<swiss.army.engin...@gmail.com> wrote:
> Can I revive this old thread to ask the status? Has pypy-1.9 (which supports
> jit+greenlet) improved the situation any?
> There's basically nothing in the python world I want more than to have
> gevent-on-pypy for my worker daemons.
I think it would be easier to port eventlet to pypy rather than gevent
since pypy offer a greenlet compatible API.
Adam Smith wrote:
> Can I revive this old thread to ask the status? Has pypy-1.9 (which
> supports jit+greenlet) improved the situation any?
The PyPy guys seem to be going in the cffi direction, so I guess gevent's core would need to be written with cffi instead of Cython.
Having that said, I don't know if there would be any performance degradation when using CPython.
While it's not a simple task, but it can be done as a separated project: lets say you have some cffi bindings for libev, you could create a PyPyLoop class mirroring gevent.core.loop and then set it on Hub.loop_class, so your core is used instead of the cython based one. While not ideal, it may be a good start.
> There's basically nothing in the python world I want more than to have
> gevent-on-pypy for my worker daemons.
I'm really curious about this, are you finding any problems / limitations? Personally I'd rather get Python 3 support than PyPy.
