|Gevent and PyPy||galfy||3/19/12 2:18 AM|
With every new version PyPy is becoming faster. If you believe the benchmarks [http://speed.pypy.org/] at the moment PyPy is 5x faster than the CPython. That makes me "crave" for a combination pypy+gevent.
And this raises questions.
What is needed in order to be able to run gevent under PyPy?
Are there any efforts from third parties to make it possible?
Are there any workarounds, like CPyExt [http://morepypy.blogspot.de/2010/04/using-cpython-extension-modules-with.html] that make this happen?
|Re: [gevent] Gevent and PyPy||Damien Churchill||3/19/12 6:27 AM|
On 19 March 2012 09:18, galfy <galfyo...@googlemail.com> wrote:
gevent would have to have support for running using PyPys stackless mode.
|Re: [gevent] Gevent and PyPy||Floris Bruynooghe||3/19/12 6:54 AM|
Actually pypy has it's own version of the greenlet module in their
|Re: Gevent and PyPy||Hong Minhee||3/19/12 2:29 PM|
As Floris Bruynooghe mentioned, PyPy alread provide its own greenlet module. Another needed one is Cython. To make gevent to run under PyPy, Cython firstly has to be ported to PyPy or gevent has to use ctypes instead of (or as a counterpart of) Cython.
|Re: [gevent] Re: Gevent and PyPy||Damien Churchill||3/19/12 4:12 PM|
On 19 March 2012 21:29, Hong Minhee <hong....@gmail.com> wrote:
Or the cython modules are re-written in python using ctypes for use in PyPy.
|Re: [gevent] Re: Gevent and PyPy||Vasile Ermicioi||3/20/12 12:28 AM|
at the moment PyPy is 5x faster than the CPython. That makes me "crave" for a combination pypy+gevent
I think greenlet module of pypy is not jit-able, that is, it may be slower than with cpython
|Re: [gevent] Re: Gevent and PyPy||Antonin AMAND||3/21/12 12:48 AM|
It seems there is a Python C level API compatibility layer for pypy.
Or am I missing something ?
|Re: [gevent] Re: Gevent and PyPy||Antonin AMAND||3/21/12 1:05 AM|
Actually this article  discourage from running gevent on top of cpyext/pypy:
"cpyext is slow, and always will be slow. It has to emulate far too
|Re: [gevent] Re: Gevent and PyPy||André Cruz||4/26/12 4:32 AM|
So, there are no efforts in progress to make gevent usable under PyPy?
More importantly, is it expected that PyPy would provide a significant performance boost?
|Re: [gevent] Re: Gevent and PyPy||Vasile Ermicioi||4/26/12 9:01 AM|
Armin Rigo is working on STM,
|Re: Re: [gevent] Gevent and PyPy||André Cruz||4/26/12 9:07 AM|
And STM is needed for gevent support?
|Re: [gevent] Re: Gevent and PyPy||Matt Joiner||4/26/12 9:12 AM|
gevent attempts to work around concurrency problems in Python by reducing the dependence on native threading. STM will likely reduce single threaded performance, and by extension gevent.
There's also a good chance that C extensions won't work well, and that PyPy's greenlet interface will be a low priority because of how it relates to the goals of introducing STM.
|Re: [gevent] Re: Gevent and PyPy||Randall Leeds||4/26/12 11:46 AM|
On Thu, Apr 26, 2012 at 09:12, Matt Joiner <anac...@gmail.com> wrote:I'm not sure I see that. There's been quite a lot of activity on the
pypy-dev list discussing ways to update PyPy's cpyext and bring some
things better in line with Cython and CPython. I can't say I've
followed it all super closely, but from my understanding making Cython
output compile with PyPy is the direction to go and the pypy-dev list
is where you want to go to talk about it.
A lot of the current discussion revolves around making C calls faster,
with fewer layers of indirection (the GC makes this difficult).
Unfortunately, calling C code from PyPy isn't brilliantly fast right
now, so it's not clear that gevent+pypy would be much better than a
pypy wsgi server without gevent.
At least... that's what I get from the discussions I'm seeing.
|Re: Re: [gevent] Gevent and PyPy||Vasile Ermicioi||4/26/12 12:03 PM|
no, I gave the link because I think it responds to your other question ( is it expected that PyPy would provide a significant performance boost )
"Notably, TM could benefit any event-based system that is written to dispatch events serially (Twisted-based, most GUI toolkit, Stackless, gevent, and so on). The events would internally be processed in parallel, while maintaining the illusion of serial execution, with all the corresponding benefits of safety. This should be possible with minimal changes to the event dispatchers. This approach has been described by the Automatic Mutual Exclusion work at Microsoft Research, but not been implemented anywhere (to the best of our knowledge)."
|Re: Re: [gevent] Gevent and PyPy||Matt Joiner||4/26/12 10:09 PM|
Indeed that sounds quite handy.
|Re: Gevent and PyPy||Sean Talts||5/9/12 9:24 AM|
It seems like we should be able to implement the gevent api in pure
python somewhat easily (since pypy already has greenlets and event
loops). Has anyone done this? Any reasons why it might be a bad
> On Apr 27, 2012 3:03 AM, "Vasile Ermicioi" <elff...@gmail.com> wrote:> > Mutual Exclusion<http://research.microsoft.com/en-us/projects/ame/default.aspx> work
|Re: [gevent] Re: Gevent and PyPy||Floris Bruynooghe||5/9/12 11:28 AM|
On 9 May 2012 17:24, Sean Talts <xit...@gmail.com> wrote:This is essentially what eventlet does. I had a quick experiment and
simple sockets seem to work with eventlet on pypy but urllib2 does
give a strange error. Not sure I'll spend much time investigating
why, but it's interesting anyway.
|Re: [gevent] Re: Gevent and PyPy||Matt Joiner||5/9/12 11:56 AM|
I've done this. It's better in that the implementation is only several 100 lines. It's worse because the only reason it's useful is to work around pathological contention in the GIL introduced by very high levels of concurrency. Gevent reduces the concurrency overhead since most of it's done in C.
|Re: [gevent] Re: Gevent and PyPy||Sean Talts||5/9/12 1:47 PM|
I've never heard of this GIL issue with coroutines... What exactly happens? Do you know if this is still a problem in pypy? Do you have the source for your "pygevent" somewhere online I could fork? Thanks!
|Re: [gevent] Re: Gevent and PyPy||Matt Joiner||5/9/12 5:29 PM|
Sorry, I explained poorly. What I'm saying is that scheduled coroutines alleviate some negative aspects of the GIL. But doing the scheduling in pure Python the overhead is very high anyway, you may as well use native threading for the improved portability and compatibility with existing code. Your mileage may vary of course.
Here's my Python 3, pure Python gevent clone: https://bitbucket.org/anacrolix/gthread The README contains an overview of my experiences with the model.
I also wrote an asynchronous framework using the new PEP 380 for Python 3.3: https://bitbucket.org/anacrolix/green380
Lastly, I wrote a pure Python script that monkey patches the stdlib and allows you to run unmodified Python code emulating threads as greenlets, which is my personal favorite: https://bitbucket.org/anacrolix/green380. python3 goco.py <usual python parameters>.
|Re: [gevent] Re: Gevent and PyPy||Matt Joiner||5/12/12 2:05 AM|
Sorry the last link should have been https://bitbucket.org/anacrolix/goco for the pure Python framework.
|Re: Gevent and PyPy||brett||6/29/12 8:29 PM|
I have started on the gevent-ctypes port for PyPy. I already removed the cython stuff, and replaced libev with a ctypes wrapper for libev.
Greenlet + JIT support is coming later on in PyPy, currently the JIT will not trace a greenlet, but at least it works.
You can get the source from here:
|Re: Gevent and PyPy||Adam Smith||8/21/12 4:01 PM|
Can I revive this old thread to ask the status? Has pypy-1.9 (which supports jit+greenlet) improved the situation any?
There's basically nothing in the python world I want more than to have gevent-on-pypy for my worker daemons.
|Re: [gevent] Re: Gevent and PyPy||Benoit Chesneau||8/22/12 12:42 AM|
On Wed, Aug 22, 2012 at 1:01 AM, Adam SmithI think it would be easier to port eventlet to pypy rather than gevent
since pypy offer a greenlet compatible API.
|Re: [gevent] Re: Gevent and PyPy||Saúl Ibarra Corretgé||8/22/12 2:27 AM|
The PyPy guys seem to be going in the cffi direction, so I guess
gevent's core would need to be written with cffi instead of Cython.
Having that said, I don't know if there would be any performance
degradation when using CPython.
While it's not a simple task, but it can be done as a separated project:
lets say you have some cffi bindings for libev, you could create a
PyPyLoop class mirroring gevent.core.loop and then set it on
Hub.loop_class, so your core is used instead of the cython based one.
While not ideal, it may be a good start.
I'm really curious about this, are you finding any problems /
limitations? Personally I'd rather get Python 3 support than PyPy.
Saúl Ibarra Corretgé
http://saghul.net/blog | http://about.me/saghul
|Re: [gevent] Re: Gevent and PyPy||smurf||8/22/12 4:05 AM|
Sa�l Ibarra Corretg�:
> Personally I'd rather get Python 3 support than PyPy.
Personally I'd rather get both, including Py3 support for PyPy.
Fortunately, people are working on it.
-- Matthias Urlichs
|Re: [gevent] Re: Gevent and PyPy||Ralf Schmitt||8/22/12 5:01 AM|
Saúl Ibarra Corretgé <sag...@gmail.com> writes:
> The PyPy guys seem to be going in the cffi direction, so I guess
> gevent's core would need to be written with cffi instead of Cython.
cython 0.17 will have basic support for pypy . that would be another
|Re: [gevent] Re: Gevent and PyPy||Saúl Ibarra Corretgé||8/22/12 11:50 PM|
Indeed. However, last time I checked the PyPy guys said that cpyext is
and will always be slow, that's one of the reasons why cffi was created
IMHO, Gevent shouldn't get slower on CPython in order to support PyPy.
Saúl Ibarra Corretgé
http://saghul.net/blog | http://about.me/saghul
|pypycore [was: [gevent] Re: Gevent and PyPy]||Ralf Schmitt||9/15/12 2:59 PM|
Saúl Ibarra Corretgé <sag...@gmail.com> writes:> Having that said, I don't know if there would be any performance
> degradation when using CPython.
> While it's not a simple task, but it can be done as a separated
> project: lets say you have some cffi bindings for libev, you could
> create a PyPyLoop class mirroring gevent.core.loop and then set it on
> Hub.loop_class, so your core is used instead of the cython based
> one. While not ideal, it may be a good start.
i've done that in: https://github.com/schmir/pypycore
it's a reimplementation of gevent.core using cffi. it's not complete at
the moment. but it already allows me to serve a hello world app with
pywsgi (with cpython).
running it on pypy currently requires the development version of
pypy. note that there seems to be a problem with pypy's greenlet
implementation. i'll have to dig a bit further into this however.
|Re: pypycore [was: [gevent] Re: Gevent and PyPy]||Saúl Ibarra Corretgé||9/18/12 1:32 PM|
I spotted that, looks great :-)
I'm also working on an alternative gevent core, based on libuv (using pyuv) and one of the things I had to "fake" was incrementing the refcount of the watchers even if no one holds a reference to them. I see you comented that line here: https://github.com/schmir/pypycore/blob/master/pypycore.py#L638 My workaround for that is to keep a list of watchers attached to the loop and add and remove them as needed. Don't you need to do something similar? Or maybe I'm misunderstanding something :-)
|Re: pypycore [was: [gevent] Re: Gevent and PyPy]||Ralf Schmitt||9/19/12 1:20 AM|
Saúl Ibarra Corretgé <sag...@gmail.com> writes:
> I'm also working on an alternative gevent core, based on libuv (usingdo you have any code to show us?
yes, I think so. At the moment the watcher may be kept alive by a
reference from the cffi callback. but it would be collected if gc
detects the cycle. I'll have to look into it.
|Re: pypycore [was: [gevent] Re: Gevent and PyPy]||Saúl Ibarra Corretgé||9/19/12 3:15 PM|
Ralf Schmitt wrote:I'll upload it as soon as I come back from a short trip.
|Re: pypycore [was: [gevent] Re: Gevent and PyPy]||Saúl Ibarra Corretgé||10/4/12 3:20 PM|
Ralf Schmitt wrote:Now I do :-) https://github.com/saghul/uvent
|Re: pypycore [was: [gevent] Re: Gevent and PyPy]||Denis Bilenko||10/5/12 5:25 AM|
On Fri, Oct 5, 2012 at 12:20 AM, Saúl Ibarra Corretgé <sag...@gmail.com> wrote:
> Now I do :-) https://github.com/saghul/uventIt seems that the best results would be achieved by using libuv's TCP
and UDP watchers in gevent.socket/ssl. Without it there's no advantage
in using this over bare libev.
Do you plan to do something like that? It would be a separate branch
of gevent, rather than just a new loop implementation.
|Re: pypycore [was: [gevent] Re: Gevent and PyPy]||Saúl Ibarra Corretgé||10/6/12 12:17 AM|
The UDP and TCP handles are implemented internally more or less like the poll handle, but doing 64k reads. It may perform better since most of the work is done in C land, but I don't have numbers on that. I'll explore the possibility of doing this.
One good advantage of using libuv instead of liev is Windows support. libuv uses some obscure Windows stuff (sometimes undocumented APIs IIRC) to have a epoll like thing for the poll handles, while libev is only able to do select AFAIK.
Also, there are a ton of file operations implemented already, and in a cross-platform way. File reads and writes are not serialized yet, but there are plans to make files like streams in libuv.
Last, libuv will remove libev (yes, it currently uses it internally) and use epoll in edge triggered mode in Linux, which should also bring a performance boost.
|Re: pypycore [was: [gevent] Re: Gevent and PyPy]||Michael So||10/6/12 1:43 AM|
a huge different is libuv use IOCP in Windows instead of select(). I have tried IOCP before, you can treat it like epoll or kqueue, but IOCP is actually better in someway.
libuv inherit libev and adding IOCP on windows.
in addition, it provides some more useful features like file operations as Saúl said.
so, we u have or need in libev, libuv also has it and has it better.
it would be great to see gevent would replace libev with libuv soon. :)