Why libev replaced libevent

5,072 views
Skip to first unread message

galfy

unread,
Apr 26, 2011, 11:43:44 AM4/26/11
to gevent: coroutine-based Python network library
I saw that in the recent months most of the development was going
towards replacing libev with libevent. Probably there is a reason for
this and I, and I guess the other gevent fans, will be happy to learn
it. Is libev faster, simpler to use, has http server that supports
streaming, etc?

Jan Persson

unread,
Apr 26, 2011, 3:09:35 PM4/26/11
to gev...@googlegroups.com
I might be totally wrong, but the thing that worries me the most, is
that switching to libev will make it impossible to get decent
performance from gevent on Windows. I personally don't care much about
that platform, but my customer has decided to use it for their
applications and I have to support them.

http://doc.dvgu.ru/devel/ev.html#win32_platform_limitations_and_workarounds

Libevent on Windows, uses I/O Completion Ports (IOCP) and scales
beautifully. From the link above, it looks like libev is using the old
select call and it is well known that this will not scale beyond a few
hundred sockets, on either Windows or Linux.

It looks like switching to libev will make gevent a UNIX-library only,
but I might be wrong.

--
Jan Persson - Esentus Technology AB - www.esentus.com - +46 702 854132 (mobile)

Travis Cline

unread,
Apr 27, 2011, 3:32:08 AM4/27/11
to gevent: coroutine-based Python network library
On Apr 26, 2:09 pm, Jan Persson <jan.pers...@gmail.com> wrote:
> I might be totally wrong, but the thing that worries me the most, is
> that switching to libev will make it impossible to get decent
> performance from gevent on Windows. I personally don't care much about
> that platform, but my customer has decided to use it for their
> applications and I have to support them.
>
> http://doc.dvgu.ru/devel/ev.html#win32_platform_limitations_and_worka...
>
> Libevent on Windows, uses I/O Completion Ports (IOCP) and scales
> beautifully. From the link above, it looks like libev is using the old
> select call and it is well known that this will not scale beyond a few
> hundred sockets, on either Windows or Linux.
>
> It looks like switching to libev will make gevent a UNIX-library only,
> but I might be wrong.

AFAICT you're basically correct. In order to support windows and unix
well, ry (node.js author) is working on https://github.com/joyent/liboio
which he announced at CodeConf.

Once it matures it's probably not unreasonable for gevent to target it
and get the ability to support both windows and file i/o well.

Travis

Jan Persson

unread,
Apr 27, 2011, 6:20:11 AM4/27/11
to gev...@googlegroups.com

Thanks for the explanation. So the plan is basically to first make a
transition to libev and then to liboio?
I hope that those of us who still needs to support Windows will have
the possible to change backend, in spite of this posting:

"Only libev will be supported in 0.14. However, other than gevent.core
the interface of gevent package is not changed, so transition should be smooth."

I (and the original poster) still wonders why there is a switch to
libev in the first place, when libevent is working? I rest assured
that there are good reasons for this, but could Denis care to explain?

Equand

unread,
Apr 27, 2011, 4:50:12 PM4/27/11
to gevent: coroutine-based Python network library
seems like problem with DNS arises with libevent only (their DNS
resolving is not fixed yet even in 2.0.)

On Apr 27, 12:20 pm, Jan Persson <jan.pers...@gmail.com> wrote:
> On Wed, Apr 27, 2011 at 09:32, Travis Cline <travis.cl...@gmail.com> wrote:
> > On Apr 26, 2:09 pm, Jan Persson <jan.pers...@gmail.com> wrote:
> > AFAICT you're basically correct. In order to support windows and unix
> > well, ry (node.js author) is working onhttps://github.com/joyent/liboio

CryptWizard

unread,
Apr 27, 2011, 9:06:59 PM4/27/11
to gev...@googlegroups.com
I think a workaround for DNS would be easier than a workaround for Windows.

Denis Bilenko

unread,
Apr 28, 2011, 1:32:52 PM4/28/11
to gev...@googlegroups.com

faster - probably somewhat, but that wasn't the reason for switch.
simpler to use - yes, the API is really nicer.
has http server - no, although a few servers have been written by 3rd parties.

I put some details in a blog post:
http://blog.gevent.org/2011/04/28/libev-and-libevent/

On Wed, Apr 27, 2011 at 2:09 AM, Jan Persson <jan.p...@gmail.com> wrote:
> Libevent on Windows, uses I/O Completion Ports (IOCP) and scales
> beautifully. From the link above, it looks like libev is using the old
> select call and it is well known that this will not scale beyond a few
> hundred sockets, on either Windows or Linux.

Libevent2 has support for IOCP, but it's not automatic, you need to
use asynchronous bufferevents to use it.
Gevent does not wrap bufferevents, let alone asynchronous. The
libevent itself does not use async bufferevents in DNS, HTTP client
and HTTP server.

Thus there's no way to take advantage of IOCP in gevent 0.13 and so
switching to libev does not imply degraded performance on Windows (it
remains poor).

> It looks like switching to libev will make gevent a UNIX-library only,
> but I might be wrong.

It won't become UNIX-only, I think gevent 0.14 will support Windows at
least to the same extent as gevent 0.13 supported it.

> On Wed, Apr 27, 2011 at 09:32, Travis Cline <travis...@gmail.com> wrote:
> ry (node.js author) is working on https://github.com/joyent/liboio


> which he announced at CodeConf.
>
> Once it matures it's probably not unreasonable for gevent to target it
> and get the ability to support both windows and file i/o well.

It wasn't the plan to use liboio as I started using libev before that
was announced, but that's an interesting project to watch and when it
matures we can consider using as a base layer for gevent.

James Simmons

unread,
Apr 28, 2011, 1:45:15 PM4/28/11
to gevent: coroutine-based Python network library
> Thus there's no way to take advantage of IOCP in gevent 0.13 and so
> switching to libev does not imply degraded performance on Windows (it
> remains poor).

I've only had a chance so far to use gevent under Windows XP. I have
my DNS settings pointing to a localhost instance of BIND9 and I've
noticed that about 10% of the HTTP GET requests I send (using the non-
monkey patched httplib) fail with DNS errors. The majority of the
failures are either "unknown" or "request timed out." Is this a
symptom of the poor performance under Windows or do you believe my
problem lies elsewhere (perhaps with my BIND configuration -- which is
default in all ways)?

Denis Bilenko

unread,
Apr 28, 2011, 1:58:03 PM4/28/11
to gev...@googlegroups.com

You are talking about gevent 0.13, right?
I think this the kind of problem that is fixed in the trunk by
switching to c-ares for DNS.
However you have to wait while trunk's Windows bugs fixed before you
can try that (or help fixing them).

James Simmons

unread,
Apr 28, 2011, 2:47:56 PM4/28/11
to gevent: coroutine-based Python network library
On Apr 28, 10:58 am, Denis Bilenko <denis.bile...@gmail.com> wrote:
> You are talking about gevent 0.13, right?
> I think this the kind of problem that is fixed in the trunk by
> switching to c-ares for DNS.
> However you have to wait while trunk's Windows bugs fixed before you
> can try that (or help fixing them).

Yes I'm using 0.13. Glad to hear that the issues I'm experiencing are
fixed in trunk. I probably won't be of much use in fixing the issues
though unless it's something specific to the Python portion of the
project. I haven't had any luck getting it to compile (though I never
have any luck getting things to compile in Windows).

Jan Persson

unread,
Apr 28, 2011, 3:50:05 PM4/28/11
to gev...@googlegroups.com
> Thus there's no way to take advantage of IOCP in gevent 0.13 and so
> switching to libev does not imply degraded performance on Windows (it
> remains poor).

Thanks for the answer. I was under the impression that IOCP was being
used, since gevent has scaled so good for our Windows project (which
has hundreds of clients) and this discussion made me a bit worried
that the good days were over.

Thanks again for this excellent library.

Galfy Pundee

unread,
Apr 29, 2011, 8:31:41 AM4/29/11
to gev...@googlegroups.com
Thanks for the information. Let's hope that simplification can lead to
cleaner and faster code :)

Regards,
Gal

Andy

unread,
Apr 30, 2011, 11:42:28 AM4/30/11
to gevent: coroutine-based Python network library


On Apr 28, 1:32 pm, Denis Bilenko <denis.bile...@gmail.com> wrote:

> faster - probably somewhat, but that wasn't the reason for switch.
> simpler to use - yes, the API is really nicer.
> has http server - no, although a few servers have been written by 3rd parties.

What http server will gevent 0.14 be using then?

Jaime

unread,
May 14, 2011, 6:27:36 AM5/14/11
to gevent: coroutine-based Python network library
I'm also very interested in the impact of libev regarding wsgi support
of gevent. We've used libevent2 wsgi module instead of pure wsgi in
order to maximize the overall performance (which is really
noticeable).
Denis, do you have any plan to support fast, native wsgi when porting
to libev? Perhaps using third-party projects like https://github.com/jonashaag/bjoern
which is also based on libev?

Denis Bilenko

unread,
May 16, 2011, 3:19:33 AM5/16/11
to gev...@googlegroups.com
On Sat, May 14, 2011 at 5:27 PM, Jaime <jjj...@gmail.com> wrote:
> Denis, do you have any plan to support fast, native wsgi when porting
> to libev? Perhaps using third-party projects like https://github.com/jonashaag/bjoern
> which is also based on libev?

Yes, I do want to have fast implementations of gevent.httplib and
gevent.wsgi in 0.14 that are at least as
good as in 0.13. Not sure about the specifics yet.

Jaime

unread,
May 16, 2011, 11:35:23 AM5/16/11
to gevent: coroutine-based Python network library
Thanks Denis. You are doing a great job!

On 16 mayo, 09:19, Denis Bilenko <denis.bile...@gmail.com> wrote:
> On Sat, May 14, 2011 at 5:27 PM, Jaime <jjja...@gmail.com> wrote:
> > Denis, do you have any plan to support fast, native wsgi when porting
> > to libev? Perhaps using third-party projects likehttps://github.com/jonashaag/bjoern
Reply all
Reply to author
Forward
0 new messages