Adding new hub implementations to gevent
bsdemon
But for the first sight it is difficult to do, mostly because of
gevent.socket (for example) directly use gevent.core module. So, for
now, I have gevent branch with wait_read, wait_write functions (from
gevent.socket module) as gevent.hub.Hub methods and all tests are
passed. So gevent.socket now uses get_hub().{wait_read,wait_write}
functions and no one from gevent.core. I am planning to do the same
with other modules, which do not depend on libevent.
There is another issue with more than one hub implementations. Some
modules (gevent.http, gevent.wsgi, gevent.dns becomes unusable with no
pyevent hub. So, I think there is need for some kind of splitting in
gevent package for API and backend.
Denis, what do you think about that? What will you do if you would
need new hub for gevent and keep the performance at the same level?
P.S. Talking about performance, I mean not performance of particular
hub, but amount of overhead introduced by gevent library.
Denis Bilenko
On Mon, Jan 25, 2010 at 8:57 PM, bsdemon <8ma...@gmail.com> wrote:
> Hello, I have an idea to add new hub implementation to gevent.
This is great! And I think it'll only make the overall design better
as a side effect.
>
> But for the first sight it is difficult to do, mostly because of
> gevent.socket (for example) directly use gevent.core module. So, for
> now, I have gevent branch with wait_read, wait_write functions (from
> gevent.socket module) as gevent.hub.Hub methods and all tests are
> passed. So gevent.socket now uses get_hub().{wait_read,wait_write}
> functions and no one from gevent.core. I am planning to do the same
> with other modules, which do not depend on libevent.
It's the right approach. The functions from gevent.core should all be
made into Hub
methods, because otherwise it's impossible to take advantage of the
support for multiple event bases in libevent.
However, is there any reason to make wait_read() into Hub method?
It seems that simply fixing it by using get_hub().read_event() instead
of core.read_event()
should be enough.
>
> There is another issue with more than one hub implementations. Some
> modules (gevent.http, gevent.wsgi, gevent.dns becomes unusable with no
> pyevent hub. So, I think there is need for some kind of splitting in
> gevent package for API and backend.
Right. Maybe core and core wrappers - dns/http/wsgi should be put
in gevent.libevent package.
Then gevent.wsgi will reimport wsgi server from the backend or raise
error if backend does
not provide a wsgi server.
gevent.http and gevent.dns don't make sense without libevent so they
should remain in libevent package.
>
> Denis, what do you think about that? What will you do if you would
> need new hub for gevent and keep the performance at the same level?
>
> P.S. Talking about performance, I mean not performance of particular
> hub, but amount of overhead introduced by gevent library.
>
I see. I think overhead of using methods instead of functions is
insignificant if exists at all
and unavoidable because it is needed to support libevent's multiple
event bases. So, don't
worry about it.
BTW, DNS functions such as gethostbyname have to be Hub's methods too. I think
it's OK to just put the original blocking versions here for the first iteration.
Thanks for working on it!
Denis.
Andrey Popp
> However, is there any reason to make wait_read() into Hub method?
> It seems that simply fixing it by using get_hub().read_event() instead
> of core.read_event()
> should be enough.
Good point. It is question of the backend API style. Should we make it
coroutine-aware or raw callback style. The first is more natural in
context of current state of things, because Hub is greenlet and there
is no need to separate one from another.
Andrey.
Denis Bilenko
Well, I think read_event and similar async functions should be available
on Hub one way or another. They are lower-level than wait_read and could
have other uses. I don't want to make using callbacks impossible.
Whether or not to put wait_read() and similar on Hub I'm not sure.
I don't see a reason for it but since you are the one implementing other
hub you may see otherwise. You can do it the way you think is best and
we'll discuss it afterwards.