[Python-ideas] Add a generic async IO poller/reactor to select module

31 views
Skip to first unread message

Giampaolo Rodolà

unread,
May 23, 2012, 9:32:42 PM5/23/12
to python...@python.org
Including an established async IO framework such as Twisted, gevent or
Tornado in the Python stdlib has always been a controversial subject.
PEP-3153 (http://www.python.org/dev/peps/pep-3153/) tried to face this
problem in the most agnostic way as possible, and it's a good starting
point IMO.
Nevertheless, it's still vague about what the actual API should look
like and AFAIK it remained stagnant so far.

There's one thing in the whole async stack which is basically the same
for all implementations though: the poller/reactor.
Could it make sense to add something similar to select module?
Differently from PEP-3153, providing such a layer on top of select(),
poll() & co. is easier and could possibly be an incentive to avoid
such code duplication.

I'm coming up with this because I recently did something similar in
pyftpdlib as an hack on top of asyncore to add support for epoll() and
kqueue(), using the excellent Tornado's io loop as source of
inspiration:
http://code.google.com/p/pyftpdlib/issues/detail?id=203
http://code.google.com/p/pyftpdlib/source/browse/trunk/pyftpdlib/lib/ioloop.py


The way I imagine it:

>>> import select
>>> dir(select)
[..., 'EpollPoller', 'PollPoller', 'SelectPoller', 'KqueuePoller']
>>> poller = select.EpollPoller()
>>> poller.register(fd, handler, poller.READ | poller.WRITE)
>>> poller.socket_map
{2 : <Handler instance at ...>}
>>> poller.modify(fd, poller.READ)
>>> poller.poll() # will call handler.handle_read_event() if/when it's the case
^C
KeyboardInterrupt
>>> poller.remove(fd)
>>> poller.close()

The handler is supposed to provide 3 methods:
- handle_read_event
- handle_write_event
- handle_error_event

Users willing to support multiple event loops such as wx, gtk etc can do:

>>> while 1:
... poller.poll(timeout=0.1, blocking=False)
... otherpoller.poll()


Basically, this would be the whole API.

Thoughts?


--- Giampaolo
http://code.google.com/p/pyftpdlib/
http://code.google.com/p/psutil/
http://code.google.com/p/pysendfile/
_______________________________________________
Python-ideas mailing list
Python...@python.org
http://mail.python.org/mailman/listinfo/python-ideas

Giampaolo Rodolà

unread,
May 23, 2012, 9:43:33 PM5/23/12
to python...@python.org
2012/5/24 Giampaolo Rodolà <g.ro...@gmail.com>:
> The handler is supposed to provide 3 methods:
> - handle_read_event
> - handle_write_event
> - handle_error_event

Further note: this is the approach I used in pyftpdlib.
An even more abstracted approach would be having poller.poll() return
a dict of {fd: events, fd, events, ...}, similarly to what Tornado
currently does.
This way we wouldn't be forcing the user to provide a handler class
with the 3 methods described above.

Ronald Oussoren

unread,
May 24, 2012, 2:47:35 AM5/24/12
to Giampaolo Rodolà, python...@python.org

On 24 May, 2012, at 3:32, Giampaolo Rodolà wrote:
>>>>
>
> The handler is supposed to provide 3 methods:
> - handle_read_event
> - handle_write_event
> - handle_error_event
>
> Users willing to support multiple event loops such as wx, gtk etc can do:
>
>>>> while 1:
> ... poller.poll(timeout=0.1, blocking=False)
> ... otherpoller.poll()
>
>
> Basically, this would be the whole API.

Isn't this a limited version of asyncore? (poller.poll == asyncore.loop, the handler is a subset of asyncore.dispatcher).

Ronald

Giampaolo Rodolà

unread,
May 24, 2012, 7:50:27 AM5/24/12
to Ronald Oussoren, python...@python.org
2012/5/24 Ronald Oussoren <ronaldo...@mac.com>:
poller.poll serves the same purpose of asyncore.loop, yes, but this is
supposed to be independent from asyncore.

Antoine Pitrou

unread,
May 24, 2012, 8:03:14 AM5/24/12
to python...@python.org
On Thu, 24 May 2012 13:50:27 +0200
Giampaolo Rodolà <g.ro...@gmail.com>
wrote:
> 2012/5/24 Ronald Oussoren <ronaldo...@mac.com>:
> >
> > On 24 May, 2012, at 3:32, Giampaolo Rodolà wrote:
> >>>>>
> >>
> >> The handler is supposed to provide 3 methods:
> >> - handle_read_event
> >> - handle_write_event
> >> - handle_error_event
> >>
> >> Users willing to support multiple event loops such as wx, gtk etc can do:
> >>
> >>>>> while 1:
> >> ...       poller.poll(timeout=0.1, blocking=False)
> >> ...       otherpoller.poll()
> >>
> >>
> >> Basically, this would be the whole API.
> >
> > Isn't this a limited version of asyncore? (poller.poll == asyncore.loop, the handler is a subset of asyncore.dispatcher).
> >
> > Ronald
>
> poller.poll serves the same purpose of asyncore.loop, yes, but this is
> supposed to be independent from asyncore.

I agree with Ronald that it looks like a less-braindead version of
asyncore. I don't think the select module is the right place.

Also, I don't know why you would specify poller.READ or poller.WRITE
explicitly. Usually you are interested in all events, no?

Regards

Antoine.

Nick Coghlan

unread,
May 24, 2012, 8:37:03 AM5/24/12
to Giampaolo Rodolà, python...@python.org
On Thu, May 24, 2012 at 9:50 PM, Giampaolo Rodolà <g.ro...@gmail.com> wrote:
> poller.poll serves the same purpose of asyncore.loop, yes, but this is
> supposed to be independent from asyncore.

I'd actually like to see something like this pitched as a
"concurrent.eventloop" PEP. PEP 3153 really wasn't what I was
expecting after the discussions at the PyCon US 2011 language summit -
I was expecting "here's a common event loop all the async frameworks
can hook into", but instead we got something a *lot* more ambitious
taht tried to merge the entire IO stack for the async frameworks,
rather than just provide a standard way for their event loops to
cooperate.

Cheers,
Nick.

--
Nick Coghlan   |   ncog...@gmail.com   |   Brisbane, Australia

Giampaolo Rodolà

unread,
May 24, 2012, 8:45:01 AM5/24/12
to Antoine Pitrou, python...@python.org
2012/5/24 Antoine Pitrou <soli...@pitrou.net>:
> On Thu, 24 May 2012 13:50:27 +0200
> Giampaolo Rodolà <g.ro...@gmail.com>
> wrote:
>> 2012/5/24 Ronald Oussoren <ronaldo...@mac.com>:
>> >
>> > On 24 May, 2012, at 3:32, Giampaolo Rodolà wrote:
>> >>>>>
>> >>
>> >> The handler is supposed to provide 3 methods:
>> >> - handle_read_event
>> >> - handle_write_event
>> >> - handle_error_event
>> >>
>> >> Users willing to support multiple event loops such as wx, gtk etc can do:
>> >>
>> >>>>> while 1:
>> >> ...       poller.poll(timeout=0.1, blocking=False)
>> >> ...       otherpoller.poll()
>> >>
>> >>
>> >> Basically, this would be the whole API.
>> >
>> > Isn't this a limited version of asyncore? (poller.poll == asyncore.loop, the handler is a subset of asyncore.dispatcher).
>> >
>> > Ronald
>>
>> poller.poll serves the same purpose of asyncore.loop, yes, but this is
>> supposed to be independent from asyncore.
>
> I agree with Ronald that it looks like a less-braindead version of
> asyncore. I don't think the select module is the right place.

Yeah, probably. Usually when I post here I'm the first one not being
sure whether what I propose is a good idea or not. =)
Anyway, it must be clear that what I have in mind is not related to
asyncore per-se.
The proposal is to add a *generic* poller/reactor to select module as
an abstraction layer on top of select(), poll(), epoll() and kqueue(),
that's all.

> Also, I don't know why you would specify poller.READ or poller.WRITE
> explicitly. Usually you are interested in all events, no?

Nope, that's what asyncore does and that's why it is significantly
slower compared to more modern and clever async loops (independenly
from the lack of epoll() / kqueue() support in asyncore).
You should only be interested in reading for accepting sockets
(servers) or when you want to receive data.
You should only be interested in writing for connecting sockets
(clients) or when you want to send data.
Being interested in both when, say, you only intend to receive data is
a considerable waste of time, especially when there are many
concurrent connections.
The performance degradation if you wildly look for both read and write
events is *huge*, see benchmarks referring to old vs. new select()
implementation here (~8.5x slowdown with 200 concurrent clients):
http://code.google.com/p/pyftpdlib/issues/detail?id=203#c6

Nick Coghlan

unread,
May 24, 2012, 8:51:36 AM5/24/12
to Giampaolo Rodolà, python...@python.org
On Thu, May 24, 2012 at 10:37 PM, Nick Coghlan <ncog...@gmail.com> wrote:
> On Thu, May 24, 2012 at 9:50 PM, Giampaolo Rodolà <g.ro...@gmail.com> wrote:
>> poller.poll serves the same purpose of asyncore.loop, yes, but this is
>> supposed to be independent from asyncore.
>
> I'd actually like to see something like this pitched as a
> "concurrent.eventloop" PEP. PEP 3153 really wasn't what I was
> expecting after the discussions at the PyCon US 2011 language summit -
> I was expecting "here's a common event loop all the async frameworks
> can hook into", but instead we got something a *lot* more ambitious
> taht tried to merge the entire IO stack for the async frameworks,
> rather than just provide a standard way for their event loops to
> cooperate.

See the final section of my notes here:
http://www.boredomandlaziness.org/2011/03/python-language-summit-rough-notes.html

Turns out the idea of a PEP 3153 level API *was* raised at the summit,
but I'd still like to see a competing PEP that targets the reactor
level API directly.

Ronald Oussoren

unread,
May 24, 2012, 8:52:59 AM5/24/12
to Antoine Pitrou, python...@python.org

On 24 May, 2012, at 14:03, Antoine Pitrou wrote:

> On Thu, 24 May 2012 13:50:27 +0200
> Giampaolo Rodolà <g.ro...@gmail.com>
> wrote:
>> 2012/5/24 Ronald Oussoren <ronaldo...@mac.com>:
>>>
>>> On 24 May, 2012, at 3:32, Giampaolo Rodolà wrote:
>>>>>>>
>>>>
>>>> The handler is supposed to provide 3 methods:
>>>> - handle_read_event
>>>> - handle_write_event
>>>> - handle_error_event
>>>>
>>>> Users willing to support multiple event loops such as wx, gtk etc can do:
>>>>
>>>>>>> while 1:
>>>> ... poller.poll(timeout=0.1, blocking=False)
>>>> ... otherpoller.poll()
>>>>
>>>>
>>>> Basically, this would be the whole API.
>>>
>>> Isn't this a limited version of asyncore? (poller.poll == asyncore.loop, the handler is a subset of asyncore.dispatcher).
>>>
>>> Ronald
>>
>> poller.poll serves the same purpose of asyncore.loop, yes, but this is
>> supposed to be independent from asyncore.
>
> I agree with Ronald that it looks like a less-braindead version of
> asyncore. I don't think the select module is the right place.

What worries me most is that it might only look like a beter version of asyncore. I'd much rather see something based on the event-handling core of Twisted because that code base is used in production and is hence more likely to be correct w.r.t. odd real-world conditions. IIRC doing this was discussed at the language summit in 2011, but as Nick mentions that doesn't seem to be the focus of PEP 3153.

I am by the way not using Twisted myself, I'm at this time still using homebrew select loops and asyncore.

>
> Also, I don't know why you would specify poller.READ or poller.WRITE
> explicitly. Usually you are interested in all events, no?

You're not always interested in write events, those are only interesting when you have data that must be written to a socket.

Ronald

Giampaolo Rodolà

unread,
May 24, 2012, 9:06:18 AM5/24/12
to Ronald Oussoren, Antoine Pitrou, python...@python.org
2012/5/24 Ronald Oussoren <ronaldo...@mac.com>:
>
> On 24 May, 2012, at 14:03, Antoine Pitrou wrote:
>
>> On Thu, 24 May 2012 13:50:27 +0200
>> Giampaolo Rodolà <g.ro...@gmail.com>
>> wrote:
>>> 2012/5/24 Ronald Oussoren <ronaldo...@mac.com>:
>>>>
>>>> On 24 May, 2012, at 3:32, Giampaolo Rodolà wrote:
>>>>>>>>
>>>>>
>>>>> The handler is supposed to provide 3 methods:
>>>>> - handle_read_event
>>>>> - handle_write_event
>>>>> - handle_error_event
>>>>>
>>>>> Users willing to support multiple event loops such as wx, gtk etc can do:
>>>>>
>>>>>>>> while 1:
>>>>> ...       poller.poll(timeout=0.1, blocking=False)
>>>>> ...       otherpoller.poll()
>>>>>
>>>>>
>>>>> Basically, this would be the whole API.
>>>>
>>>> Isn't this a limited version of asyncore? (poller.poll == asyncore.loop, the handler is a subset of asyncore.dispatcher).
>>>>
>>>> Ronald
>>>
>>> poller.poll serves the same purpose of asyncore.loop, yes, but this is
>>> supposed to be independent from asyncore.
>>
>> I agree with Ronald that it looks like a less-braindead version of
>> asyncore. I don't think the select module is the right place.
>
> What worries me most is that it might only look like a beter version of asyncore.

Please, forget about asyncore: this has nothing to do with it per-se
as it's just a reactor - it doesn't aim to provide any connection
handling.
Given the poor asyncore API I doubt it would be even integrable with
it without breaking backward compatibility.

Antoine Pitrou

unread,
May 24, 2012, 9:06:59 AM5/24/12
to python...@python.org
On Thu, 24 May 2012 22:37:03 +1000
Nick Coghlan <ncog...@gmail.com> wrote:
> On Thu, May 24, 2012 at 9:50 PM, Giampaolo Rodolà <g.rodola-Re5JQE...@public.gmane.org> wrote:
> > poller.poll serves the same purpose of asyncore.loop, yes, but this is
> > supposed to be independent from asyncore.
>
> I'd actually like to see something like this pitched as a
> "concurrent.eventloop" PEP.

Sounds like a good idea to me. By the way, it should also have some
support for delayed calls to be actually useful (something that
asyncore *still* doesn't have, AFAIK).

Regards

Antoine.

Giampaolo Rodolà

unread,
May 24, 2012, 2:40:31 PM5/24/12
to Nick Coghlan, python...@python.org
2012/5/24 Nick Coghlan <ncog...@gmail.com>:
> On Thu, May 24, 2012 at 10:37 PM, Nick Coghlan <ncog...@gmail.com> wrote:
>> On Thu, May 24, 2012 at 9:50 PM, Giampaolo Rodolà <g.ro...@gmail.com> wrote:
>>> poller.poll serves the same purpose of asyncore.loop, yes, but this is
>>> supposed to be independent from asyncore.
>>
>> I'd actually like to see something like this pitched as a
>> "concurrent.eventloop" PEP. PEP 3153 really wasn't what I was
>> expecting after the discussions at the PyCon US 2011 language summit -
>> I was expecting "here's a common event loop all the async frameworks
>> can hook into", but instead we got something a *lot* more ambitious
>> taht tried to merge the entire IO stack for the async frameworks,
>> rather than just provide a standard way for their event loops to
>> cooperate.
>
> See the final section of my notes here:
> http://www.boredomandlaziness.org/2011/03/python-language-summit-rough-notes.html
>
> Turns out the idea of a PEP 3153 level API *was* raised at the summit,
> but I'd still like to see a competing PEP that targets the reactor
> level API directly.
>
> Cheers,
> Nick.


It's not clear to me what such a PEP should address in particular,
anyway here's a bunch of semi-random ideas.


=== Idea #1 ===

4 classes (SelectPoller, PollPoller, EpollPoller, KqueuePoller) within
concurrent.eventloop namespace all sharing the same API:

- register(fd, events, callback) # callback gets called with events as arg
- modify(fd, events)
- unregister(fd)
- call_later(timeout, callback, errback=None)
- call_every(timeout, callback, errback=None)
- poll(timeout=1.0, blocking=True)
- close()

call_later() and call_every() can return an object having cancel() and
reset() methods.
The user willing to register a new handler will do:

>>> poller.register(sock.fileno(), poller.READ | poller.WRITE, callback)

...then, in the callback:

def callback(events):
if events & poller.ERROR and not events & poller.READ:
disconnect()
else:
if events & poller.READ:
read()
if events & poller.WRITE:
write()


pros: highly customizable
cons: too low level, requires manual handling

=== Idea #2 ===

same as #1 except:

- register(fd, events)
- poll(timeout=1.0) # desn't block, return {fd:events, fd:events, ...}


=== Idea #3 ===

same as #1 except:

- register(fd, events, handler)
- poll(timeout=1.0, blocking=True)

...poll() will call handler.handle_X_event() depending on the current
event (READ, WRITE or ERROR).
An internal map such as {fd:handler, fd:handler} will be maintaned internally.

- pros: easier to use
- cons: more rigid, requires a "contract" with the handler

Terry Reedy

unread,
May 24, 2012, 4:04:40 PM5/24/12
to python...@python.org
On 5/24/2012 2:40 PM, Giampaolo Rodolà wrote:

> It's not clear to me what such a PEP should address in particular,
> anyway here's a bunch of semi-random ideas.

I have been reading for perhaps a decade how bad asyncore is. So I hope
you stick with trying to thrash out something different, even if the
discussion gets tedious or contentions.

> === Idea #1 ===
>
> 4 classes (SelectPoller, PollPoller, EpollPoller, KqueuePoller) within
> concurrent.eventloop namespace all sharing the same API:

For new classes, the first question is what concept (and data/function
grouping) they and their instances represent. As a naive event loop
user, I might think in terms of event sources (or sets of sources) and
corresponding handlers. For events generated by 'file' polling, the
particular method would seem like a secondary issue.

Your proposed classes are named after methods and you give no
initialization api. This suggests to me that you mean for all files
being polled by the same method to be grouped together. If so, there
would only need 0 or 1 instance of each 'class', in while case, they
could just as well be modules.

In other words, I am unsure what concept these classes would represent.
I am perhaps thinking at too high a level.

> - register(fd, events, callback) # callback gets called with events as arg
> - modify(fd, events)
> - unregister(fd)
> - call_later(timeout, callback, errback=None)
> - call_every(timeout, callback, errback=None)
> - poll(timeout=1.0, blocking=True)
> - close()
>
> call_later() and call_every() can return an object having cancel() and
> reset() methods.

--
Terry Jan Reedy

Cameron Simpson

unread,
May 24, 2012, 6:37:52 PM5/24/12
to Antoine Pitrou, python...@python.org
On 24May2012 14:03, Antoine Pitrou <soli...@pitrou.net> wrote:
| Also, I don't know why you would specify poller.READ or poller.WRITE
| explicitly. Usually you are interested in all events, no?

Personally, I would want specificity. If I only care about write (eg I'm
only sending), I would only specify poller.WRITE and have my handler
only know and care about that. Possibly it would be good to be able to
raise an exception for events I hadn't handled, but I'd be half inclined
to have my handler do that, were it wanted (yes, there is some tension
in this sentence).

Unless I'm missing something here.

Just my 2c,
--
Cameron Simpson <c...@zip.com.au> DoD#743
http://www.cskk.ezoshosting.com/cs/

I just didn't give up, not riding it out wasn't an option. You don't
crash, until you do. The longer you ride it out the more likely you
are to ride it out. Throwing it away, saves nothing. - J. Pridmore

Ronald Oussoren

unread,
May 25, 2012, 2:39:23 AM5/25/12
to Giampaolo Rodolà, python...@python.org

On 24 May, 2012, at 20:40, Giampaolo Rodolà wrote:

> 2012/5/24 Nick Coghlan <ncog...@gmail.com>:
>> On Thu, May 24, 2012 at 10:37 PM, Nick Coghlan <ncog...@gmail.com> wrote:
>>> On Thu, May 24, 2012 at 9:50 PM, Giampaolo Rodolà <g.ro...@gmail.com> wrote:
>>>> poller.poll serves the same purpose of asyncore.loop, yes, but this is
>>>> supposed to be independent from asyncore.
>>>
>>> I'd actually like to see something like this pitched as a
>>> "concurrent.eventloop" PEP. PEP 3153 really wasn't what I was
>>> expecting after the discussions at the PyCon US 2011 language summit -
>>> I was expecting "here's a common event loop all the async frameworks
>>> can hook into", but instead we got something a *lot* more ambitious
>>> taht tried to merge the entire IO stack for the async frameworks,
>>> rather than just provide a standard way for their event loops to
>>> cooperate.
>>
>> See the final section of my notes here:
>> http://www.boredomandlaziness.org/2011/03/python-language-summit-rough-notes.html
>>
>> Turns out the idea of a PEP 3153 level API *was* raised at the summit,
>> but I'd still like to see a competing PEP that targets the reactor
>> level API directly.
>>
>> Cheers,
>> Nick.
>
>
> It's not clear to me what such a PEP should address in particular,
> anyway here's a bunch of semi-random ideas.


All of these are probably too low level to be the only API because they don't encapsulate error handling.

A slightly higher level API would have a callback with received data and a buffered API for sending data. That way the networking library can deal with lowlevel socket API errors and translate them to usefull abtract errors. It would also handle some errors like and EGAIN error itself.

Also: how would you use SSL with these APIs?

The API would probably end up with functionality simular to Twisted's reactor and transport APIs (and possibly endpoints but I don't know how stable that API is).

Ronald

Calvin Spealman

unread,
May 25, 2012, 5:32:23 AM5/25/12
to Giampaolo Rodolà, python...@python.org

Frankly, I don't think this deserves a PEP at all, or even to consider
one *yet*.

Building a new API and a new library from scratch seems a frail
comparison to testing
a library in the real world, it having real uses, and then being
incorporated into the
stdlib. The problem here, of course, is that all the real-world
solutions (ie, Twisted)
include far more than the reactor.

--
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://techblog.ironfroggy.com/
Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy

Devin Jeanpierre

unread,
May 25, 2012, 6:33:22 AM5/25/12
to ironf...@gmail.com, python...@python.org
On Fri, May 25, 2012 at 5:32 AM, Calvin Spealman <ironf...@gmail.com> wrote:
> Frankly, I don't think this deserves a PEP at all, or even to consider
> one *yet*.
>
> Building a new API and a new library from scratch seems a frail
> comparison to testing
> a library in the real world, it having real uses, and then being
> incorporated into the
> stdlib. The problem here, of course, is that all the real-world
> solutions (ie, Twisted)
> include far more than the reactor.

To be fair, PEP-3153 was built based largely on experience from the
Twisted project and input from Twisted developers, who know what they
are talking about and how to build a useful system. The entire
transport/protocol separation is lifted directly out of it.

-- Devin

Calvin Spealman

unread,
May 25, 2012, 7:54:13 AM5/25/12
to Devin Jeanpierre, python...@python.org
On Fri, May 25, 2012 at 6:33 AM, Devin Jeanpierre
<jeanpi...@gmail.com> wrote:
> On Fri, May 25, 2012 at 5:32 AM, Calvin Spealman <ironf...@gmail.com> wrote:
>> Frankly, I don't think this deserves a PEP at all, or even to consider
>> one *yet*.
>>
>> Building a new API and a new library from scratch seems a frail
>> comparison to testing
>> a library in the real world, it having real uses, and then being
>> incorporated into the
>> stdlib. The problem here, of course, is that all the real-world
>> solutions (ie, Twisted)
>> include far more than the reactor.
>
> To be fair, PEP-3153 was built based largely on experience from the
> Twisted project and input from Twisted developers, who know what they
> are talking about and how to build a useful system. The entire
> transport/protocol separation is lifted directly out of it.
>
> -- Devin

My comments were in response to this post, not PEP-3153

--
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://techblog.ironfroggy.com/
Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy

Nick Coghlan

unread,
May 25, 2012, 9:53:04 AM5/25/12
to ironf...@gmail.com, python...@python.org
On May 25, 2012 7:33 PM, "Calvin Spealman" <ironf...@gmail.com> wrote:
On Wed, May 23, 2012 at 9:32 PM, Giampaolo Rodolà <g.ro...@gmail.com> wrote:
> Users willing to support multiple event loops such as wx, gtk etc can do:
>
>>>> while 1:
> ...       poller.poll(timeout=0.1, blocking=False)
> ...       otherpoller.poll()
>
>
> Basically, this would be the whole API.
>
> Thoughts?
>

Frankly, I don't think this deserves a PEP at all, or even to consider
one *yet*.

Building a new API and a new library from scratch seems a frail
comparison to testing
a library in the real world, it having real uses, and then being
incorporated into the
stdlib. The problem here, of course, is that all the real-world
solutions (ie, Twisted)
include far more than the reactor.

No, the specific call at the PyCon US 2011 language summit was for a PEP that proposed a *new* event loop for the standard library that:
1. Provides simple event loop functionality in the standard library, as an improved alternative to asyncore for small apps that don't require the full power of a framework like Twisted (think things like little IRC bots, TCP echo servers, or testing of async components)
2. Provides a clean migration path to a production grade reactor like Twisted's
3. Makes it easier for multiple event loop based frameworks (e.g. tkinter, wxPython, PySide, Twisted) to all cooperate within the same process

What we're after is something for the stdlib that is to event loops/reactors as wsgiref is to production grade WSGI servers like mod_wsgi and nginx. asyncore isn't it, because the migration path isn't clean.

PEP 3153 currently spends a lot of time talking about transports and protocols, but doesn't answer those 3 core questions:

1. How do I write a simple IRC bot or TCP echo server?
2. How do I migrate my simple app to a production grade reactor like Twisted's?
3. How do I run two different concurrent.eventloop compatible reactors in the same process?

As far as I can tell, PEP 3153 wants to handle all that by merging the I/O stacks of all the frameworks first, which strikes me as being *way* too ambitious for a first step. If we can't even figure out a common abstraction for the reactor level (ala WSGI), how are we ever going to agree on a standard async I/O abstraction?

Cheers,
Nick.

Nick Coghlan

unread,
May 25, 2012, 9:54:51 AM5/25/12
to ironf...@gmail.com, python...@python.org
On Fri, May 25, 2012 at 11:53 PM, Nick Coghlan <ncog...@gmail.com> wrote:
> as wsgiref is to production grade WSGI servers like mod_wsgi and nginx.

s/nginx/gunicorn/

Confusing-my-software-stack-levels'ly,
Nick.

--
Nick Coghlan   |   ncog...@gmail.com   |   Brisbane, Australia

Calvin Spealman

unread,
May 26, 2012, 5:05:57 PM5/26/12
to Nick Coghlan, python...@python.org

Obviously, for a man with many opinions I miss out on too many conversations
and too many potential actions. I should make steps to correct this in
the future.
Thanks for clearing this up.

> Cheers,
> Nick.

Reply all
Reply to author
Forward
0 new messages