A while back, I volunteered to update asyncore and asynchat for py3k. I posted a patch, and in response to feedback posted a more complicated patch+modification.
It's a big patch, but I'll try applying it to the current py3k branch -- does it apply? -- and try a few things with it. I'm concerned about how well it behaves with things like Medusa (which probably needs its own py3k update).
They applied when posted them, but subsequent patches seem to have broken them. I've now posted updated patches that apply cleanly against revision 60780.
On Feb 13, 2008 6:52 PM, Bill Janssen <jans...@parc.com> wrote:
> It's a big patch, but I'll try applying it to the current py3k branch > -- does it apply? -- and try a few things with it. I'm concerned > about how well it behaves with things like Medusa (which probably > needs its own py3k update).
asyncore and asynchat are in a difficult position right now since a lot of patches for both modules are pending and no decisions are taken. In detail I'm talking about patches 1519, 1541, 2073 and 1736190 which is the most important one since it includes a lot of fixes for other reported issues (e.g. 1740572, 1736101, 909005). As I said in 1563, IMHO your patch could be reviewed and eventually committed only after 1736190 and others are committed and 2.x related problems are solved. Converting the code to 3.0 should be the last step also because my perception is that your patch changes too much of the existing code: a new iterator_producer, a different collect_incoming_data() behavior, name mangling of internal variables and others. ...A lot of stuff which could be included in a second time.
I'm going to inform Josiah Carlson about this topic since he should be the most experienced one about asyncore and asynchat.
On 14 Feb, 03:52, Bill Janssen <jans...@parc.com> wrote:
> It's a big patch, but I'll try applying it to the current py3k branch > -- does it apply? -- and try a few things with it. I'm concerned > about how well it behaves with things like Medusa (which probably > needs its own py3k update).
> asyncore and asynchat are in a difficult position right now since a > lot of patches for both modules are pending and no decisions are > taken. > In detail I'm talking about patches 1519, 1541, 2073 and 1736190 which > is the most important one since it includes a lot of fixes for other > reported issues (e.g. 1740572, 1736101, 909005).
I took a look to some of these in other opportunity, and IIRC, the main issue here is exactly what you say: lots of issues with problem *and* fixes, but some decisions needs to be taken.
No decision taken, the pile of issues (and real problems behind them) accumulate through time. I think it's a good idea to bring this discussion here, and with luck a result will come up regarding them.
So, it would be great if you (please) summarize here the problems behind those issues and the decisions that must be taken... Thanks!
Ok, I'll try to take a look at all asyncore/chat reports and try to summarize them by splitting patches which solve bugs and patches which add enhancements or functionalities.
On 14 Feb, 16:12, "Facundo Batista" <facundobati...@gmail.com> wrote:
> > asyncore and asynchat are in a difficult position right now since a > > lot of patches for both modules are pending and no decisions are > > taken. > > In detail I'm talking about patches 1519, 1541, 2073 and 1736190 which > > is the most important one since it includes a lot of fixes for other > > reported issues (e.g. 1740572, 1736101, 909005).
> I took a look to some of these in other opportunity, and IIRC, the > main issue here is exactly what you say: lots of issues with problem > *and* fixes, but some decisions needs to be taken.
> No decision taken, the pile of issues (and real problems behind them) > accumulate through time. I think it's a good idea to bring this > discussion here, and with luck a result will come up regarding them.
> So, it would be great if you (please) summarize here the problems > behind those issues and the decisions that must be taken... Thanks!
First of all, you're conflating my two possible patches.
One patch just addresses the problem of strings and bytes, as GvR asked me to do, and adds an 8-line class called iterator_producer that adapts iterators into producers. The patch doesn't change how the module works at all, and iterator_producer is not invoked anywhere within the code; it's purely for user convenience. I consider this the primary patch and would like to focus attention there if possible.
The other patch combines the string and bytes fix with a porting of 1736190 and the other things you complain about, most of which scratch personal itches. If the patches you mention are actually going to be applied, then this patch isn't the way to go, and I'll maybe submit parts of it as separate patches. If they're just going to waste away in the bug tracker, though, this patch should be seriously considered.
I'm quite willing to re-construct my string and bytes patch against a version of py3k in which the pre-existing patches are already applied. There needs to be forward progress, though: If nothing at all gets done, asyncore is going to be removed from the standard lib. I don't want to see that happen.
On Thu, Feb 14, 2008 at 5:55 AM, Giampaolo Rodola' <gne...@gmail.com> wrote: > (wrong quoting: obvioulsly I was talking to Daniel)
> Ok, I'll try to take a look at all asyncore/chat reports and try to > summarize them by splitting patches which solve bugs and patches which > add enhancements or functionalities.
> On 14 Feb, 16:12, "Facundo Batista" <facundobati...@gmail.com> wrote:
> > > asyncore and asynchat are in a difficult position right now since a > > > lot of patches for both modules are pending and no decisions are > > > taken. > > > In detail I'm talking about patches 1519, 1541, 2073 and 1736190 which > > > is the most important one since it includes a lot of fixes for other > > > reported issues (e.g. 1740572, 1736101, 909005).
> > I took a look to some of these in other opportunity, and IIRC, the > > main issue here is exactly what you say: lots of issues with problem > > *and* fixes, but some decisions needs to be taken.
> > No decision taken, the pile of issues (and real problems behind them) > > accumulate through time. I think it's a good idea to bring this > > discussion here, and with luck a result will come up regarding them.
> > So, it would be great if you (please) summarize here the problems > > behind those issues and the decisions that must be taken... Thanks!
- 1736190 which includes fixes for the following issues among other improvements: - 1063924 (asyncore should handle ECONNRESET in send()) - 1736101 (asyncore should handle ECONNABORTED in recv()) - 760475 (handle_error() should call handle_close() instead of close()) - 1740572 (refill_buffer() should call handle_close() rather than close()) - 777588 (wrong "connection refused" behavior on Windows) - 889153 (wrong connect() behavior) - 953599 (asyncore misses socket closes when poll is used) - 1025525 (asyncore.file_dispatcher should not take fd as argument)
- 1519 (async_chat.__init__() and asyncore.dispatcher.__init__ parameters inconsistency) - 1541 (Bad OOB data management when using asyncore with select.poll()) - 2073 (asynchat push always sends 512 bytes (ignoring ac_out_buffer_size))
=== Open issues with no patches (need review) ===
- 658749 (asyncore connect() and winsock errors) - 1161031 (neverending warnings from asyncore)
=== Enhancements & new features ===
- 1641 (add delayed calls feature) - 1563 (conversion to py3k and some other changes)
IMHO the first thing to do should be modifying 1736190 patch to fix the minor issues came out in comments 52767 (re-add simple_producer and fifo classes) and 57581 (look at what is specified in ac_out_buffer_size) and then commit it. I've used that same modified asynchat version in my pyftpdlib project and I can tell that it works pretty good. I guess that Josiah Carlson could do that pretty quickly if he has time to do so.
Independently from all a nice thing to do would be adding tests for many asyncore/chat behaviors which currently aren't covered by the test suite. It could be very useful to know if behaviors between different commits remain the same, hence it would be better working on that (the test suite) before committing 1736190.
Sorry I haven't been available for this recently, I've been working far too much (10-14 hours/day, 6 days/week, since November) to really do any "fun" programming. Also, sorry for top-posting.
As I stated 2+ and 6+ months ago, the patches I submitted 9+ months ago work (I've been using them in my own projects without issue). The only issue with the bugfix/rearrangement that I last heard about with regards to the 2.x stuff was that I removed a class that no one was using in the wild, which I believe Giampaolo added in a subsequent patch in the last couple months.
My suggestion: 1. Apply the most recent fixes to 2.6 asyncore/asynchat that Giampaolo has updated. 1.a. Figure out what the hell is up with OOB data, how to handle it, and stop making it use handle_expt(). 2. Use the 2.x -> 3.x tool to convert the fixes over. 3. Apply any 3.x specific fixes (for string/bytes, etc.) to the 3.x branch as necessary (make them global constants in both 2.x and 3.x so that they are easy to track). 4. Consider enhancements to 2.6 if they aren't to big, consider slightly larger enhancements to 3.x. *
- Josiah
* Scheduled tasks are not a valid enhancement for either; anyone who wants them can trivially add them with their own asyncore.loop() variant and call asyncore.poll() as necessary (I did it in about 15 lines in the summer of 2002, I hope others can do better now). If you want my opinion on other async-related features, feel free to email me directly (use the gmail address you see here, then it ends up in my inbox, not the overflowing python folder).
On Thu, Feb 14, 2008 at 9:09 AM, Giampaolo Rodola' <gne...@gmail.com> wrote: > On 14 Feb, 16:36, "Giampaolo Rodola'" <gne...@gmail.com> wrote: > > Ok, I'll try to take a look at all asyncore/chat reports and try to > > summarize them by splitting patches which solve bugs and patches which > > add enhancements or functionalities.
> > On 14 Feb, 16:12, "Facundo Batista" <facundobati...@gmail.com> wrote:
> > > > asyncore and asynchat are in a difficult position right now since a > > > > lot of patches for both modules are pending and no decisions are > > > > taken. > > > > In detail I'm talking about patches 1519, 1541, 2073 and 1736190 which > > > > is the most important one since it includes a lot of fixes for other > > > > reported issues (e.g. 1740572, 1736101, 909005).
> > > I took a look to some of these in other opportunity, and IIRC, the > > > main issue here is exactly what you say: lots of issues with problem > > > *and* fixes, but some decisions needs to be taken.
> > > No decision taken, the pile of issues (and real problems behind them) > > > accumulate through time. I think it's a good idea to bring this > > > discussion here, and with luck a result will come up regarding them.
> > > So, it would be great if you (please) summarize here the problems > > > behind those issues and the decisions that must be taken... Thanks!
> - 1736190 which includes fixes for the following issues among other > improvements: > - 1063924 (asyncore should handle ECONNRESET in send()) > - 1736101 (asyncore should handle ECONNABORTED in recv()) > - 760475 (handle_error() should call handle_close() instead of > close()) > - 1740572 (refill_buffer() should call handle_close() rather than > close()) > - 777588 (wrong "connection refused" behavior on Windows) > - 889153 (wrong connect() behavior) > - 953599 (asyncore misses socket closes when poll is used) > - 1025525 (asyncore.file_dispatcher should not take fd as argument)
> - 1519 (async_chat.__init__() and asyncore.dispatcher.__init__ > parameters inconsistency) > - 1541 (Bad OOB data management when using asyncore with > select.poll()) > - 2073 (asynchat push always sends 512 bytes (ignoring > ac_out_buffer_size))
> === Open issues with no patches (need review) ===
> - 658749 (asyncore connect() and winsock errors) > - 1161031 (neverending warnings from asyncore)
> === Enhancements & new features ===
> - 1641 (add delayed calls feature) > - 1563 (conversion to py3k and some other changes)
> IMHO the first thing to do should be modifying 1736190 patch to fix > the minor issues came out in comments 52767 (re-add simple_producer > and fifo classes) and 57581 (look at what is specified in > ac_out_buffer_size) and then commit it. > I've used that same modified asynchat version in my pyftpdlib project > and I can tell that it works pretty good. > I guess that Josiah Carlson could do that pretty quickly if he has > time to do so.
> Independently from all a nice thing to do would be adding tests for > many asyncore/chat behaviors which currently aren't covered by the > test suite. > It could be very useful to know if behaviors between different commits > remain the same, hence it would be better working on that (the test > suite) before committing 1736190.
On Thu, Feb 14, 2008 at 06:24:04PM -0800, Josiah Carlson wrote: > 1.a. Figure out what the hell is up with OOB data, how to handle it, > and stop making it use handle_expt().
Does OOB data actually need to be supported? For a long time TCP implementations usually had buggy OOB implementations because it was so rarely used; for all I know, that's still the case. Maybe the feature should just be dropped.
On Fri, Feb 15, 2008 at 09:24:14AM -0500, A.M. Kuchling wrote: > On Thu, Feb 14, 2008 at 06:24:04PM -0800, Josiah Carlson wrote: > > 1.a. Figure out what the hell is up with OOB data, how to handle it, > > and stop making it use handle_expt().
> Does OOB data actually need to be supported? For a long time TCP > implementations usually had buggy OOB implementations because it was > so rarely used; for all I know, that's still the case. Maybe the > feature should just be dropped.
On 15 Feb, 03:24, "Josiah Carlson" <josiah.carl...@gmail.com> wrote:
> As I stated 2+ and 6+ months ago, the patches I submitted 9+ months > ago work (I've been using them in my own projects without issue). The > only issue with the bugfix/rearrangement that I last heard about with > regards to the 2.x stuff was that I removed a class that no one was > using in the wild, which I believe Giampaolo added in a subsequent > patch in the last couple months.
I provided the patch for the other issue (look at what is specified in ac_out_buffer_size) but I didn't re-add fifo and simple_producer classes. What should be done here? Re-add or discard them? However, I will send to you by e-mail the modified asynchat version. It is based on your patch therefore a first commit could be finally done.
> My suggestion: > 1. Apply the most recent fixes to 2.6 asyncore/asynchat that Giampaolo > has updated. > 1.a. Figure out what the hell is up with OOB data, how to handle it, > and stop making it use handle_expt().
If we want to leave OOB untouched shouldn't handle_expt be the right place where manage such kind of events? Alternatively we could also remove the OOB management at all (e.g. Twisted does that by ignoring such kind of data). OOB could be actually useful to implement some protocols such as FTP (in details ABOR and STAT commands which require OOB data) but it would be probably better not having it than having it but not implemented properly. I am saying this also because later or soonish we might need to care of epoll and kqueue (http://bugs.python.org/issue1657).
> * Scheduled tasks are not a valid enhancement for either; anyone who > wants them can trivially add them with their own asyncore.loop() > variant and call asyncore.poll() as necessary (I did it in about 15 > lines in the summer of 2002, I hope others can do better now). If you > want my opinion on other async-related features, feel free to email me > directly (use the gmail address you see here, then it ends up in my > inbox, not the overflowing python folder).
How's your solution? Could you post it here or send it to me by mail? IMO scheduled tasks is a very important feature for implementing a lot of interesting stuff like connection timeouts and bandwidth throttling. A lot of people have to learn/use Twisted just because of that. Moreover I think that Bill Janssen could need that feature to make the ssl module work with asyncore.
On Fri, Feb 15, 2008 at 11:45 AM, Giampaolo Rodola' <gne...@gmail.com> wrote: > On 15 Feb, 03:24, "Josiah Carlson" <josiah.carl...@gmail.com> wrote:
> > As I stated 2+ and 6+ months ago, the patches I submitted 9+ months > > ago work (I've been using them in my own projects without issue). The > > only issue with the bugfix/rearrangement that I last heard about with > > regards to the 2.x stuff was that I removed a class that no one was > > using in the wild, which I believe Giampaolo added in a subsequent > > patch in the last couple months.
> I provided the patch for the other issue (look at what is specified in > ac_out_buffer_size) but I didn't re-add fifo and simple_producer > classes. > What should be done here? Re-add or discard them? > However, I will send to you by e-mail the modified asynchat version. > It is based on your patch therefore a first commit could be finally > done.
> > My suggestion: > > 1. Apply the most recent fixes to 2.6 asyncore/asynchat that Giampaolo > > has updated.
> > 1.a. Figure out what the hell is up with OOB data, how to handle it, > > and stop making it use handle_expt().
> If we want to leave OOB untouched shouldn't handle_expt be the right > place where manage such kind of events? > Alternatively we could also remove the OOB management at all (e.g. > Twisted does that by ignoring such kind of data). > OOB could be actually useful to implement some protocols such as FTP > (in details ABOR and STAT commands which require OOB data) but it > would be probably better not having it than having it but not > implemented properly. > I am saying this also because later or soonish we might need to care > of epoll and kqueue (http://bugs.python.org/issue1657).
> > * Scheduled tasks are not a valid enhancement for either; anyone who > > wants them can trivially add them with their own asyncore.loop() > > variant and call asyncore.poll() as necessary (I did it in about 15 > > lines in the summer of 2002, I hope others can do better now). If you > > want my opinion on other async-related features, feel free to email me > > directly (use the gmail address you see here, then it ends up in my > > inbox, not the overflowing python folder).
> How's your solution? Could you post it here or send it to me by mail? > IMO scheduled tasks is a very important feature for implementing a lot > of interesting stuff like connection timeouts and bandwidth > throttling. > A lot of people have to learn/use Twisted just because of that. > Moreover I think that Bill Janssen could need that feature to make the > ssl module work with asyncore.
> PS - I have been reading this mailing list for a short time therefore > I have no clue whether or not someone has ever thought about including > the Twisted core - only itself - in Python stdlib.
I think we should just replace the current "loop" with this (and add the "schedule" function). Then other folks won't have to figure out how the module works and write their own loop. I'll be happy to update the documentation to document it.
> On Fri, Feb 15, 2008 at 11:45 AM, Giampaolo Rodola' <gne...@gmail.com> wrote: > > On 15 Feb, 03:24, "Josiah Carlson" <josiah.carl...@gmail.com> wrote:
> > > As I stated 2+ and 6+ months ago, the patches I submitted 9+ months > > > ago work (I've been using them in my own projects without issue). The > > > only issue with the bugfix/rearrangement that I last heard about with > > > regards to the 2.x stuff was that I removed a class that no one was > > > using in the wild, which I believe Giampaolo added in a subsequent > > > patch in the last couple months.
> > I provided the patch for the other issue (look at what is specified in > > ac_out_buffer_size) but I didn't re-add fifo and simple_producer > > classes. > > What should be done here? Re-add or discard them? > > However, I will send to you by e-mail the modified asynchat version. > > It is based on your patch therefore a first commit could be finally > > done.
> > > My suggestion: > > > 1. Apply the most recent fixes to 2.6 asyncore/asynchat that Giampaolo > > > has updated.
> > > 1.a. Figure out what the hell is up with OOB data, how to handle it, > > > and stop making it use handle_expt().
> > If we want to leave OOB untouched shouldn't handle_expt be the right > > place where manage such kind of events? > > Alternatively we could also remove the OOB management at all (e.g. > > Twisted does that by ignoring such kind of data). > > OOB could be actually useful to implement some protocols such as FTP > > (in details ABOR and STAT commands which require OOB data) but it > > would be probably better not having it than having it but not > > implemented properly. > > I am saying this also because later or soonish we might need to care > > of epoll and kqueue (http://bugs.python.org/issue1657).
> > > * Scheduled tasks are not a valid enhancement for either; anyone who > > > wants them can trivially add them with their own asyncore.loop() > > > variant and call asyncore.poll() as necessary (I did it in about 15 > > > lines in the summer of 2002, I hope others can do better now). If you > > > want my opinion on other async-related features, feel free to email me > > > directly (use the gmail address you see here, then it ends up in my > > > inbox, not the overflowing python folder).
> > How's your solution? Could you post it here or send it to me by mail? > > IMO scheduled tasks is a very important feature for implementing a lot > > of interesting stuff like connection timeouts and bandwidth > > throttling. > > A lot of people have to learn/use Twisted just because of that. > > Moreover I think that Bill Janssen could need that feature to make the > > ssl module work with asyncore.
> > PS - I have been reading this mailing list for a short time therefore > > I have no clue whether or not someone has ever thought about including > > the Twisted core - only itself - in Python stdlib.
On 15 Feb, 21:36, Bill Janssen <jans...@parc.com> wrote:
> I think we should just replace the current "loop" with this (and add > the "schedule" function). Then other folks won't have to figure out > how the module works and write their own loop. I'll be happy to update > the documentation to document it.
Bill Janssen wrote: > I think we should just replace the current "loop" with this (and add > the "schedule" function). Then other folks won't have to figure out > how the module works and write their own loop.
Having beaten my way down this road of broken glass, I would like args and kwargs if you are adding this:
I've discussed a lot with Josiah via e-mail and I provided an updated version of the patch proposed in bug #1736190 including a fix for the two issues raised by me in the bug report. The update has been needed also because the original patch has been out-dated by some commits after r53734 involving the test suite and the documentation, which I both took off this patch. I also re-added simple_producer and fifo classes in asynchat.py since it seems they are needed for passing tests.
As far as I can tell, the asyncore.py, asynchat.py, and updated test_asyncore.py are good. I have been using earlier variants in my own projects (prior to their updating to pass the test suite) for quite a few months now. The updated modules provide better performance, features, and support for real-world async socket servers, never mind fixing more than a half dozen outstanding bugs.
- Josiah
On Sun, Mar 2, 2008 at 3:19 PM, Giampaolo Rodola' <gne...@gmail.com> wrote: > I've discussed a lot with Josiah via e-mail and I provided an updated > version of the patch proposed in bug #1736190 including a fix for the > two issues raised by me in the bug report. > The update has been needed also because the original patch has been > out-dated by some commits after r53734 involving the test suite > and the documentation, which I both took off this patch. > I also re-added simple_producer and fifo classes in asynchat.py since > it seems they are needed for passing tests.
> The test suite has passed on Windows XP using Python 2.6 alpha 1. > I have also successfully run the test suite of a project of mine based > on asynchat which includes over 40 tests.
On Fri, Feb 15, 2008 at 9:11 PM, Josiah Carlson <josiah.carl...@gmail.com> wrote:
> Twisted core has been proposed, but I believe the consensus was that > it wasn't desirable, generally.
I remember only a couple of dissenting voices, and only a small number of participants. Of the dissenting voices, I do not recall any actual arguments about undesireability, just misunderstandings of how Twisted actually works. Getting Twisted core (meaning Deferreds, a simple reactor and the Protocol class) into the core is still on my TODO list.
I'm also pretty sure that people learn twisted because everyone learns
> twisted. It's one of those buzz-words ;).
I think that's quite an unfair assessment, even in jest :) Twisted is well worth learning to actually use it, as it's a very versatile event loop and does it best to integrate nicely with other event systems. And including it in the standard library improves integration with other event loops by creating a single interface. It's not a matter of dropping it in, though; it requires some careful structuring to avoid embarrassing situations like we have with the xml package, but still people to provide their own reactor.
In case you're wondering how the twisted reactor in the stdlib is useful to people not using Twisted, take a look at what you currently need to do to combine stdlib modules like urllib and ftplib with event systems like Tkinter and PyGTK. Not to mention that the Twisted implementations of various protocols are really quite, quite good -- in many cases quite a lot better than the stdlib ones. But including those takes yet more time.
On Mon, Mar 24, 2008 at 3:04 PM, Thomas Wouters <tho...@python.org> wrote:
> On Fri, Feb 15, 2008 at 9:11 PM, Josiah Carlson <josiah.carl...@gmail.com> > wrote: > > Twisted core has been proposed, but I believe the consensus was that > > it wasn't desirable, generally.
> I remember only a couple of dissenting voices, and only a small number of > participants. Of the dissenting voices, I do not recall any actual arguments > about undesireability, just misunderstandings of how Twisted actually works. > Getting Twisted core (meaning Deferreds, a simple reactor and the Protocol > class) into the core is still on my TODO list.
> > I'm also pretty sure that people learn twisted because everyone learns > > twisted. It's one of those buzz-words ;).
> I think that's quite an unfair assessment, even in jest :) Twisted is well > worth learning to actually use it, as it's a very versatile event loop and > does it best to integrate nicely with other event systems. And including it > in the standard library improves integration with other event loops by > creating a single interface. It's not a matter of dropping it in, though; it > requires some careful structuring to avoid embarrassing situations like we > have with the xml package, but still people to provide their own reactor.
> In case you're wondering how the twisted reactor in the stdlib is useful to > people not using Twisted, take a look at what you currently need to do to > combine stdlib modules like urllib and ftplib with event systems like > Tkinter and PyGTK. Not to mention that the Twisted implementations of > various protocols are really quite, quite good -- in many cases quite a lot > better than the stdlib ones. But including those takes yet more time.
In that sense it'd be competing with safethread for inclusion in Python. Whereas safethread requires little if any API changes, twisted requires an entirely new API that can be event-driven. Worse, we'd likely to be stuck maintaining both APIs for a long time, if not forever.
Twisted may be one of the best (if not *the* best) ways of writing concurrent programs today, but it doesn't need to be in the stdlib for that. If safethread is going to solve many of the same problems, with less changes required by the users of the language, then this is the wrong time to add twisted.
On Mon, Mar 24, 2008 at 10:21 PM, Adam Olsen <rha...@gmail.com> wrote: > On Mon, Mar 24, 2008 at 3:04 PM, Thomas Wouters <tho...@python.org> wrote:
> > On Fri, Feb 15, 2008 at 9:11 PM, Josiah Carlson < > josiah.carl...@gmail.com> > > wrote: > > > Twisted core has been proposed, but I believe the consensus was that > > > it wasn't desirable, generally.
> > I remember only a couple of dissenting voices, and only a small number > of > > participants. Of the dissenting voices, I do not recall any actual > arguments > > about undesireability, just misunderstandings of how Twisted actually > works. > > Getting Twisted core (meaning Deferreds, a simple reactor and the > Protocol > > class) into the core is still on my TODO list.
> > > I'm also pretty sure that people learn twisted because everyone learns > > > twisted. It's one of those buzz-words ;).
> > I think that's quite an unfair assessment, even in jest :) Twisted is > well > > worth learning to actually use it, as it's a very versatile event loop > and > > does it best to integrate nicely with other event systems. And including > it > > in the standard library improves integration with other event loops by > > creating a single interface. It's not a matter of dropping it in, > though; it > > requires some careful structuring to avoid embarrassing situations like > we > > have with the xml package, but still people to provide their own > reactor.
> > In case you're wondering how the twisted reactor in the stdlib is useful > to > > people not using Twisted, take a look at what you currently need to do > to > > combine stdlib modules like urllib and ftplib with event systems like > > Tkinter and PyGTK. Not to mention that the Twisted implementations of > > various protocols are really quite, quite good -- in many cases quite a > lot > > better than the stdlib ones. But including those takes yet more time.
> In that sense it'd be competing with safethread for inclusion in > Python. Whereas safethread requires little if any API changes, > twisted requires an entirely new API that can be event-driven. Worse, > we'd likely to be stuck maintaining both APIs for a long time, if not > forever.
I am not going to worry about this for the time being. Even if safethread goes in and becomes ubiquitous, Twisted is still a very valid approach to the problem. And I'm not just saying that because I don't subscribe to your enthusiasm of safethreads as a concept or as an implementation :)
> Twisted may be one of the best (if not *the* best) ways of writing > concurrent programs today, but it doesn't need to be in the stdlib for > that. If safethread is going to solve many of the same problems, with > less changes required by the users of the language, then this is the > wrong time to add twisted.
You must have missed the part where we already have a large set of event loops, and not having a single interface to them is in fact hurting people. Twisted goes out of its way to interact nicely with event loops, but it can only do that with ones it knows about (and are versatile enough to hook into.) Having a single event system in the standard library is definitely advantageous, even if safethreads were available everywhere and the performance in the common case was satisfactory. It used to be the case that people thought asyncore was this standard event system, but it's long since ceased to be.
On Mon, Mar 24, 2008 at 10:04:20PM +0100, Thomas Wouters wrote: > I remember only a couple of dissenting voices, and only a small number of > participants. Of the dissenting voices, I do not recall any actual arguments
Weren't some of those dissenting voices the Twisted developers, though?
On Mon, Mar 24, 2008 at 3:37 PM, Thomas Wouters <tho...@python.org> wrote:
> On Mon, Mar 24, 2008 at 10:21 PM, Adam Olsen <rha...@gmail.com> wrote:
> > Twisted may be one of the best (if not *the* best) ways of writing > > concurrent programs today, but it doesn't need to be in the stdlib for > > that. If safethread is going to solve many of the same problems, with > > less changes required by the users of the language, then this is the > > wrong time to add twisted.
> You must have missed the part where we already have a large set of event > loops, and not having a single interface to them is in fact hurting people. > Twisted goes out of its way to interact nicely with event loops, but it can > only do that with ones it knows about (and are versatile enough to hook > into.) Having a single event system in the standard library is definitely > advantageous, even if safethreads were available everywhere and the > performance in the common case was satisfactory. It used to be the case that > people thought asyncore was this standard event system, but it's long since > ceased to be.
I'm not opposed to standardizing on twisted as the canonical way to do events in python, or to having an event system in python. My concern is that may be used as an excuse to slowly rewrite the entire language into an event-driven model.
However, that was based on the assumption that modules like urllib2 weren't already event-driven. Looking now, it seems it already is, and wraps it in a blocking API for simple use cases.
So long as we're only talking about replacing existing event-driven stuff, and so long as we retain the existing blocking API (including calling from threads!), I don't think I have any valid opposition.
|
