[Python-Dev] Py3k and asyncore/asynchat

82 views
Skip to first unread message

Daniel Arbuckle

unread,
Feb 13, 2008, 3:28:57 PM2/13/08
to pytho...@python.org
A while back, I volunteered to update asyncore and asynchat for py3k.
I posted a patch, and in response to feedback posted a more
complicated patch+modification.

Both versions have been languishing at
http://bugs.python.org/issue1563 for a couple of months now without
any further feedback or action. I need some more experienced member of
the community to take some sort of action: reviews and suggestions are
welcome, as is advice about which version I should continue on with.
Committing the patch would also be acceptable :)
_______________________________________________
Python-Dev mailing list
Pytho...@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-dev2-garchive-22421%40googlegroups.com

Bill Janssen

unread,
Feb 13, 2008, 9:52:48 PM2/13/08
to Daniel Arbuckle, pytho...@python.org
It's a big patch, but I'll try applying it to the current py3k branch
-- does it apply? -- and try a few things with it. I'm concerned
about how well it behaves with things like Medusa (which probably
needs its own py3k update).

Bill

Daniel Arbuckle

unread,
Feb 14, 2008, 1:01:09 AM2/14/08
to Bill Janssen, pytho...@python.org
They applied when posted them, but subsequent patches seem to have
broken them. I've now posted updated patches that apply cleanly
against revision 60780.

On Feb 13, 2008 6:52 PM, Bill Janssen <jan...@parc.com> wrote:
> It's a big patch, but I'll try applying it to the current py3k branch
> -- does it apply? -- and try a few things with it. I'm concerned
> about how well it behaves with things like Medusa (which probably
> needs its own py3k update).
>
> Bill
>

Giampaolo Rodola'

unread,
Feb 14, 2008, 8:42:46 AM2/14/08
to pytho...@python.org
asyncore and asynchat are in a difficult position right now since a
lot of patches for both modules are pending and no decisions are
taken.
In detail I'm talking about patches 1519, 1541, 2073 and 1736190 which
is the most important one since it includes a lot of fixes for other
reported issues (e.g. 1740572, 1736101, 909005).
As I said in 1563, IMHO your patch could be reviewed and eventually
committed only after 1736190 and others are committed and 2.x related
problems are solved.
Converting the code to 3.0 should be the last step also because my
perception is that your patch changes too much of the existing code:
a new iterator_producer, a different collect_incoming_data() behavior,
name mangling of internal variables and others.
...A lot of stuff which could be included in a second time.

I'm going to inform Josiah Carlson about this topic since he should be
the most experienced one about asyncore and asynchat.


On 14 Feb, 03:52, Bill Janssen <jans...@parc.com> wrote:
> It's a big patch, but I'll try applying it to the current py3k branch
> -- does it apply? -- and try a few things with it.  I'm concerned
> about how well it behaves with things like Medusa (which probably
> needs its own py3k update).
>
> Bill
> _______________________________________________
> Python-Dev mailing list

> Python-...@python.orghttp://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:http://mail.python.org/mailman/options/python-dev/python-dev2-garchiv...

Giampaolo Rodola'

unread,
Feb 14, 2008, 8:55:24 AM2/14/08
to pytho...@python.org
(wrong quoting: obvioulsly I was talking to Daniel)

Facundo Batista

unread,
Feb 14, 2008, 10:12:49 AM2/14/08
to Giampaolo Rodola', pytho...@python.org
2008/2/14, Giampaolo Rodola' <gne...@gmail.com>:

> asyncore and asynchat are in a difficult position right now since a
> lot of patches for both modules are pending and no decisions are
> taken.
> In detail I'm talking about patches 1519, 1541, 2073 and 1736190 which
> is the most important one since it includes a lot of fixes for other
> reported issues (e.g. 1740572, 1736101, 909005).

I took a look to some of these in other opportunity, and IIRC, the
main issue here is exactly what you say: lots of issues with problem
*and* fixes, but some decisions needs to be taken.

No decision taken, the pile of issues (and real problems behind them)
accumulate through time. I think it's a good idea to bring this
discussion here, and with luck a result will come up regarding them.

So, it would be great if you (please) summarize here the problems
behind those issues and the decisions that must be taken... Thanks!

Regards,

--
. Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/

Giampaolo Rodola'

unread,
Feb 14, 2008, 10:36:55 AM2/14/08
to pytho...@python.org
Ok, I'll try to take a look at all asyncore/chat reports and try to
summarize them by splitting patches which solve bugs and patches which
add enhancements or functionalities.


On 14 Feb, 16:12, "Facundo Batista" <facundobati...@gmail.com> wrote:
> 2008/2/14, Giampaolo Rodola' <gne...@gmail.com>:
>
> > asyncore and asynchat are in a difficult position right now since a
> >  lot of patches for both modules are pending and no decisions are
> >  taken.
> >  In detail I'm talking about patches 1519, 1541, 2073 and 1736190 which
> >  is the most important one since it includes a lot of fixes for other
> >  reported issues (e.g. 1740572, 1736101, 909005).
>
> I took a look to some of these in other opportunity, and IIRC, the
> main issue here is exactly what you say: lots of issues with problem
> *and* fixes, but some decisions needs to be taken.
>
> No decision taken, the pile of issues (and real problems behind them)
> accumulate through time. I think it's a good idea to bring this
> discussion here, and with luck a result will come up regarding them.
>
> So, it would be great if you (please) summarize here the problems
> behind those issues and the decisions that must be taken... Thanks!
>
> Regards,
>
> --
> .    Facundo
>
> Blog:http://www.taniquetil.com.ar/plog/
> PyAr:http://www.python.org/ar/
> _______________________________________________
> Python-Dev mailing list

Daniel Arbuckle

unread,
Feb 14, 2008, 10:44:55 AM2/14/08
to Giampaolo Rodola', pytho...@python.org
First of all, you're conflating my two possible patches.

One patch just addresses the problem of strings and bytes, as GvR
asked me to do, and adds an 8-line class called iterator_producer that
adapts iterators into producers. The patch doesn't change how the
module works at all, and iterator_producer is not invoked anywhere
within the code; it's purely for user convenience. I consider this the
primary patch and would like to focus attention there if possible.

The other patch combines the string and bytes fix with a porting of
1736190 and the other things you complain about, most of which scratch
personal itches. If the patches you mention are actually going to be
applied, then this patch isn't the way to go, and I'll maybe submit
parts of it as separate patches. If they're just going to waste away
in the bug tracker, though, this patch should be seriously considered.

I'm quite willing to re-construct my string and bytes patch against a
version of py3k in which the pre-existing patches are already applied.
There needs to be forward progress, though: If nothing at all gets
done, asyncore is going to be removed from the standard lib. I don't
want to see that happen.

On Thu, Feb 14, 2008 at 5:55 AM, Giampaolo Rodola' <gne...@gmail.com> wrote:
> (wrong quoting: obvioulsly I was talking to Daniel)
>
>
> _______________________________________________
> Python-Dev mailing list
> Pytho...@python.org
> http://mail.python.org/mailman/listinfo/python-dev

> Unsubscribe: http://mail.python.org/mailman/options/python-dev/djarb%40highenergymagic.org

Giampaolo Rodola'

unread,
Feb 14, 2008, 12:09:38 PM2/14/08
to pytho...@python.org

=== Patches for existing issues ===

- 1736190 which includes fixes for the following issues among other
improvements:
- 1063924 (asyncore should handle ECONNRESET in send())
- 1736101 (asyncore should handle ECONNABORTED in recv())
- 760475 (handle_error() should call handle_close() instead of
close())
- 1740572 (refill_buffer() should call handle_close() rather than
close())
- 777588 (wrong "connection refused" behavior on Windows)
- 889153 (wrong connect() behavior)
- 953599 (asyncore misses socket closes when poll is used)
- 1025525 (asyncore.file_dispatcher should not take fd as argument)

- 1519 (async_chat.__init__() and asyncore.dispatcher.__init__
parameters inconsistency)
- 1541 (Bad OOB data management when using asyncore with
select.poll())
- 2073 (asynchat push always sends 512 bytes (ignoring
ac_out_buffer_size))


=== Open issues with no patches (need review) ===

- 658749 (asyncore connect() and winsock errors)
- 1161031 (neverending warnings from asyncore)


=== Enhancements & new features ===

- 1641 (add delayed calls feature)
- 1563 (conversion to py3k and some other changes)


IMHO the first thing to do should be modifying 1736190 patch to fix
the minor issues came out in comments 52767 (re-add simple_producer
and fifo classes) and 57581 (look at what is specified in
ac_out_buffer_size) and then commit it.
I've used that same modified asynchat version in my pyftpdlib project
and I can tell that it works pretty good.
I guess that Josiah Carlson could do that pretty quickly if he has
time to do so.

Independently from all a nice thing to do would be adding tests for
many asyncore/chat behaviors which currently aren't covered by the
test suite.
It could be very useful to know if behaviors between different commits
remain the same, hence it would be better working on that (the test
suite) before committing 1736190.


-- Giampaolo

Josiah Carlson

unread,
Feb 14, 2008, 9:24:04 PM2/14/08
to Giampaolo Rodola', pytho...@python.org
Hey everyone,

Sorry I haven't been available for this recently, I've been working
far too much (10-14 hours/day, 6 days/week, since November) to really
do any "fun" programming. Also, sorry for top-posting.

As I stated 2+ and 6+ months ago, the patches I submitted 9+ months
ago work (I've been using them in my own projects without issue). The
only issue with the bugfix/rearrangement that I last heard about with
regards to the 2.x stuff was that I removed a class that no one was
using in the wild, which I believe Giampaolo added in a subsequent
patch in the last couple months.

My suggestion:
1. Apply the most recent fixes to 2.6 asyncore/asynchat that Giampaolo
has updated.
1.a. Figure out what the hell is up with OOB data, how to handle it,
and stop making it use handle_expt().
2. Use the 2.x -> 3.x tool to convert the fixes over.
3. Apply any 3.x specific fixes (for string/bytes, etc.) to the 3.x
branch as necessary (make them global constants in both 2.x and 3.x so
that they are easy to track).
4. Consider enhancements to 2.6 if they aren't to big, consider
slightly larger enhancements to 3.x. *

- Josiah

* Scheduled tasks are not a valid enhancement for either; anyone who
wants them can trivially add them with their own asyncore.loop()
variant and call asyncore.poll() as necessary (I did it in about 15
lines in the summer of 2002, I hope others can do better now). If you
want my opinion on other async-related features, feel free to email me
directly (use the gmail address you see here, then it ends up in my
inbox, not the overflowing python folder).

A.M. Kuchling

unread,
Feb 15, 2008, 9:24:14 AM2/15/08
to pytho...@python.org
On Thu, Feb 14, 2008 at 06:24:04PM -0800, Josiah Carlson wrote:
> 1.a. Figure out what the hell is up with OOB data, how to handle it,
> and stop making it use handle_expt().

Does OOB data actually need to be supported? For a long time TCP
implementations usually had buggy OOB implementations because it was
so rarely used; for all I know, that's still the case. Maybe the
feature should just be dropped.

--amk

Jon Ribbens

unread,
Feb 15, 2008, 9:28:32 AM2/15/08
to pytho...@python.org
On Fri, Feb 15, 2008 at 09:24:14AM -0500, A.M. Kuchling wrote:
> On Thu, Feb 14, 2008 at 06:24:04PM -0800, Josiah Carlson wrote:
> > 1.a. Figure out what the hell is up with OOB data, how to handle it,
> > and stop making it use handle_expt().
>
> Does OOB data actually need to be supported? For a long time TCP
> implementations usually had buggy OOB implementations because it was
> so rarely used; for all I know, that's still the case. Maybe the
> feature should just be dropped.

Only if you're happy to make it impossible to implement some protocols
using asyncore.

Giampaolo Rodola'

unread,
Feb 15, 2008, 2:45:14 PM2/15/08
to pytho...@python.org
On 15 Feb, 03:24, "Josiah Carlson" <josiah.carl...@gmail.com> wrote:

> As I stated 2+ and 6+ months ago, the patches I submitted 9+ months
> ago work (I've been using them in my own projects without issue). The
> only issue with the bugfix/rearrangement that I last heard about with
> regards to the 2.x stuff was that I removed a class that no one was
> using in the wild, which I believe Giampaolo added in a subsequent
> patch in the last couple months.

I provided the patch for the other issue (look at what is specified in
ac_out_buffer_size) but I didn't re-add fifo and simple_producer
classes.
What should be done here? Re-add or discard them?
However, I will send to you by e-mail the modified asynchat version.
It is based on your patch therefore a first commit could be finally
done.

> My suggestion:
> 1. Apply the most recent fixes to 2.6 asyncore/asynchat that Giampaolo
> has updated.

> 1.a. Figure out what the hell is up with OOB data, how to handle it,
> and stop making it use handle_expt().

If we want to leave OOB untouched shouldn't handle_expt be the right
place where manage such kind of events?
Alternatively we could also remove the OOB management at all (e.g.
Twisted does that by ignoring such kind of data).
OOB could be actually useful to implement some protocols such as FTP
(in details ABOR and STAT commands which require OOB data) but it
would be probably better not having it than having it but not
implemented properly.
I am saying this also because later or soonish we might need to care
of epoll and kqueue (http://bugs.python.org/issue1657).

> * Scheduled tasks are not a valid enhancement for either; anyone who
> wants them can trivially add them with their own asyncore.loop()
> variant and call asyncore.poll() as necessary (I did it in about 15
> lines in the summer of 2002, I hope others can do better now). If you
> want my opinion on other async-related features, feel free to email me
> directly (use the gmail address you see here, then it ends up in my
> inbox, not the overflowing python folder).

How's your solution? Could you post it here or send it to me by mail?
IMO scheduled tasks is a very important feature for implementing a lot
of interesting stuff like connection timeouts and bandwidth
throttling.
A lot of people have to learn/use Twisted just because of that.
Moreover I think that Bill Janssen could need that feature to make the
ssl module work with asyncore.


PS - I have been reading this mailing list for a short time therefore
I have no clue whether or not someone has ever thought about including
the Twisted core - only itself - in Python stdlib.

Josiah Carlson

unread,
Feb 15, 2008, 3:11:31 PM2/15/08
to Giampaolo Rodola', pytho...@python.org
Twisted core has been proposed, but I believe the consensus was that
it wasn't desirable, generally.

I'm also pretty sure that people learn twisted because everyone learns
twisted. It's one of those buzz-words ;).

As for task scheduling, I believe it was something like...

import asyncore
import time
import heapq

tasks = []

def schedule(delta, callme):
heapq.heappush(tasks, (time.time()+delta, callme))

def loop_with_schedule(timeout=30):
while 1:
now = time.time()
while tasks and tasks[0][0] < now:
callme = heapq.heappop(tasks)[1]
try:
callme()
except (KeyboardInterrupt, SystemExit):
raise
except:
#do something useful
pass
thistimeout = timeout
if tasks:
thistimeout = max(min(thistimeout, tasks[0][0]-now), 0)

asyncore.poll(timeout=thistimeout)

- Josiah

> Unsubscribe: http://mail.python.org/mailman/options/python-dev/josiah.carlson%40gmail.com

Bill Janssen

unread,
Feb 15, 2008, 3:36:46 PM2/15/08
to Josiah Carlson, Giampaolo Rodola', pytho...@python.org
I think we should just replace the current "loop" with this (and add
the "schedule" function). Then other folks won't have to figure out
how the module works and write their own loop. I'll be happy to update
the documentation to document it.

Bill

> Unsubscribe: http://mail.python.org/mailman/options/python-dev/janssen%40parc.com

Giampaolo Rodola'

unread,
Feb 15, 2008, 3:55:40 PM2/15/08
to pytho...@python.org
On 15 Feb, 21:36, Bill Janssen <jans...@parc.com> wrote:
> I think we should just replace the current "loop" with this (and add
> the "schedule" function).  Then other folks won't have to figure out
> how the module works and write their own loop.  I'll be happy to update
> the documentation to document it.
>
> Bill

I'm -1.
This does not permit to reset, cancel or "re-schedule" the scheduled
tasks.
Something like a connection timeout after a certain time of client's
inactivity would not be possible to implement.

Joel Bender

unread,
Feb 15, 2008, 4:05:18 PM2/15/08
to Bill Janssen, pytho...@python.org
Bill Janssen wrote:
> I think we should just replace the current "loop" with this (and add
> the "schedule" function). Then other folks won't have to figure out
> how the module works and write their own loop.

Having beaten my way down this road of broken glass, I would like args
and kwargs if you are adding this:

def schedule(delta, fn, *args, **kwargs):
heap.heappush(tasks, (time.time()+delta, (fn, args, kwargs)))

...

callme[0](*callme[1], **callme[2])


Joel

Giampaolo Rodola'

unread,
Mar 2, 2008, 6:19:22 PM3/2/08
to pytho...@python.org
I've discussed a lot with Josiah via e-mail and I provided an updated
version of the patch proposed in bug #1736190 including a fix for the
two issues raised by me in the bug report.
The update has been needed also because the original patch has been
out-dated by some commits after r53734 involving the test suite
and the documentation, which I both took off this patch.
I also re-added simple_producer and fifo classes in asynchat.py since
it seems they are needed for passing tests.

The test suite has passed on Windows XP using Python 2.6 alpha 1.
I have also successfully run the test suite of a project of mine based
on asynchat which includes over 40 tests.

Josiah Carlson

unread,
Mar 2, 2008, 6:59:42 PM3/2/08
to Giampaolo Rodola', pytho...@python.org
As far as I can tell, the asyncore.py, asynchat.py, and updated
test_asyncore.py are good. I have been using earlier variants in my
own projects (prior to their updating to pass the test suite) for
quite a few months now. The updated modules provide better
performance, features, and support for real-world async socket
servers, never mind fixing more than a half dozen outstanding bugs.

- Josiah

Giampaolo Rodola'

unread,
Mar 24, 2008, 3:43:33 PM3/24/08
to pytho...@python.org
I'm going to refresh this discussion since it seems no decisions are
still taken.
Any chance to see a commit finally done?


--- Giampaolo
http://code.google.com/p/pyftpdlib

Thomas Wouters

unread,
Mar 24, 2008, 5:04:20 PM3/24/08
to Josiah Carlson, Giampaolo Rodola', pytho...@python.org
On Fri, Feb 15, 2008 at 9:11 PM, Josiah Carlson <josiah....@gmail.com> wrote:
Twisted core has been proposed, but I believe the consensus was that
it wasn't desirable, generally.

I remember only a couple of dissenting voices, and only a small number of participants. Of the dissenting voices, I do not recall any actual arguments about undesireability, just misunderstandings of how Twisted actually works. Getting Twisted core (meaning Deferreds, a simple reactor and the Protocol class) into the core is still on my TODO list.

I'm also pretty sure that people learn twisted because everyone learns
twisted.  It's one of those buzz-words ;).

I think that's quite an unfair assessment, even in jest :) Twisted is well worth learning to actually use it, as it's a very versatile event loop and does it best to integrate nicely with other event systems. And including it in the standard library improves integration with other event loops by creating a single interface. It's not a matter of dropping it in, though; it requires some careful structuring to avoid embarrassing situations like we have with the xml package, but still people to provide their own reactor.

In case you're wondering how the twisted reactor in the stdlib is useful to people not using Twisted, take a look at what you currently need to do to combine stdlib modules like urllib and ftplib with event systems like Tkinter and PyGTK. Not to mention that the Twisted implementations of various protocols are really quite, quite good -- in many cases quite a lot better than the stdlib ones. But including those takes yet more time.

--
Thomas Wouters <tho...@python.org>

Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

Adam Olsen

unread,
Mar 24, 2008, 5:21:53 PM3/24/08
to Thomas Wouters, Giampaolo Rodola', pytho...@python.org

In that sense it'd be competing with safethread for inclusion in
Python. Whereas safethread requires little if any API changes,
twisted requires an entirely new API that can be event-driven. Worse,
we'd likely to be stuck maintaining both APIs for a long time, if not
forever.

Twisted may be one of the best (if not *the* best) ways of writing
concurrent programs today, but it doesn't need to be in the stdlib for
that. If safethread is going to solve many of the same problems, with
less changes required by the users of the language, then this is the
wrong time to add twisted.


--
Adam Olsen, aka Rhamphoryncus

Thomas Wouters

unread,
Mar 24, 2008, 5:37:43 PM3/24/08
to Adam Olsen, Giampaolo Rodola', pytho...@python.org

I am not going to worry about this for the time being. Even if safethread goes in and becomes ubiquitous, Twisted is still a very valid approach to the problem. And I'm not just saying that because I don't subscribe to your enthusiasm of safethreads as a concept or as an implementation :)
 

Twisted may be one of the best (if not *the* best) ways of writing
concurrent programs today, but it doesn't need to be in the stdlib for
that.  If safethread is going to solve many of the same problems, with
less changes required by the users of the language, then this is the
wrong time to add twisted.

You must have missed the part where we already have a large set of event loops, and not having a single interface to them is in fact hurting people. Twisted goes out of its way to interact nicely with event loops, but it can only do that with ones it knows about (and are versatile enough to hook into.) Having a single event system in the standard library is definitely advantageous, even if safethreads were available everywhere and the performance in the common case was satisfactory. It used to be the case that people thought asyncore was this standard event system, but it's long since ceased to be.

A.M. Kuchling

unread,
Mar 24, 2008, 6:18:23 PM3/24/08
to pytho...@python.org
On Mon, Mar 24, 2008 at 10:04:20PM +0100, Thomas Wouters wrote:
> I remember only a couple of dissenting voices, and only a small number of
> participants. Of the dissenting voices, I do not recall any actual arguments

Weren't some of those dissenting voices the Twisted developers, though?

--amk

Adam Olsen

unread,
Mar 24, 2008, 6:25:51 PM3/24/08
to Thomas Wouters, Giampaolo Rodola', pytho...@python.org

I'm not opposed to standardizing on twisted as the canonical way to do
events in python, or to having an event system in python. My concern
is that may be used as an excuse to slowly rewrite the entire language
into an event-driven model.

However, that was based on the assumption that modules like urllib2
weren't already event-driven. Looking now, it seems it already is,
and wraps it in a blocking API for simple use cases.

So long as we're only talking about replacing existing event-driven
stuff, and so long as we retain the existing blocking API (including
calling from threads!), I don't think I have any valid opposition.

Josiah Carlson

unread,
Mar 24, 2008, 6:51:56 PM3/24/08
to a...@amk.ca, pytho...@python.org
Let us not get side-tracked in this discussion. Whether or not to
include any portion of Twisted into Python 2.6 is well past being a
reasonable question; 2.6 alpha 1 has been released. It's a question
as to whether someone with commit access can or will commit the patch
as posted, run the tests to verify that they aren't broken, and
perhaps actually look at the code to verify that we (Giampaolo and I)
aren't insane. Mind you, I've been using very similar variants of
this patch for months; it fixes outstanding bugs, improves
performance, makes the handle* interfaces more consistent, and even
offers a 'sample' implementation of a basic protocol in the source
(for those who are willing to look). Do we want to fix
asyncore/asynchat with work that has already been done and tested?

If you want a reason as to why twisted shouldn't *replace*
asyncore/asynchat, I'll give you a few: As stated previously by Guido
and others (please see previous threads about this over the course of
the last 4 years), asyncore/asynchat provide a more or less minimal
interface for asynchronous sockets in Python. Any module/package that
seeks to implement asynchronous sockets will need to, at least,
implement that much. Asyncore/asynchat at present will play nicely
with any event loop available, given certain caveats*. Further, if
someone spends a half hour reading the source of a reasonably written
asyncore server/client, they should understand the basics and be able
to begin using it directly (see any one of the simple echo variants).

As to whether twisted should be added to the standard library at some
point in the future as a "this is better than asyncore in every way,
use this instead"; that is a different discussion, not related to 2.6
(perhaps not even related to the 2.x series at all, depending on the
future of 2.x).


- Josiah

* If your application strictly responds to socket IO, then implement
your application as part of handle_* methods. If your application
does more, then call asyncore.poll() often enough for data to be
handled sufficiently fast. If neither offer sufficient performance or
flexibility (maybe something like bittorrent + wxPython), use one
thread for your GUI, one thread for your sockets, and use
Queue.Queue() or some other threadsafe message delivery method.

> Unsubscribe: http://mail.python.org/mailman/options/python-dev/josiah.carlson%40gmail.com

A.M. Kuchling

unread,
Mar 25, 2008, 7:00:51 AM3/25/08
to pytho...@python.org
On Mon, Mar 24, 2008 at 03:51:56PM -0700, Josiah Carlson wrote:
> reasonable question; 2.6 alpha 1 has been released. It's a question
> as to whether someone with commit access can or will commit the patch
> as posted, run the tests to verify that they aren't broken, and
> perhaps actually look at the code to verify that we (Giampaolo and I)
> aren't insane. Mind you, I've been using very similar variants of

I think we should just give you commit access so that you can commit
changes to asyncore/asynchat yourself; it doesn't seem as if any of
the committers use asyncore enough to check patches for it.

--amk

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
http://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-dev2-garchive-22421%40googlegroups.com

Neal Norwitz

unread,
Mar 26, 2008, 12:00:18 AM3/26/08
to Giampaolo Rodola', pytho...@python.org
On Thu, Feb 14, 2008 at 10:09 AM, Giampaolo Rodola' <gne...@gmail.com> wrote:
> On 14 Feb, 16:36, "Giampaolo Rodola'" <gne...@gmail.com> wrote:
> > Ok, I'll try to take a look at all asyncore/chat reports and try to
> > summarize them by splitting patches which solve bugs and patches which
> > add enhancements or functionalities.
>

That's a lot of patches. My immediate concern is that test_asynchat
is very flaky and fails often. Sometimes it causes other tests to
fail. Is there a patch that addresses this? If you need examples,
just look through the buildbot mails that mention test_asynchat in:
http://mail.python.org/pipermail/python-checkins/

Some platforms have more problems than others, but almost all
platforms have failed test_asynchat at one point or another.

Please break up the patches into 2 sets and prioritize the patches
with the set.

Set #1: Patches that have a test and doc updates if required
Set #2: Patches that don't have a test or doc updates as required

For the patches that fall into Set #1, list them in priority order.
Top priority would be a problem that fixes failures seen in the
buildbots. Next priority would go to the patches that solve more
serious problems. Post the results here. I will try to look at them.

For every patch you list, make sure that it conforms to the proper
style (e.g, PEP 8) and is essentially perfect and ready for inclusion.
This means that there is a single file to download that contains all
the modifications. The changes are appropriately commented, lines are
less than 80 characters, etc. One of the modifications should be an
entry in Misc/NEWS.

n

Josiah Carlson

unread,
Mar 26, 2008, 3:21:18 AM3/26/08
to Neal Norwitz, Python-Dev
On Tue, Mar 25, 2008 at 11:26 PM, Neal Norwitz <nnor...@gmail.com> wrote:
> Any reason this was sent just to me and not the list?

Because gmail only replies to the sender by default. I need to
remember to cc python-dev when I reply (I used the same email client
for 8 1/2 years, remembering the quirks of gmail may take some time).

> On Tue, Mar 25, 2008 at 10:34 PM, Josiah Carlson
> <josiah....@gmail.com> wrote:

> > No, it's one patch. All of the fixes were performed mostly by myself
> > last spring, tested and verified in personal servers, then re-used by
> > Giampaolo in his async ftp server (which pointed out a few small bugs,
> > which have been fixed).


> >
> >
> > > Some platforms have more problems than others, but almost all
> > > platforms have failed test_asynchat at one point or another.
> >

> > Certainly that is the case. And according to my reading of a few
> > buildbot failures, aside from someone breaking asyncore itself, the
> > failures seem to stem from the test being unable to create a port to
> > listen on in order to test the server/client functionality. This is a
> > buildbot configuration issue (per host), not an asyncore issue.
>
> That was the case a long time (~3? months) ago, but hasn't been the
> case recently. See my recent message about the release.

I'll look for it tomorrow. For reference, searches of
'site:mail.python.org test_asynchat failure buildbot' only seem to
produce the socket listen error. If there is a better incantation to
get google to produce the proper errors (and/or a link), I would
appreciate the help.

> > > Please break up the patches into 2 sets and prioritize the patches
> > > with the set.
> > >
> > > Set #1: Patches that have a test and doc updates if required
> > > Set #2: Patches that don't have a test or doc updates as required
> > >
> > > For the patches that fall into Set #1, list them in priority order.
> > > Top priority would be a problem that fixes failures seen in the
> > > buildbots. Next priority would go to the patches that solve more
> > > serious problems. Post the results here. I will try to look at them.
> > >
> > > For every patch you list, make sure that it conforms to the proper
> > > style (e.g, PEP 8) and is essentially perfect and ready for inclusion.
> > > This means that there is a single file to download that contains all
> > > the modifications. The changes are appropriately commented, lines are
> > > less than 80 characters, etc. One of the modifications should be an
> > > entry in Misc/NEWS.
> >

> > I lied earlier; really there are two patches. The first is a patch to
> > asyncore.py and asynchat.py . It addresses those bugs that Giampaolo
> > has listed, it is tested, and works. The second patch is to update
> > the documentation to mention the sample methods in asynchat for use as
> > examples, as well as any other changes to the language used in the
> > documentation that I had made last spring, but which are out of date
> > from my posting of the original patch. I can update the documentation
> > in the next week.
>
> Can you provide a link to the patches? Do the patches include changes
> to test_asyncore and test_asynchat? The next release is April 2. I
> would like to commit any patches before Monday to ensure they are
> stable. Can you get me the patches by this Saturday?

See http://bugs.python.org/issue1736190 for an updated patch for the
modules. The current test cases pass without issue, though we may
want to add tests, which I need to look at the original patch and the
original file from which it was created against, then compare it with
the most recent changes to the tests from Facundo last June or July.

I should have the time to get patches for tests and documentation by Monday.

- Josiah

Josiah Carlson

unread,
Mar 30, 2008, 10:44:03 PM3/30/08
to Neal Norwitz, Python-Dev
(sorry for top posting)

I haven't really had time to update the tests/documentation, but
again, I wasn't experiencing any strange test failures with
asyncore/asynchat, nor have I been able to find the buildbot failures
that you are referring to. Could someone please link the failures
that are not related to being unable to discover a port number?

According to the release schedule, we should have at least a couple
more months for documentation and tests to be updated (I can get
patches ready for alpha 3).

- Josiah

On Wed, Mar 26, 2008 at 12:21 AM, Josiah Carlson

Neal Norwitz

unread,
Mar 31, 2008, 3:11:49 AM3/31/08
to Josiah Carlson, Python-Dev
On Sun, Mar 30, 2008 at 7:44 PM, Josiah Carlson
<josiah....@gmail.com> wrote:
>
> I haven't really had time to update the tests/documentation, but
> again, I wasn't experiencing any strange test failures with
> asyncore/asynchat, nor have I been able to find the buildbot failures
> that you are referring to. Could someone please link the failures
> that are not related to being unable to discover a port number?

3 different platforms, 3 different problems:

http://www.python.org/dev/buildbot/all/alpha%20Tru64%205.1%20trunk/builds/2790/step-test/0
http://www.python.org/dev/buildbot/all/x86%20FreeBSD%203%203.0/builds/0/step-test/0
http://www.python.org/dev/buildbot/all/x86%20XP-4%203.0/builds/643

> According to the release schedule, we should have at least a couple
> more months for documentation and tests to be updated (I can get
> patches ready for alpha 3).

When you get the patches with tests and doc, I'll be happy to check in.

n

Josiah Carlson

unread,
May 4, 2008, 2:13:53 PM5/4/08
to Neal Norwitz, Python-Dev
On Mon, Mar 31, 2008 at 12:11 AM, Neal Norwitz <nnor...@gmail.com> wrote:
> On Sun, Mar 30, 2008 at 7:44 PM, Josiah Carlson
>
> <josiah....@gmail.com> wrote:
> >
>
> > I haven't really had time to update the tests/documentation, but
> > again, I wasn't experiencing any strange test failures with
> > asyncore/asynchat, nor have I been able to find the buildbot failures
> > that you are referring to. Could someone please link the failures
> > that are not related to being unable to discover a port number?
>
> 3 different platforms, 3 different problems:
>
> http://www.python.org/dev/buildbot/all/alpha%20Tru64%205.1%20trunk/builds/2790/step-test/0
> http://www.python.org/dev/buildbot/all/x86%20FreeBSD%203%203.0/builds/0/step-test/0
> http://www.python.org/dev/buildbot/all/x86%20XP-4%203.0/builds/643
>
>
> > According to the release schedule, we should have at least a couple
> > more months for documentation and tests to be updated (I can get
> > patches ready for alpha 3).
>
> When you get the patches with tests and doc, I'll be happy to check in.
>
> n

I have updated the documentation, and as much of the tests as was
required to pass on my Windows XP machine. In looking at the
buildbots, I'm not seeing any more common issues. Unfortunately, I
also hit the urls provided above too late, and I wasn't able to see
the actual issues (if they still persist).

If possible, the syntax in the documentation that I added should be
checked, as I needed to convert from my older TeX docs to the new ReST
doc format.

I have attached the patch file, and am in the process of regaining
access to the bug tracker. In the mean time, Giampaolo will be
posting the patch to the tracker in issue 1736190.

If anyone has any questions, please ask.

- Josiah

full_async_patch.patch
Reply all
Reply to author
Forward
0 new messages