Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
asynchronous downloading
35 views
Skip to first unread message
Plumo
Feb 23, 2012, 1:58:10 AM2/23/12
to
I want to download content asynchronously. This would be straightforward to do threaded or across processes, but difficult asynchronously so people seem to rely on external libraries (twisted / gevent / eventlet).
(I would use gevent under different circumstances, but currently need to stick to standard libraries.)
I looked around and found there is little interest in developing a proper HTTP client on asyncore. The best I found stopped development a decade ago: http://sourceforge.net/projects/asynchttp/
What do you recommend?
And why is there poor support for asynchronous execution?
Richard
(I would use gevent under different circumstances, but currently need to stick to standard libraries.)
I looked around and found there is little interest in developing a proper HTTP client on asyncore. The best I found stopped development a decade ago: http://sourceforge.net/projects/asynchttp/
What do you recommend?
And why is there poor support for asynchronous execution?
Richard
Justin Ezequiel
Feb 23, 2012, 2:25:32 AM2/23/12
to
have you seen http://www.doughellmann.com/PyMOTW/asyncore/
Mark Hammond
Feb 23, 2012, 6:20:40 AM2/23/12
to Plumo, pytho...@python.org
On 23/02/2012 5:58 PM, Plumo wrote:
> I want to download content asynchronously. This would be
> straightforward to do threaded or across processes, but difficult
> asynchronously so people seem to rely on external libraries (twisted
> / gevent / eventlet).
Exactly - the fact it's difficult is why those tools compete.
> I want to download content asynchronously. This would be
> straightforward to do threaded or across processes, but difficult
> asynchronously so people seem to rely on external libraries (twisted
> / gevent / eventlet).
> (I would use gevent under different circumstances, but currently need
> to stick to standard libraries.)
modest tasks. If your needs go beyond modest, I'd reevaluate your need
to stick with just the stdlib - even demanding *sync* http apps often
wind up using modules outside the stdlib. Look into virtualenv etc if
permission to install packages is the issue.
Batteries included free, but turbo-chargers are an extra ;)
Mark
Paul Rubin
Feb 23, 2012, 6:46:00 AM2/23/12
to
Plumo <rich...@gmail.com> writes:
> What do you recommend?
Threads.
> And why is there poor support for asynchronous execution?
The freenode #python crowd seems to hate threads and prefer twisted,
which seems to have the features you want and probably handles very
large #'s of connections better than POSIX threads do. But I find the
whole event-driven model to be an annoying abstraction inversion and
threads to be simpler, so I've stayed with threads. I keep hearing
boogieman stories about the evil hazards of race conditions etc. but
none of that stuff has ever happened to me (yet) as far as I can tell.
The main thing is to avoid sharing mutable data between threads to the
extent that you can. Keep the threads isolated from each other except
for communication through Queues and not too much can go wrong. The
last program I wrote had around 20 threads and one or two condition
variables and I don't think any significant bugs resulted from that.
FWIW, the Erlang language is built around the above concept, it uses
super-lightweight userland threads so it can handle millions of them
concurrently, and it's used successfully for ultra-high-reliability
phone switches and similar applications that are not allowed to fail, so
it must be doing something right.
There are a few schemes like Camaelia (sp?) implementing concurrency
with Python generators or coroutines, but I think they're not widely
used, and Python coroutines are kind of crippled because they don't
carry any stack below their entry point.
> What do you recommend?
Threads.
> And why is there poor support for asynchronous execution?
which seems to have the features you want and probably handles very
large #'s of connections better than POSIX threads do. But I find the
whole event-driven model to be an annoying abstraction inversion and
threads to be simpler, so I've stayed with threads. I keep hearing
boogieman stories about the evil hazards of race conditions etc. but
none of that stuff has ever happened to me (yet) as far as I can tell.
The main thing is to avoid sharing mutable data between threads to the
extent that you can. Keep the threads isolated from each other except
for communication through Queues and not too much can go wrong. The
last program I wrote had around 20 threads and one or two condition
variables and I don't think any significant bugs resulted from that.
FWIW, the Erlang language is built around the above concept, it uses
super-lightweight userland threads so it can handle millions of them
concurrently, and it's used successfully for ultra-high-reliability
phone switches and similar applications that are not allowed to fail, so
it must be doing something right.
There are a few schemes like Camaelia (sp?) implementing concurrency
with Python generators or coroutines, but I think they're not widely
used, and Python coroutines are kind of crippled because they don't
carry any stack below their entry point.
Richard Baron Penman
Feb 23, 2012, 8:50:34 AM2/23/12
to Mark Hammond, pytho...@python.org
>> I want to download content asynchronously. This would be
>> straightforward to do threaded or across processes, but difficult
>> asynchronously so people seem to rely on external libraries (twisted
>> / gevent / eventlet).
>
>
>> straightforward to do threaded or across processes, but difficult
>> asynchronously so people seem to rely on external libraries (twisted
>> / gevent / eventlet).
>
>
> Exactly - the fact it's difficult is why those tools compete.
It is difficult in Python because the async libraries do not offer
much. Straightforward in some other languages.
Do you know why there is little support for asynchronous execution in
the standard libraries?
For large scale downloading I found thread pools do not scale well.
Richard
Giampaolo Rodolà
Feb 23, 2012, 12:31:54 PM2/23/12
to Plumo, pytho...@python.org
> http://mail.python.org/mailman/listinfo/python-list
If you want to stick with asyncore try to take a look at this:
https://gist.github.com/1519999
> And why is there poor support for asynchronous execution?
There are plenty of choices amongst third party modules though.
To say one, I particularly like tornado which is simple and powerful:
http://www.tornadoweb.org/documentation/httpclient.html
--- Giampaolo
http://code.google.com/p/pyftpdlib/
http://code.google.com/p/psutil/
http://code.google.com/p/pysendfile/
Plumo
Feb 23, 2012, 7:28:05 PM2/23/12
to
My current implementation works fine below a few hundred threads. But each thread takes up a lot of memory so does not scale well.
I have been looking at Erlang for that reason, but found it is missing useful libraries in other areas.
I have been looking at Erlang for that reason, but found it is missing useful libraries in other areas.
Plumo
Feb 23, 2012, 8:10:25 PM2/23/12
to comp.lan...@googlegroups.com, pytho...@python.org, Plumo
that example is excellent - best use of asynchat I have seen so far.
I read through the python-dev archives and found the fundamental problem is no one maintains asnycore / asynchat.
I read through the python-dev archives and found the fundamental problem is no one maintains asnycore / asynchat.
Plumo
Feb 23, 2012, 8:10:25 PM2/23/12
to Plumo, pytho...@python.org
Fayaz Yusuf Khan
Feb 23, 2012, 9:31:09 PM2/23/12
to pytho...@python.org
On Thursday 23 Feb 2012 5:10:25 PM Plumo wrote:
> I read through the python-dev archives and found the fundamental problem is
> no one maintains asnycore / asynchat.
By all means, scratch your own itch. :)
> I read through the python-dev archives and found the fundamental problem is
> no one maintains asnycore / asynchat.
--
Fayaz Yusuf Khan
Cloud developer and architect
Dexetra SS, Bangalore, India
fayaz.yusuf.khan_AT_gmail_DOT_com
fayaz_AT_dexetra_DOT_com
+91-9746-830-823
Giampaolo Rodolà
Feb 24, 2012, 4:03:45 AM2/24/12
to Plumo, pytho...@python.org, comp.lan...@googlegroups.com
Il 24 febbraio 2012 02:10, Plumo <rich...@gmail.com> ha scritto:
> that example is excellent - best use of asynchat I have seen so far.
>
> I read through the python-dev archives and found the fundamental problem is no one maintains asnycore / asynchat.
Well, actually I do/did.
> that example is excellent - best use of asynchat I have seen so far.
>
> I read through the python-dev archives and found the fundamental problem is no one maintains asnycore / asynchat.
Point with asyncore/asynchat is that it's original design is so flawed
and simplicistic it doesn't allow actual customization without
breaking compatibility.
See for example:
http://bugs.python.org/issue6692
Richard Baron Penman
Feb 25, 2012, 4:44:16 PM2/25/12
to Giampaolo Rodolà, pytho...@python.org, comp.lan...@googlegroups.com
>> I read through the python-dev archives and found the fundamental problem is no one maintains asnycore / asynchat.
>
> Well, actually I do/did.
>
ah OK. I had read this comment from a few years back:
"IIRC, there was a threat to remove asyncore because there were no
maintainers, no one was fixing bugs, no one was improving it, and no
one was really using it"
> Point with asyncore/asynchat is that it's original design is so flawed
> and simplicistic it doesn't allow actual customization without
> breaking compatibility.
Richard
0 new messages