Make Multiple Asynchronous Requests

[mattd]

What's the best way to make more than one async call in a single
RequestHandler method?

[Carl S. Yestrau Jr.]

You need some way to keep track of them.

The easiest solution is to keep a number and increment the count on
every callback. Once the number is at the right count write the
response.

I wrote a buffered writer class to help me with multi async request situations:
See BufferedWriter:
http://github.com/docyes/splunk_blaze/blob/master/splunkblaze/web.py

Maybe someone has a better best pattern or practice.

Carl

On Thu, Apr 1, 2010 at 4:13 PM, mattd <mattde...@gmail.com> wrote:
> What's the best way to make more than one async call in a single
> RequestHandler method?
>
>

> --
> To unsubscribe, reply using "remove me" as the subject.
>

[Otávio Souza]

Maybe the use of python lambas help better ;)

2010/4/1 Carl S. Yestrau Jr. <ca...@featureblend.com>

--
Otávio Souza
* KinuX Linux Creator <http://kinuxlinux.org>
* Participante Linux-SE
* Criador dos lunatiKos (Grupo de usuários KDE do Nordeste)
* Linux User #415774

[Otávio Souza]

Sorry, typo error, python lambdas =)

Em 2 de abril de 2010 01:41, Otávio Souza <aragao...@gmail.com> escreveu:

Maybe the use of python lambas help better ;)

[Carl S. Yestrau Jr.]

Do you have an example?

2010/4/1 Otávio Souza <aragao...@gmail.com>:

[Otávio Souza]

Sure: http://gist.github.com/353138

As tornado or pycurl has what I think is a bug in AsyncHTTPClient, just performing one fetch per-time, I wrote a more complete one, but you may test the two of them =)

2010/4/2 Carl S. Yestrau Jr. <ca...@featureblend.com>

Do you have an example?

[Thomas Rampelberg]

AsyncHTTPClient can do more than one request at a time. Make sure that
you change the max pool size and it'll work. I've had more than 2k
requests going at once without much problem after screwing with the
pool size.

tornado.httpclient.AsyncHTTPClient(max_clients=1000)

2010/4/2 Otávio Souza <aragao...@gmail.com>:

[L-R]

I know this thread is quite old - but is this solution still relevant? If I wanted to wait for multiple operations to finish in a loop, having a counter is still a good way to keep track of operations?

[A. Jesse Jiryu Davis]

I just wrote this today, is this what people are looking for?:

https://yieldpoints.readthedocs.org/en/latest/examples/index.html

[L-R]

This definitively looks helpful - seems like Tornado could use more tools for dealing with different async patterns.

I meant more generally - if I have an unpredictable number of async operations to perform, and want to wait for all to be done, one answer in this thread suggests keeping track of them with a counter... 

The easiest solution is to keep a number and increment the count on
every callback. Once the number is at the right count write the
response.

[Ben Darnell]

There are a number of patterns you can use.  If you're using tornado.gen, you can yield a list of gen.Tasks or gen.Waits to wait for all of them in parallel (or use Jesse's yieldpoints library for some additional options like WaitAny).  If you're not using gen.engine you have a lot of flexibility and what works best for you will probably depend on the style of the rest of your application.  Personally I like this pattern: https://gist.github.com/741041

-Ben