Best approach for simultaneous async API calls

[gb]

I need to render a page that requires a number of REST API calls to external web services that take a significant amount of time to return.

What is the best approach to fire all of these calls simultaneously instead of sequentially?

Should I try something like Celery:   
    https://groups.google.com/forum/#!topic/web2py/js0VAKK6Dlw
    https://code.google.com/p/web2py-celery/

Seems like this approach wasn't fully developed.

I thought about using the web2py scheduler: http://web2py.com/books/default/chapter/29/04/the-core#web2py-Scheduler
but this seemed more appropriate for background cronjobs (like email batching), rather than live page rendering.

Also, I'm not sure if it's truly asynchronous.

I realize this usually would be done client side (There are some security reasons for doing this server-side rather than just JS ajax calls).

I've also read that due to the python UWSGI that true multi-threading might not be achievable in web2py.  Is this the case?

Am I going about this the wrong way and should I use a hammer (node.js) instead of a nice powertool drill (web2py)?

[Niphlod]

woah. that's a lot of unconcerned copy-paste around the web. Did you even try to start "small" and then see what your issues would be ?

let's take every statement in order.

I need to render a page that requires a number of REST API calls to external web services that take a significant amount of time to return.

that's not a "problem" for any serious programming language. The thing is, usually webservers put a hard timeout on the time it takes to generate a result, so if your "rest api calls" are taking more than -e.g.- 60 seconds, the right thing to do is to avoid using the webserver and let another "thing" to do it, and store the results somewhere your page can fetch them, without incurring in the timeout
 

What is the best approach to fire all of these calls simultaneously instead of sequentially?

ajax would be my first and obvious choice.
 

Should I try something like Celery:   
    https://groups.google.com/forum/#!topic/web2py/js0VAKK6Dlw
    https://code.google.com/p/web2py-celery/

Seems like this approach wasn't fully developed.

that is the "old way" to let celery integrate with python. the newer celery is usable without any dependency on specific code (or so it's advertised to)
 

I thought about using the web2py scheduler: http://web2py.com/books/default/chapter/29/04/the-core#web2py-Scheduler
but this seemed more appropriate for background cronjobs (like email batching), rather than live page rendering.

Also, I'm not sure if it's truly asynchronous.

trying wouldn't hurt, since setting it up takes 3 lines of code. I can process 3000 tasks divided upon 5 workers in under 200 seconds, on modest hardware. If 15 tasks per second are not enough, sure, scheduler isn't for you, but you may very well being working on something at the "twitter" level....... you'd need some power-tools to let your application behave like it should.
 

I realize this usually would be done client side (There are some security reasons for doing this server-side rather than just JS ajax calls).

I don't see why you couldn't do this server-side, taken for granted the timeout considerations in the first sentence
 

I've also read that due to the python UWSGI that true multi-threading might not be achievable in web2py.  Is this the case?

I really don't know where you copy-pasted this: it's blatantly false, and web2py would have no practical use for nobody if that was the case
 

Am I going about this the wrong way and should I use a hammer (node.js) instead of a nice powertool drill (web2py)?

Don't know really.
<ironic mode on>
You could need node.js workers attached to a hive powered by pig cluster fetching data continously from cassandra, storing temporal results on mongodb, searching them with elasticsearch or solr, decentralizing your resultsets on foundationdb, keeping scores with redis backed by lua scripts, fetching them asyncronously with goroutines and denormalizing them in couchdb....
then do a graph analysis on neo4j, passing messages around with the help of rabbitmq.
All of it running on a severely optimized kernel recompilation of freebsd done by yourself to power your own fleet of ARM physical servers. Additionally, you may use websockets provided by pusher.com and adopt a CDN to speed things up even further than approaching amazon S3, or even better cloudflare.
You can serve the layout from web2py, and that would be pretty much about your application.
<ironic mode off>

Sorry, I was feeling a taddle bit poetic. Nothing in web2py is stopping you from using ANY other tool to do the job, but premature optimization is the root of all evil. Start small and go from there, without copy-pasting the entire web of buzzword you can find around. See for yourself.

[Dave S]

On Tuesday, February 24, 2015 at 1:13:59 PM UTC-8, Niphlod wrote:

[...]

 

The thing is, usually webservers put a hard timeout on the time it takes to generate a result, so if your "rest api calls" are taking more than -e.g.- 60 seconds, the right thing to do is to avoid using the webserver and let another "thing" to do it, and store the results somewhere your page can fetch them, without incurring in the timeout
 

What is the best approach to fire all of these calls simultaneously instead of sequentially?

ajax would be my first and obvious choice.
 

I have a small internal app that polls several machines to display status information.  I did the message handling in-line for a first cut, but I'm thinking about moving that part into the scheduler, maybe using a temporary table for the results although I expect a temp file would also work, while having the original page just be an ajax load of the-results-so-far that gets updated by a timer until enough time has passed.

Since the enhancement is a background task, it may take me a while to get started, even though the coding looks small.

/dps