Trying to do RPC with some of the RPC calls being async coroutines (ie long running downloads)

[Dharhas Pothina]

Hi,

I'm seeing if I can use pulsar for my use case.

I need to use RPC, but some of the functions the RPC will be calling may be long running downloads or maybe computations so I would like to run them using async. I'm new to async programming and I'm having trouble understanding how to combine the two pieces.

My current attempt at using pulsar for a proof of concept is here:

https://gist.github.com/dharhas/4cce183e954dcbb36c83

This runs but the long_download function currently blocks and doesn't run as async.

The long_download function would in real life actually be calling another python module called 'ulmo' that uses 'requests' extensively so my understanding is that it would be blocking and would need to be run in an executer loop.

Any help/examples on how to do this would be greatly appreciated.

thanks,

- dharhas

[lsbardel]

Hi, 

I need to use RPC, but some of the functions the RPC will be calling may be long running downloads or maybe computations so I would like to run them using async. I'm new to async programming and I'm having trouble understanding how to combine the two pieces.

Long running downloads can be handled quite well using pulsar.

Long running computations are a different problem since they are CPU bound operations rather than IO bound.

For CPU bound calculations you should use the event loop executor.

My current attempt at using pulsar for a proof of concept is here:

https://gist.github.com/dharhas/4cce183e954dcbb36c83

This runs but the long_download function currently blocks and doesn't run as async.

Why are you saying that?

Your example is fine and the long_download function does not block the server!

You can try by sending more than one request at once:

from asyncio import wait
from pulsar.apps import rpc

r = rpc.JsonProxy('http://127.0.0.1:8060')
r._loop.run_until_complete(wait([r.long_download(), r.long_download()], loop=r._loop))

 And you will see the server logging two requests at once

The long_download function would in real life actually be calling another python module called 'ulmo' that uses 'requests' extensively so my understanding is that it would be blocking and would need to be run in an executer loop.

That is correct, requests is a blocking http client and therefore you should use the event loop executor.

If you can use pulsar HttpClient rather than requests, your server could be fully asynchronous.

[Dharhas Pothina]

Hi Isbardel,

Thanks for helping out. I'm new to async type programming and I may be trying to do the wrong thing. My python api will eventually be called from a C++ qt application so I've been testing using curl, i.e.:

curl -H "Content-Type: application/json" -X POST -d '{"jsonrpc": "2.0", "method": "long_download", "params": [0], "id": 0}' 127.0.0.1:8060

this does seem to block when I try it. Or at least if I get a delay of 5 secs, I have to wait 5 seconds before the curl command completes.

What I need is for the long_download function to do is return control immediately along with a 'operation started' message and then continue to conduct the download in the background. Later the qt app may request a status or list of running jobs.

As for using requests, I looked at replacing it but the existing codebase uses 'requests' extensively and I also need to maintain python 2 compatibility for that part of the codebase. I was also unclear on how I call the event loop executer with my long_download function.

Thanks again for your help and patience.

- dharhas

--
You received this message because you are subscribed to a topic in the Google Groups "python-pulsar" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-pulsar/TuctWsfZ3qw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to python-pulsa...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[lsbardel]

On Monday, September 21, 2015 at 1:36:13 PM UTC+1, Dharhas Pothina wrote:

Hi Isbardel,

Thanks for helping out. I'm new to async type programming and I may be trying to do the wrong thing. My python api will eventually be called from a C++ qt application so I've been testing using curl, i.e.:

curl -H "Content-Type: application/json" -X POST -d '{"jsonrpc": "2.0", "method": "long_download", "params": [0], "id": 0}' 127.0.0.1:8060

this does seem to block when I try it. Or at least if I get a delay of 5 secs, I have to wait 5 seconds before the curl command completes.

You are confusing client and server here.

The python server, written with pulsar, is asynchronous, in the sense that it continues to serve requests even if other long running requests as not yet finished. This is what coroutines are for.

However, the client (curl in your case) is not asynchronous, and it will block until the request is returned, which is when the coroutine of your long running function finish.

 

What I need is for the long_download function to do is return control immediately along with a 'operation started' message and then continue to conduct the download in the background. Later the qt app may request a status or list of running jobs.

If you want pulsar to return the response immediately, run the asynchronous function in a asyncio task and return. In this way your client will get a response immediately, however you don't know if the asynchronous long function was successful or not.

As for using requests, I looked at replacing it but the existing codebase uses 'requests' extensively and I also need to maintain python 2 compatibility for that part of the codebase. I was also unclear on how I call the event loop executer with my long_download function.

Pulsar works for python 3.4+ only. Python 2 is not supported.

PS: Please try to avoid top-posting :) https://en.wikipedia.org/wiki/Posting_style#Trimming_and_reformatting

[Dharhas Pothina]

You are confusing client and server here.

The python server, written with pulsar, is asynchronous, in the sense that it continues to serve requests even if other long running requests as not yet finished. This is what coroutines are for.

However, the client (curl in your case) is not asynchronous, and it will block until the request is returned, which is when the coroutine of your long running function finish.

makes sense thanks.

 

 

What I need is for the long_download function to do is return control immediately along with a 'operation started' message and then continue to conduct the download in the background. Later the qt app may request a status or list of running jobs.

If you want pulsar to return the response immediately, run the asynchronous function in a asyncio task and return. In this way your client will get a response immediately, however you don't know if the asynchronous long function was successful or not.

I'll try that. Thanks.

Pulsar works for python 3.4+ only. Python 2 is not supported.

 

I realize that, the async stuff is all going to be py 3.4+ only but other parts of the codebase will be py 2.7 compatible.

PS: Please try to avoid top-posting :) https://en.wikipedia.org/wiki/Posting_style#Trimming_and_reformatting

Apologies.

Thanks for the help.

 

- dharhas