Blocking Issues?

[Landon]

Hey Josh,

We are running into an issue with tornadoRPC:


we are trying to fetch an object from server A, and during that
function call, server A will need to make another RPC call to server B
to fetch some additional data before returning to the process in
server A and processing the additional details.

some psudocode:

def getA(self, var):
ServerB = rpcclient('http://xxx.xxx.xxx.xxx:7007')
##Nothing happens after this point.
mylistb = ServerB.getList(other_var)
return mylistb


on server B

def getList(self, other_var):
mylist = [a,b,d,f,e]
return mylist

The flow in this case starts on server A, which opens the connection
to server B, retreives the data, and then returns the data from A to
the originating request.


For the life of us, we cannot figure out why this will not work. We
have concluded at this point that we cannot run an RPC request while
another RPC request from the same server is already running.

We are able to perform these calls from the python interpreter from
any server in our network without issue- however when we request them
from tornado we see the process time out when the request-within-a-
request is executed. Stopping the tornado daemon process will result
in the following error:
-----------------------------------------------------------------------------
ERROR IN getA - (PARAMS: ['879253fe01bbe1ec981f2fadd0816389'])
-----------------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python2.6/socket.py", line 397, in readline
data = recv(1)
error: [Errno 104] Connection reset by peer


We are still confused with your Async example- we are very interested
in making this work, but at this point we have hit a dead end and are
reaching out for help as a last resort.

Thanks,

-Landon

[Josh Marshall]

Hey Landon --

Sorry about the delay in response, I was on Christmas vacation.

I was able to call an external tornadorpc server within a request without a problem, however I had two separate servers running -- what you are describing sounds to me like either a request that just takes too long for the default socket timeout, or you are blocking yourself by calling the same server that the request is coming from. Due to Tornado's asynchronous nature, a request that calls the same server will infintely block unless you use an asynchronous library like AsyncHTTPClient -- TornadoRPC simply inherits this behavior. 

To show this, you can use the attached script -- if you run just one server on the default port and call server.external() on it, it will block forever. However, if you run one server on the default port and a second server on port 8082 (for example), and call server2.external(), it will work fine and return 11.

Right now, the @async decorator is how you tell Tornado that you will be using an asynchronous callback to finish the request, and not to complete the request until you call self.result(). In a normal Tornado asynchronous request, you would put @asynchronous over the get() (or post, etc) method. Then, in the request, you would use an asynchronous library (like AsyncHTTPClient) with a callback. Tornado's normal behavior is to call self.finish() when a request method returns, but the @asynchronous decorator tells Tornado that you will call self.finish() yourself in the final callback function, so leave the request open. 

The TornadoRPC library uses this same idea, but instead of caling self.finish(), you are returning the result with self.result(). However, you still have to provide the actual asynchronous library (or use one of the built-in Tornado ones). One of the things on my list is to make an asynchronous RPC client based on Tornado's AsyncHTTPClient and include it in tornadorpc, which would handle this for you, but I haven't had time yet to incorporate this. When I get that finished, you will definitely want to change your xmlrpclib.Server calls to that library, because a long-running external call using the xmlrpclib client will actually block any other requests on that Tornado instance (as it sounds like you are discovering)

I hope I understood the issue properly and that this helps -- let me know if I missed the boat or you have any other questions (I will respond much more quickly now that I am back from my vacation. :))

Josh Marshall