Usage best practices to eliminate LostRemote exceptions

[Harel Malka]

Hi there, 

I've started implementing zerorpc into a set of Django based microservices. 

A simple usage scenario I have in one service is to provide a 'list' and 'get' rpc calls to get many or one model data. 

The 'list' method yields (stream) each model as a dict, while the 'get' method simply returns the model as a dict by id. 

All is fine, except that I'm getting fairly random "LostRemote: Lost remote after 10s heartbeat" exceptions in my local tests.

I am instantiating a client:

client = zerorpc.Client()
client.connect('tcp://127.0.0.1:4242')

And then issuing (in the shell) a series of 'gets' and lists:

> many = client.get_many()
> next(records)
> {'record':   12345}
> next(records)
> {'record':   98765}
> client.get_one(123)
> {'record': 123}
> client.get_one(123)
> {'record': 123}
> client.get_one(123)
> {'record': 123}
> LostRemote: Lost remote after 10s heartbeat
> client.get_one(123)
> {'record': 123}

At this point, after i fire a few of those calls, even without any break between them, I will get a LostRemote exception thrown. This seems to be pretty random. Even if a request is made every half a second I will still get that raised.

My question is then, what is the best practice to AVOID those? I don't want to lose requests because of that. 

Do I need to instantiate a new client before each call?

How can  I debug this? Is there documentation about all the arguments to the server and client? For example how to run in a debug mode? 

Do i need to manage those failures myself?

Thanks

Harel

[François-Xavier Bourlet]

Hello Harel,

zerorpc python required gevent to run. gevent is an ioloop and a coroutine implementation. all your coroutines are running cooperatively in the same main python thread. You can never block any thread else zeroropc wont be able to process hearbeat messages. If you are using django without any special setup, it will run every request one by one, fully blocking the main thread. Which is likely your problem. You have two solutions: 1) disable heartbeat (on the server and client, heartbeat=None as a parameter) 2) Or use something like gunicorn to run django with gevent.

Disabling heartbeat is the least favorite solution, as you will not catch network error as quickly.

Best,

fx

--
You received this message because you are subscribed to the Google Groups "zerorpc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zerorpc+u...@googlegroups.com.
To post to this group, send email to zer...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/zerorpc/607c9950-0dc1-4db3-9b01-607c546309bf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Harel Malka]

I ended up disabling heartbeat for now. Although the app does run in a Django context, these calls were made on the shell (but does call up some django models).

I'll look into gunicorn/gevent for Django, though really in my use case dropping the heartbeat and managing process state in supervisor also seems like a good enough idea. 

Thanks

Harel