ZeroRPC with Asynchronous IO?

[Eric Mangold]

Hi,

I'm searching for an RPC solution to run on top of ZeroMQ.

Requirements are that we be able to program our rpc clients and responders/handlers in an asynchronous fashion. Preferably using Twisted/txZMQ.

Is the zerorpc protocol implementation exposed in a well-factored way that isn't tied to pyzmq or a particular transport? Such that we could just hook it up, or copy the code in to a Twisted Protocol? (Twisted has very good separation between Protocol (what you do with bytes) and Transport (how you get bytes from A to B and vice versa).

I also need a client implementation of the protocol for ActionScript 3... any ideas? There is already ZeroMQ support for AS3, but I haven't seen anything for ZeroRPC...

I may just write my own protocol using msgpack on top of zeromq... if this isn't really well supported by zerorpc already...

Any tips, pointers, doc, examples, etc are GREATLY appreciated!

Thanks a ton,
-E

[François-Xavier Bourlet]

Quick response. zerorpc python cannot be used without gevent, and the code is written in blocking style (made async thanks to coroutines), you would need to rewrite it from scratch to support an event/callback based style.

And there is no action script implementation that I know of.

I have been working on and off on solving this problem by rewriting the core of zerorpc in its own library (exposing a C API), leaving in the language bindings the work to integrate the language, lib and framework specific parts. But this is never going to happen soon, as I was experimenting with different ideas and design.

zerorpc bring few things atop zmq + msgpack:
- a natural rpc API, almost like there no (zero) RPC
- multiplexing atop a single zmq (usually tcp) connection, similar to http2.
- exception/error reporting
- heartbeat and timeouts
- streaming responses
- convenient CLI tool

Unless you want to implement the zerorpc protocol and/or need to be compatible with other zerorpc implementation (nodejs, php, Python), you might be better off writing your own small lib tailored to your need.

If you only need simple request / response you can easily follow the basics of the zerorpc protocol (no heartbeat, no streaming), similar to the current php implementation. And only when you need later, you can add heartbeat and streaming.

I can definitively describe and assist you with the zerorpc protocol.

Best,
fx

--
You received this message because you are subscribed to the Google Groups "zerorpc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zerorpc+u...@googlegroups.com.
To post to this group, send email to zer...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/zerorpc/49ccb898-b92f-4841-bff0-812ee7310cc5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Eric Mangold]

Thanks for the quick reply... comments inline below...

On Monday, March 28, 2016 at 11:56:07 AM UTC-6, François-Xavier Bourlet wrote:

Quick response. zerorpc python cannot be used without gevent, and the code is written in blocking style (made async thanks to coroutines), you would need to rewrite it from scratch to support an event/callback based style.

OK. A bit unfortunate as you allude to later... Twisted does have a generator-based style of implementing asynchronous code if you prefer to use it (see @inlineCallbacks). Python does not have actual co-routines outside of Stackless AFAIK. So I assume you are referring to generators.
 

And there is no action script implementation that I know of.

OK.. too bad, but not unexpected. zerorpc seems to be quite a simple protocol, similar in some ways to AMP (Asynchronous Message Protocol, https://web.archive.org/web/20150707082739/http://amp-protocol.net/ - NEW website maintainer is forthcoming with a new site).

I have been working on and off on solving this problem by rewriting the core of zerorpc in its own library (exposing a C API), leaving in the language bindings the work to integrate the language, lib and framework specific parts. But this is never going to happen soon, as I was experimenting with different ideas and design.

OK cool! What language have you settled on to implement your C-ABI library in? Please say Rust.
 

zerorpc bring few things atop zmq + msgpack:
- a natural rpc API, almost like there no (zero) RPC
- multiplexing atop a single zmq (usually tcp) connection, similar to http2.
- exception/error reporting
- heartbeat and timeouts
- streaming responses
- convenient CLI tool

I assume of course that zerorpc does NOT enforce synchronous request-reply on the wire? i.e. you can fire off a request, receives OTHER replies, fire off OTHER requests, HANDLE requests on your end, before you get the original REPLY back?
 

Unless you want to implement the zerorpc protocol and/or need to be compatible with other zerorpc implementation (nodejs, php, Python), you might be better off writing your own small lib tailored to your need.

I will probably implement zeroRPC or AMP. Or I might implement AMP but using msgpack instead of the normal AMP encoding (which would not technically be AMP, but just a clone using msgpack).
 

If you only need simple request / response you can easily follow the basics of the zerorpc protocol (no heartbeat, no streaming), similar to the current php implementation. And only when you need later, you can add heartbeat and streaming.

Don't need heartbeat or streaming ATM...
 

I can definitively describe and assist you with the zerorpc protocol.

Awesome! Isn't the proto documented well though?
 

Best,
fx

Cheers,
-E
 

[François-Xavier Bourlet]

answers inline.

On Mon, Mar 28, 2016 at 3:49 PM Eric Mangold <tera...@gmail.com> wrote:

Thanks for the quick reply... comments inline below...

On Monday, March 28, 2016 at 11:56:07 AM UTC-6, François-Xavier Bourlet wrote:

Quick response. zerorpc python cannot be used without gevent, and the code is written in blocking style (made async thanks to coroutines), you would need to rewrite it from scratch to support an event/callback based style.

OK. A bit unfortunate as you allude to later... Twisted does have a generator-based style of implementing asynchronous code if you prefer to use it (see @inlineCallbacks). Python does not have actual co-routines outside of Stackless AFAIK. So I assume you are referring to generators.

 

zerorpc-python is based on gevent, so I was talked about gevent coroutines (which are called greenlets I think). If you can integrate gevent in what you are doing then you can use the current version of zerorpc-python.

 

And there is no action script implementation that I know of.

OK.. too bad, but not unexpected. zerorpc seems to be quite a simple protocol, similar in some ways to AMP (Asynchronous Message Protocol, https://web.archive.org/web/20150707082739/http://amp-protocol.net/ - NEW website maintainer is forthcoming with a new site).

Protocol is even simpler than that really, since everything is encoded via msgpack. 

I have been working on and off on solving this problem by rewriting the core of zerorpc in its own library (exposing a C API), leaving in the language bindings the work to integrate the language, lib and framework specific parts. But this is never going to happen soon, as I was experimenting with different ideas and design.

OK cool! What language have you settled on to implement your C-ABI library in? Please say Rust.

I first wrote a prototype in C++ when Rust was not yet mature enough and then wrote a second prototype in Rust when it started becoming more mainstream. I do indeed want to implement this library in Rust. I would be crazy not using it!

 

zerorpc bring few things atop zmq + msgpack:
- a natural rpc API, almost like there no (zero) RPC
- multiplexing atop a single zmq (usually tcp) connection, similar to http2.
- exception/error reporting
- heartbeat and timeouts
- streaming responses
- convenient CLI tool

I assume of course that zerorpc does NOT enforce synchronous request-reply on the wire? i.e. you can fire off a request, receives OTHER replies, fire off OTHER requests, HANDLE requests on your end, before you get the original REPLY back?

Correct. What zerorpc calls channels are similar to what http2 calls stream. Same concept really.

 

Unless you want to implement the zerorpc protocol and/or need to be compatible with other zerorpc implementation (nodejs, php, Python), you might be better off writing your own small lib tailored to your need.

I will probably implement zeroRPC or AMP. Or I might implement AMP but using msgpack instead of the normal AMP encoding (which would not technically be AMP, but just a clone using msgpack).
 

If you only need simple request / response you can easily follow the basics of the zerorpc protocol (no heartbeat, no streaming), similar to the current php implementation. And only when you need later, you can add heartbeat and streaming.

Don't need heartbeat or streaming ATM...
 

I can definitively describe and assist you with the zerorpc protocol.

Awesome! Isn't the proto documented well though?

Mmm well the current protocol doc is awful, but at least is not wrong. Without heartbeat nor streaming, the protocol is really really simple and can be explained quickly with few rules:

Only a client can initiate a request to the server (hence why its called a client and the other, the server).

Requests and responses are sent via a zmq message composed of a single part (zmq messages can have multiple parts).

The client uses a zmq DEALER socket, the server uses a ROUTER socket (so you can send/recv messages asyncrounosuly).

The zmq message contains a msgpack array: [header, action, arguments]

The header is a dict of "string" -> value. It must contains at least:

{ "message_id": "<UNIQUE_ID>", "v": 3 }

The message_id is a random string, unique to a specific request.

A response header will also reference the original request message id in the response_to field:

{ "message_id": "<UNIQUE_ID>", "response_to": "<REQUEST_ID>" }

For a request, action will be the remote function to call.

For a response it will be either the string "REP" or "ERR".

For a request, arguments will be an array of arguments to the function (empty if there is no arguments of course).

For a successful response (action=="REP"), arguments will be the result from the function (like an int, or a string, of an array for multiple returns etc).

For an error response (action=="ERR"). arguments will be an array describing the error [error_name, human_msg, human_traceback].

The error_name could be something like "InvalidArgument", the human_msg is could be printed to the user, and the traceback would be the language traceback on the remote. human_msg and human_traceback can be empty strings.

Feel free to be compatible or not! This is surely not the prettiest protocol ;)

best!

 

Best,
fx

Cheers,
-E
 

On Mon, Mar 28, 2016, 10:20 Eric Mangold <tera...@gmail.com> wrote:

Hi,

I'm searching for an RPC solution to run on top of ZeroMQ.

Requirements are that we be able to program our rpc clients and responders/handlers in an asynchronous fashion. Preferably using Twisted/txZMQ.

Is the zerorpc protocol implementation exposed in a well-factored way that isn't tied to pyzmq or a particular transport? Such that we could just hook it up, or copy the code in to a Twisted Protocol? (Twisted has very good separation between Protocol (what you do with bytes) and Transport (how you get bytes from A to B and vice versa).

I also need a client implementation of the protocol for ActionScript 3... any ideas? There is already ZeroMQ support for AS3, but I haven't seen anything for ZeroRPC...

I may just write my own protocol using msgpack on top of zeromq... if this isn't really well supported by zerorpc already...

Any tips, pointers, doc, examples, etc are GREATLY appreciated!

Thanks a ton,
-E

--
You received this message because you are subscribed to the Google Groups "zerorpc" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zerorpc+u...@googlegroups.com.
To post to this group, send email to zer...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/zerorpc/49ccb898-b92f-4841-bff0-812ee7310cc5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "zerorpc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zerorpc+u...@googlegroups.com.
To post to this group, send email to zer...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/zerorpc/e6d0898e-a661-4468-97f4-c34758ddfe49%40googlegroups.com.