Transferring numpy arrays

[DaveO]

I've got a need to transfer numpy arrays over the wire. What's the best practice for using zerorpc with custom objects?

I looked through the msgpack docs and see hints of being able to define custom serializations, but saw no examples. I also poked around in zerorpc's source to try and find workable points but I didn't find anything that wouldn't be a giant hack. I could also simply handle serialization myself with wrapper functions on both sides but that, to me, defeats the purpose of using zerorpc. I've already got a working solution with string based marshaling, and a primitive client/server interface with pyzmq. I was hoping to nuke the whole code-base with zerorpc but numpy arrays are giving me trouble. FWIW I'm using non-copy sends as detailed here (http://minrk.github.com/scipy-tutorial-2011/zeromq.html#non-copying-sends).

Thanks,
Dave

[Francois-Xavier Bourlet]

Hey Dave,

ZeroRPC transfer all JSON datatypes only.
This is what power ZeroRPC ability to be so interoperable between languages.
It is also the insurance that you can always deserialize anything that
you transfer between a ZeroRPC client & servers.

Technically, JSON datatypes can represent 99% of the real world
structures. After all, mostly everything can fit in
array/dictionary/string/integer & floating point numbers.

You could convert your datatypes into a generic representation,
limited to JSON supported datatypes. It then obviously works with
ZeroRPC and more generally it will works with any technology (web
browser?) that understand JSON. It also means that your server and
client are not coupled around specifics datatypes anymore.

If this is not what you want, you can:
- use something like protobuf or thrift -> these tools let you
specify exactly what kind of datatype you want to transfer, in the
expense of having to share the datatypes representation between the
client and the server.
- pickle your datatype and send it like a blob over the network (via
ZeroRPC or anything else). Since its something like transferring
python bytecode, its not the best you can do in term of remote code
execution safety. It is also no portable among languages and probably
not really well between different version of python.

I am sure there is other ways, but I believe that none will be as
flexible as using JSON datatypes as the common medium for your data.

Regards,
fx

> --
> You received this message because you are subscribed to the Google Groups
> "zerorpc" group.
> To post to this group, send email to zer...@googlegroups.com.
> To unsubscribe from this group, send email to
> zerorpc+u...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/zerorpc/-/1FnHmbd4Dt8J.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>



--
François-Xavier Bourlet