ZeroMQ for RPC

1,004 views
Skip to first unread message

Marcus Ottosson

unread,
Mar 17, 2014, 12:13:28 PM3/17/14
to python_in...@googlegroups.com
This is especially targeted towards you, Justin, as I just found this..

..as I was looking around for inspiration on how to implement the very same thing!

How was your experience with 0mq for RPC? Are you still using this implementation or any like it? Are you still using 0mq? If so, what for?

For everyone else, are you using 0mq? What for, and what have your experiences been with it so far?

Best,
Marcus

Justin Israel

unread,
Mar 17, 2014, 2:33:33 PM3/17/14
to python_in...@googlegroups.com

I threw that code together when I was first learning ZeroMQ and wanted to play with the idea of making it a simple RPC mechanism. Turned out some guys at RedHat were using it and asked for some more thing, so I expanded it a bit more.

I really like ZeroMQ. Its a lot of lower level elements that allow you to put together the patterns you need. While I didn't use pyRpc very much, I did use ZeroMQ for other things like pubsub. I think its really strong for a number of use cases, with RPC just being one of them. Its good for when you want to seamlessly scale work from interthread to interprocess to local network to even remote network situations, since all that changes is the socket types and devices you use to connect them.

Its just a really flexible library for communication, messaging, and concurrent processing.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/61159046-f043-4a56-b2a5-a67ef77fc178%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Marcus Ottosson

unread,
Mar 17, 2014, 3:11:36 PM3/17/14
to python_in...@googlegroups.com
That pretty much sums up my impression of 0mq as well (having used it for about a week).

About pyRpc, what inspired the use of publishing methods towards a server? Seems a common pattern in RPC implementations, perhaps that's what RPC is all about? RPyC (my only other experience) uses the same concept, except it publishes them to a separate class (a "Service") that gets passed into the server upon instantiation. What are you using for RPC currently?



For more options, visit https://groups.google.com/d/optout.



--
Marcus Ottosson
konstr...@gmail.com

Justin Israel

unread,
Mar 17, 2014, 3:33:12 PM3/17/14
to python_in...@googlegroups.com

I'm not sure what inspired me specifically to do that. It just seemed appropriate to make it easy to export functions as services and then make them discoverable when queried. I figured it was useful to be able to have first class registration of what is available remotely. I've actually never used RPyc yet.

Currently for my RPC solutions I am working with Apache Thrift. It allows you to generate bindings for all the modern languages so you can have a server in one Lang and 5 clients each in different langs. It handled all of the network aspects and wire formats.

Marcus Ottosson

unread,
Mar 17, 2014, 5:08:25 PM3/17/14
to python_in...@googlegroups.com
Ah, interesting. I've only looked briefly at Thrift.

Would you recommend Thrift for inter-thread/process communication, as well as for communicating across languages? What other languages do you find it useful to bridge with RPC at the moment, and for what?



For more options, visit https://groups.google.com/d/optout.



--
Marcus Ottosson
konstr...@gmail.com

Justin Israel

unread,
Mar 17, 2014, 8:15:23 PM3/17/14
to python_in...@googlegroups.com
I've never tried to use it for inter-thread/process communication, but I assume it would work. I just don't know why kind of overhead it would be in comparison with using ZeroMQ for the same solution. ZeroMQ seems a bit lighter-weight but I guess it depends on exactly what you are doing. 

I've been using it in situations where our application server is written in Java, and we need python, C++, and jquery clients to access it. It makes it really nice to have a consistent API between any of the languages that is auto-generated for you. Thrift takes care of type checking, serializing/deserialzing, sending/receiving. 


Marcus Ottosson

unread,
Mar 18, 2014, 3:15:00 AM3/18/14
to python_in...@googlegroups.com
My hope is to find one method of communicating across any process, both local and remote, but I'm not sure yet that it's feasible. Performance isn't an issue yet - its mainly calling os.listdir or maya.cmds.ls at the moment - but I suspect it may become in the future - producing logs, signalling subscribers of events. ZMQ seems to do it all but I've been spoiled by the simplicity of RPyC and can't bring myself to re-implement the benefits of using it in ZMQ.

Thrift does sound interesting, but it also seems perpendicular to what ZMQ does, although ZMQ seems rather closer to the metal and would need more work to get going.

Having worked with ZMQ before, what made you switch to Thrift?



For more options, visit https://groups.google.com/d/optout.



--
Marcus Ottosson
konstr...@gmail.com

Justin Israel

unread,
Mar 18, 2014, 4:04:46 AM3/18/14
to python_in...@googlegroups.com
ZeroMQ has concepts that Thrift doesn't have, and likewise. Thrift has no concept of events via publishing-subscriber patterns, or streaming. It is purely RPC, which would be equivalent to the req-rep pattern in ZeroMQ. And ZeroMQ doesn't have all extra layers that actually make it an RPC system. You would have to roll that functionality all yourself. But the thing ZeroMQ and Thrift both give you over RPyc is language-agnostic communication. 

The really attractive thing with Thrift is again that IDL language that generates a contract for your RPC interfaces. It handles all of the validation for you before it goes onto the wire, and does stuff like Exception raising, and also a versioned approach to upgrading your API without breaking older clients. If what you need is the event stuff with subscribers, I don't think you will get that with Thrift. But if you really do want RPC, then Thrift does most of the work for you. Just right a .thrift IDL file, use the thrift tool to generate your bindings, and implement the server side of the code. You can then either use the client API raw, or write a thin wrapper over it. 


Marcus Ottosson

unread,
Mar 18, 2014, 4:31:41 AM3/18/14
to python_in...@googlegroups.com
Exactly, the language-agnosticism is the only reason I'm switching from RPyC to something else.

Have you had a look at Google Protocol Buffers? It seems similar to the contract you speak of. You write a generic cross-language file, parse it to your language, in this case Python, and use it for sending and receiving messages.

It was really great, but I couldn't get comfortable with the fact that it couldn't deal with arbitrary data-types. E.g. a response with a "message" attribute of type string, would have to return string and nothing else. Great in cases where the request and response are known, but for RPC, where some requests might return a list and others a bool.. I got into trouble.

Does that sound similar to IDL and if so how does it deal with these cases?



For more options, visit https://groups.google.com/d/optout.



--
Marcus Ottosson
konstr...@gmail.com

Justin Israel

unread,
Mar 18, 2014, 6:43:55 AM3/18/14
to python_in...@googlegroups.com

Protocol buffer is similar for part of what Thrift does but only covers the serialization spec. It doesn't give you the RPC mechanisms. You still have to handle that layer yourself. Its just a versioning serialization/wire protocol.
But yes you would have to say what type a service will return to allow you to fulfil the ability to generate code for static typed languages. Also it is needed to guarantee the versionable wire spec since it has to be able to encode the right sizes for types. But then you get all the free type validation. Thrift doesn't even put it on the wire if you construct things wrong.

RPyc makes perfect sense if you are dealing with 100% python systems as it probably strives to remain pythonically flexible. Thrift has to work in a way that the IDL can generate code for different languages. I suggest you try it out and see what you think.

Marcus Ottosson

unread,
Mar 18, 2014, 9:04:47 AM3/18/14
to python_in...@googlegroups.com
Thanks Justin, things are a lot clearer now.



For more options, visit https://groups.google.com/d/optout.



--
Marcus Ottosson
konstr...@gmail.com

Justin Israel

unread,
Mar 19, 2014, 2:13:50 AM3/19/14
to python_in...@googlegroups.com

Report back with your experience with Thrift, since you are able to make comparisons to RPyc. Would be interested to know your impression.

Marcus Ottosson

unread,
Mar 19, 2014, 3:47:09 AM3/19/14
to python_in...@googlegroups.com
I'm giving ZMQ a shot first, I'd much prefer the use of only one network library as opposed to several. But I'll post back here if and when I meet up with Thrift.



For more options, visit https://groups.google.com/d/optout.



--
Marcus Ottosson
konstr...@gmail.com

Marcus Ottosson

unread,
Mar 24, 2014, 9:10:03 AM3/24/14
to python_in...@googlegroups.com
Had some issues getting PyZMQ running in Maya and made a note of the process for future and others reference here
--
Marcus Ottosson
konstr...@gmail.com

Justin Israel

unread,
Mar 24, 2014, 2:38:50 PM3/24/14
to python_in...@googlegroups.com

Oh Windows, why are you so complicated? :-)

Can you explain this part?
" PyZMQ come with a Python extension for Python 2.7, but is compiled with MSC v.1500 which means Maya's Python won't be able to work with it"

Does this mean you were using a binary distribution of PyZMQ specifically for Python2.7? The actual source should be a cython-based extension that supports >= 2.6

Marcus Ottosson

unread,
Mar 24, 2014, 4:22:15 PM3/24/14
to python_in...@googlegroups.com

Oh Windows, why are you so complicated? :-)

It's cross-platform but alas Windows had to tag along. :(

Can you explain this part?
" PyZMQ come with a Python extension for Python 2.7, but is compiled with MSC v.1500 which means Maya's Python won't be able to work with it"

Does this mean you were using a binary distribution of PyZMQ specifically for Python2.7? The actual source should be a cython-based extension that supports >= 2.6

Do feel free to correct me, but here is my understanding. PyZMQ, being a mere binding to a C++ library, ships with a binary, a Python extension, compiled using VS2008.

Loading this extension made Maya (who is on the VS2010 side of the fence) blurt out that it was missing VS2008 dependencies. So my reaction was to compile the bindings and the the extension with VS2010.

The source does indeed support 2.6, both VS2008 and VS2010 versions, and PyZMQ apparently prefers to use binaries from the pure ZMQ installation, which does come with a VS2010 version, this is fine. But as I'm looking to bundle PyZMQ with my repository, I didn't see any other immediate solutions than to bundle the python extension rather than the .dlls and that wasn't included in either the ZMQ nor the PyZMQ installations.

This is all new to me, so if you can think of a better way I'm all ears. Again, the goal is to bundle ZMQ with a cross-platform repository which means I'm including the extensions for both Nt and Posix systems.




For more options, visit https://groups.google.com/d/optout.



--
Marcus Ottosson
konstr...@gmail.com

Justin Israel

unread,
Mar 24, 2014, 5:05:08 PM3/24/14
to python_in...@googlegroups.com
PyZMQ uses cython to create bindings against ZeroMQ, which is written in C. 
I think the issue is that you were grabbing the prebuilt Windows Installers. Otherwise, the source should not be including any binary files. It would be those windows installers that are delivering you precompiled binaries.

If you build ZeroMQ from source, and build PyZMQ from source, you should end up with binaries that are specific to your target environment. You are right that it is pretty similar to the PyQt/PySide workflow, where you can't rely on grabbing standard binary builds, since they will probably conflict with Maya. You kinda have to build them from source. 




Marcus Ottosson

unread,
Mar 25, 2014, 2:35:37 AM3/25/14
to python_in...@googlegroups.com
That makes sense, Justin, but I'm still confused.

Does that mean that binaries compiled in VS2008 should work with a Python compiled in VS2010? Isn't there some binary compatibility issues with that?



For more options, visit https://groups.google.com/d/optout.



--
Marcus Ottosson
konstr...@gmail.com

Justin Israel

unread,
Mar 25, 2014, 2:46:34 AM3/25/14
to python_in...@googlegroups.com
No, as far as I know you are totally right about that. But what I am suggesting is that if you *build* from source against your target (Maya), then it *will* be binary compatible. This would be instead of relying on any binary distributions that you download, which were already compiled under a possibly incompatible environment. 

I was just asking about whether you encountered problem based on downloading and using binary distributions, instead of building the source against Maya.


Marcus Ottosson

unread,
Mar 25, 2014, 3:19:28 AM3/25/14
to python_in...@googlegroups.com
I see. It was a downloaded binary distribution. I've never encountered the problem before and assumed that the only requirement from a compatibility perspective was that it would have to be Windows, it would have to be x64 and it would have to be version 2.7 of Python, but it turned out there was a fourth requirement (and possibly more).

Anyway, thanks for helping me sort things out. I'll update the post to reflect this.



For more options, visit https://groups.google.com/d/optout.



--
Marcus Ottosson
konstr...@gmail.com

Asi Sudai

unread,
Jan 11, 2015, 7:01:28 PM1/11/15
to python_in...@googlegroups.com
Hey Marcus!

I wander if you found a way to get zmq working inside Maya (Windows and/or Linux)?
running into the same issue now too :)
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.



--
Marcus Ottosson
konstr...@gmail.com

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.



--
Marcus Ottosson
konstr...@gmail.com

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.



--
Marcus Ottosson
konstr...@gmail.com

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.



--
Marcus Ottosson
konstr...@gmail.com

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.



--
Marcus Ottosson
konstr...@gmail.com

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.



--
Marcus Ottosson
konstr...@gmail.com




--
Marcus Ottosson
konstr...@gmail.com

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.



--
Marcus Ottosson
konstr...@gmail.com

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.



--
Marcus Ottosson
konstr...@gmail.com

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_maya+unsub...@googlegroups.com.



--
Marcus Ottosson
konstr...@gmail.com
...

Marcus Ottosson

unread,
Jan 12, 2015, 4:06:25 AM1/12/15
to python_in...@googlegroups.com

Hey Asi, what problems were you running into?

Here's what I did to get it up and running:
http://abstractfactory.io/blog/building-pyzmq-for-python-2-7-msc-v-1600/

​

Asi Sudai

unread,
Jan 15, 2015, 12:24:39 PM1/15/15
to python_in...@googlegroups.com
You're amazing! thanks Marcus!

LIJU kunnummal

unread,
Jan 15, 2015, 6:06:30 PM1/15/15
to python_in...@googlegroups.com
This is indeed one of the most informative thread, thanks for both Marcus and Justin for sharing your thoughts.

Cheers
Ljk

LIJU kunnummal

unread,
Jan 16, 2015, 10:03:02 AM1/16/15
to python_in...@googlegroups.com
You guys might also be interested to have a look at Google's Protocol buffers, seems to be similar to Thrift.

Cheers
Liju K



On Monday, 17 March 2014 16:13:28 UTC, Marcus Ottosson wrote:

Justin Israel

unread,
Jan 16, 2015, 2:27:08 PM1/16/15
to python_in...@googlegroups.com
Protocol Buffers are just the serialization format. They don't contain an RPC spec. Thrift is more focused on pluggable serialization formats, as well as pluggable transports, and directly supporting all of the RPC/network stuff. If you need RPC and don't want to handle all of that layer yourself, the Thrift may be a better choice than protobufs. Also, the guy who wrote protobufs actually wrote a new product called Cap'n Proto, which has all of the extra RPC smarts.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/815da1d6-9ef8-47df-81b4-8f9386bfa60f%40googlegroups.com.

Marcus Ottosson

unread,
Jan 19, 2015, 3:37:18 AM1/19/15
to python_in...@googlegroups.com
Cap'n Proto has a really great website. :)​ Thanks for sharing.

LIJU kunnummal

unread,
Jan 19, 2015, 5:38:14 PM1/19/15
to python_in...@googlegroups.com
It's worth looking..thanks Justin for sharing.

Regards
Liju
Reply all
Reply to author
Forward
0 new messages