simple and fast platform independent IPC
News123
I wondered what IPC library might be best simplest for following task?
I'm having a few python scripts all running on the same host (linux or
win), which are started manually in random order. (no common parent process)
Each process might be identified by an integer (1,2,3) or by a symbolic
name ( 'dad' , 'mom' , 'dog' )
these scripts want to send short messages to each other ( mostly
integers, max a few bytes, short string), which would be enqueued in
message queues of the receiving process.
example:
'dad' wants to tell 'mom': 'cook'
'dog' wants to tell 'dad' : 'feedme'
'mom' wants to tell 'dad' : 'cookyourself'
the receiver dos not necesarily have to know who sent the message
a message shall be dropped silently if the receiving process is not running
a message shall be reliably delivered if the receiving process is up
xmlrpc seems to be a little heavy for such tasks.
signals don't allow to exchange data
a shared memory message queue would probably a good solution, but
python's Multiprocessing.Queue seems to require a common parent process
thanks a lot for any ideas / suggestions
N
N
Gabriel Genellina
> I'm having a few python scripts all running on the same host (linux or
> win), which are started manually in random order. (no common parent
> process)
> Each process might be identified by an integer (1,2,3) or by a symbolic
> name ( 'dad' , 'mom' , 'dog' )
>
> these scripts want to send short messages to each other ( mostly
> integers, max a few bytes, short string), which would be enqueued in
> message queues of the receiving process.
>
> example:
>
> 'dad' wants to tell 'mom': 'cook'
> 'dog' wants to tell 'dad' : 'feedme'
> 'mom' wants to tell 'dad' : 'cookyourself'
Try using a named pipe between each pair of processes (os.mkfifo + open on
Linux, open(r"\\.\pipe\desired_name", ...) on Windows)
--
Gabriel Genellina
Vinay Sajip
Gabriel's suggestion is very good; if you need something which is a
little more like RPC but still quite lightweight, consider Pyro
(http://pyro.sourceforge.net/)
Regards,
Vinay Sajip
Joan Miller
I've read that Pyro is not safe. Anyway, you have in mind that respect
to speed:
shared memory > named pipes > Unix domain socket > TCP socket
I don't sure about if the message queues would be faster that Unix
domain sockets
Another thing. Using shared memory would be as to use a single thread
but using message queues would be as multiple-threading.
News123
I'll look at it.
I wasn't aware about named pipes for windows.
bye
N
Tim Golden
>>> I wondered what IPC library might be best simplest for following task?
...
>>> xmlrpc seems to be a little heavy for such tasks.
>>
>>> signals don't allow to exchange data
>>
>>> a shared memory message queue would probably a good solution, but
>>> python's Multiprocessing.Queue seems to require a common parent process
[Vinay Sajip]
>> Gabriel's suggestion is very good; if you need something which is a
>> little more like RPC but still quite lightweight, consider Pyro
>> (http://pyro.sourceforge.net/)
> I've read that Pyro is not safe.
That's a fairly broad thing to say. I've read lots
of things. What does "is not safe" mean, in any case?
I assume you've got a valid concern in mind which is
worth passing on to a would-be user, but what exactly
is it? FWIW I've used Pyro on and off over the years
without any problems. Certainly my computer's never
blown up as a result of using it.
Obviously Pyro is Python-only so interaction with non-Python
code would be problematic. But the OP only mentions Python
scripts so hopefully that wouldn't be an issue...
>Anyway, you have in mind that respect to speed:
>
> shared memory> named pipes> Unix domain socket> TCP socket
True, but the OP didn't mention speed; rather simplicity. Not
saying it isn't a consideration but premature optimisation and
all that...
> Another thing. Using shared memory would be as to use a single thread
> but using message queues would be as multiple-threading.
And therefore...?
I think you need to make your points more clearly.
TJG
Eden Kirin
> Hi,
>
> I wondered what IPC library might be best simplest for following task?
Consider using Thrift (http://incubator.apache.org/thrift/). It is
multiplatform multilanguage RPC and IPC solution. I implemented it in
couple of my projects and it works seamlessly.
Joan Miller
> [News123<news...@free.fr>]
>
> >>> I wondered what IPC library might be best simplest for following task?
>
> ...
>
> >>> xmlrpc seems to be a little heavy for such tasks.
>
> >>> signals don't allow to exchange data
>
> >>> a shared memory message queue would probably a good solution, but
> >>> python's Multiprocessing.Queue seems to require a common parent process
>
> [Vinay Sajip]
>
> >> Gabriel's suggestion is very good; if you need something which is a
> >> little more like RPC but still quite lightweight, consider Pyro
> >> (http://pyro.sourceforge.net/)
>
>
> > I've read that Pyro is not safe.
>
> That's a fairly broad thing to say. I've read lots
> of things. What does "is not safe" mean, in any case?
> I assume you've got a valid concern in mind which is
> worth passing on to a would-be user, but what exactly
> is it? FWIW I've used Pyro on and off over the years
> without any problems. Certainly my computer's never
> blown up as a result of using it.
"Pyro has never been truly designed to provide a secure communication
mechanism, nor has it had a security review or -test by a security
expert."
http://pyro.sourceforge.net/features.html
Paul Rubin
> I'm having a few python scripts all running on the same host (linux or
> win), which are started manually in random order. (no common parent process)
> Each process might be identified by an integer (1,2,3) or by a symbolic
> name ( 'dad' , 'mom' , 'dog' )
If they are running on the same host with no untrusted local users, you
can use unix-domain sockets instead of TCP sockets and then the server
should be unreachable from the internet. Then you're less exposed to
possible security issues with libraries like Pyro. Personally I've just
used SocketServer/SimpleHTTPServer on the listening side and simplejson
for serialization, but that was for low-rent, low-performance
applications. If you want something faster, zeromq (www.zeromq.org)
looks interesting.
Terry Reedy
>>> I've read that Pyro is not safe.
>>
>> That's a fairly broad thing to say. I've read lots
>> of things. What does "is not safe" mean, in any case?
>> I assume you've got a valid concern in mind which is
>> worth passing on to a would-be user, but what exactly
>> is it? FWIW I've used Pyro on and off over the years
>> without any problems. Certainly my computer's never
>> blown up as a result of using it.
>> From its own page:
> "Pyro has never been truly designed to provide a secure communication
> mechanism, nor has it had a security review or -test by a security
> expert."
> http://pyro.sourceforge.net/features.html
For communication between processes on one machine, that hardly seems to
be a worry. If it were, I would expect that sending encrypted strings
would substantially improve things.
That aside, I would wonder whether you could use a master process with a
gui to haphazardly launch subprocess, so as to avail oneself of
multiprocessing.Queue.
Terry Jan Reedy
News123
>
>> Anyway, you have in mind that respect to speed:
>>
>> shared memory> named pipes> Unix domain socket> TCP socket
>
> True, but the OP didn't mention speed; rather simplicity. Not
> saying it isn't a consideration but premature optimisation and
> all that...
>
Yes true. I'm looking for something simple, platform agnostic. Lateron I
can always improve performance.
Perhaps I'll stick initially with xmlrpc, as it is quite simple, though
a little heavy.
I just have to see how to deal with servers which are not up, no more
up, being restarted.
N
News123
Terry Reedy wrote:
>
> That aside, I would wonder whether you could use a master process with a
> gui to haphazardly launch subprocess, so as to avail oneself of
> multiprocessing.Queue.
>
T
This is also an option I'm looking.
insted of the python scripts, thet users would normally click at I had
to implement small sripts, that just communicate with the master
process, which would then launch the application.
Paul Rubin
> Perhaps I'll stick initially with xmlrpc, as it is quite simple,
> though a little heavy. I just have to see how to deal with servers
> which are not up, no more up, being restarted.
Something wrong wtih nagios?
bobicanprogram
Have a look at the SIMPL toolkit (http://www.icanprogram.com/simpl or
http://www.icanprogram.com/06py/main.html). It should do everything
you describe at least on the Linux end of the spectrum. Under the
hood SIMPL uses a shared memory scheme for local message exchange.
SIMPL would be especially effective if you want to move your modules
at some future point onto different nodes. SIMPL-Python can work
seamlessly on heterogeneous networks provided at least one node is
Linux.
bob