Fwd: more on IPC-ish python stuff

[Grant Bowman]

---------- Forwarded message ----------
From: jim <j...@systemateka.com>
Date: Sat, Dec 4, 2010 at 10:29 AM
Subject: more on IPC-ish python stuff
To: Grant Bowman <gran...@gmail.com>

this is a great tutorial:
http://www.doughellmann.com/PyMOTW/os/

  the  os  module has several pipe*() functions as
well as a fork() function. there's a signal module
that supports sending signals between a child and
parent (and probably any two processes, assuming
each knows the other's PID).
  the pipe*() variants work a little differently,
mainly with respect to the use of stdin, stdout,
and stderr, but seems to me at least one of the
pipe*() functions might provide the IPC capability
needed.
  you probably know it, but just to be sure (or
expose my ignorance), a signal is an integer value
from 0 to 0xFF. there are a few signals that are
reserved for use by the kernel, but most values in
the range are undefined, so IPC design with signals
is a matter of defining some of the unreserved
values, one for each possible change in state.

[Jeremy Sturdivant]

If you want *real* IPC, it's built into Python ;) http://docs.python.org/library/mmap.html

Of course, combining that with signals is helpful too. In general, hijacking unnamed signals for your own use seems like poor practice. If you do need instant notification to a specific process, rather than a passive check to a shared mmapped file, one of the named-but-unused signals, such as SIGUSR1, or a commonly hijacked signal like SIGHUP (Often used by daemons to force a reread of configuration files) can be used to trigger an update of a process' state from the mmapped region. Using mmap to communicate is as simple as two processes mmapping the same file (preferably something in tmpfs to avoid unneeded disk writes), and voila, a common memory space is shared. From there, it's as easy as any other shared memory IPC, that is to say, not very. Moving along.

Pipes are also handy for IPC when something needs to be directly and uni-directionally communicated in a manner that can be blocked on, and the os module contains another useful function for just this, os.mkfifo(), comparable to the mkfifo command, which creates a named pipe. A named pipe, as you may be aware, is a special file that when opened, acts as a pipe, information written to it can then be read, and multiple processes can contribute to this common pool, while in general it's best to have only one process reading from it. Pipes are clunky, not very well supported or understood by many, and can be unpredictable, especially with multiple readers. Once more, moving on.

A third, and eternally handy tool for IPC is the classic example, the socket. I'm sure you know how IP-based sockets work (AF_INET, or AF_INET6 for IPv6, usually used with TCP or UDP), but an underused tool that slips by many for IPC is the UNIX socket (AF_UNIX), which is yet another way of causing two processes to communicate, using a file. In this case, the file itself *is* a socket, similar to a special file, but really more special than a device node or a named pipe. UNIX sockets behave similarly to TCP sockets, in that there is a listener, and other processes can then connect to the socket, and the listener can accept() on the socket to pick up connections. Slightly more portable, and nearly identical in usability with an increase in portability, one can simply listen using a tradition TCP socket bound to the loopback address on a pre-arranged port, but UNIX sockets do have the advantages of being named, not taking up potentially valuable port space, and with relative paths, multiple copies of the same program running from say, different users' home directories can use the same configured location for the socket, yet still be disassociated if desired. Either UNIX sockets or local TCP sockets provide fast, simple IPC that's hard to beat, and if needed, can be later modified to suit clustering needs merely by listening on an external interface, and having connecting programs connect to the common node running the listener.

So in summation, while signals, shared memory, pipes (named and otherwise), and even dark magic such as multithreading have their place in the world of communication between tasks, Sockets provide a fast, easy, and effective way to communicate between different implementations, different languages, and even different operating systems.


TL;DR, Sockets, they work. Nice stuff so far though, guys. Keep up the good work, and lemme know if there's something specific I can lend a hand with!

 -Jeremy