Interprocess communication

[i...@und.com]

I'm trying to understand interprocess communication in a better way.
Suppose that I had some C++ program running, and perhaps some python or
rust or whatnot program running that is to pass information to the C++
program, and receive information from the C++ program. What is the right
way to do this?

I believe that sockets might be the answer. Is there a good resource to
learn about these sockets with C++ in mind?

Thanks, IOP

[Marcel Mueller]

Am 29.07.20 um 07:49 schrieb i...@und.com:

> I'm trying to understand interprocess communication in a better way.
> Suppose that I had some C++ program running, and perhaps some python or
> rust or whatnot program running that is to pass information to the C++
> program, and receive information from the C++ program. What is the right
> way to do this?

There is no "right way". There is a way adequate to the specific
requirements.

Furthermore IPC is not in the scope of a programming language.
Especially not if you want to cross language (i.e. runtime library)
boundaries.
IPC is always platform dependent. Even in case of simple files they
might be part of the language but the synchronization for access is not.

> I believe that sockets might be the answer. Is there a good resource to
> learn about these sockets with C++ in mind?

AFAIK the C++ standard never covered sockets. You need to use a platform
specific library if you want to use sockets for IPC.

But there are other options too. E.g. pipes (sometimes also called
fifos). They can be accessed mostly like normal files.


Marcel

[Felix Palmen]

* i...@und.com <i...@und.com>:

> I'm trying to understand interprocess communication in a better way.
> Suppose that I had some C++ program running, and perhaps some python or
> rust or whatnot program running that is to pass information to the C++
> program, and receive information from the C++ program. What is the right
> way to do this?

There are numerous ways for processes to communicate with each other,
for example (in no specific order and probably incomplete)

* local/Unix sockets
* TCP sockets
* SYSV/POSIX message queues
* SYSV/POSIX shared memory
* pipes
* named pipes
* even signals...

What's the "right way" to do it depends on the scenario (target
operating systems, languages involved, and of course, the actual design
and purpose of your programs).

--
Dipl.-Inform. Felix Palmen <fe...@palmen-it.de> ,.//..........
{web} http://palmen-it.de {jabber} [see email] ,//palmen-it.de
{pgp public key} http://palmen-it.de/pub.txt // """""""""""
{pgp fingerprint} A891 3D55 5F2E 3A74 3965 B997 3EF2 8B0A BC02 DA2A

[Paavo Helde]

29.07.2020 08:49 i...@und.com kirjutas:
> I'm trying to understand interprocess communication in a better way.
> Suppose that I had some C++ program running, and perhaps some python or
> rust or whatnot program running that is to pass information to the C++
> program, and receive information from the C++ program. What is the right
> way to do this?

In principle this does not require multiple processes. One can embed
Python interpreter in a C++ program, and vice versa, you can implement
new Python modules in C++. This would not be trivial though, and might
mean too tight encapsulation.

>
> I believe that sockets might be the answer. Is there a good resource to
> learn about these sockets with C++ in mind?

HTTP connections over network sockets (and REST API-s in particular) is
indeed a popular way for interprocess communication. It allows for extra
flexibility in that the processes may reside on different computers (or
in different VM-s or containers, nowadays).

In C++ you can implement both an the server and client sides via the
Boost.ASIO library (https://en.wikipedia.org/wiki/Asio_C%2B%2B_library).
There is some chance it will be included into the C++ standard at some
point in the future.

So one arguably right way to do interprocess communication is to develop
a REST API based on Boost.ASIO library. Depending on your needs this
might be an overkill however, if all you want is to send a one-bit
notification to the other process, then a POSIX signal or a Windows
Event object might do the trick.

[Jorgen Grahn]

On Wed, 2020-07-29, <i...@und.com> wrote:
> I'm trying to understand interprocess communication in a better way.
> Suppose that I had some C++ program running, and perhaps some python or
> rust or whatnot program running that is to pass information to the C++
> program, and receive information from the C++ program. What is the right
> way to do this?

Sockets are not the solution to all IPC problems. Others in this
thread listed many different technologies, but the thing about these
is they are applicable in different situations, for different
problems.

For example, if you can make the data going between the programs
unidirectional, then you don't have to do anything. In a Unix shell:

% foo | bar

That's also IPC, and it would be foolish to choose sockets if a
pipeline would work.

> I believe that sockets might be the answer. Is there a good resource to
> learn about these sockets with C++ in mind?

I learned from W R Stevens' books, but I learned more from "TCP/IP
Illustrated" than from the ones about the APIs.

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

[Juha Nieminen]

One prominent example of interprocess communication is what happens
between an UCI chess engine and a chess GUI.

An UCI chess engine is its own independent command-line program that
does nothing else than read commands in human-readable ascii form
from stdin, and output results to stdout. In fact, you can run
such an engine directly and write commands to it by hand, and it
will respond by printing whatever you asked or it's doing.

A chess GUI program that supports UCI engines will run such a program as a
background process and connect itself to its stdin and stdout streams,
and communicate with it in that manner.

The advantage of this is that such a GUI program can support any UCI chess
engine and, indeed, you can usually freely add new engines to it, and they
will work with it. You can even write your own UCI chess engine and use it
with that GUI if you want. No need for plugins or dlls or whatever.

[Öö Tiib]

On Wednesday, 29 July 2020 08:50:06 UTC+3, i...@und.com wrote:
> I'm trying to understand interprocess communication in a better way.
> Suppose that I had some C++ program running, and perhaps some python or
> rust or whatnot program running that is to pass information to the C++
> program, and receive information from the C++ program. What is the right
> way to do this?

Depends where these programs run and how. I can recommend good
book if you are interested about UNIX systems in particular:
"UNIX Network Programming, Volume 2: Interprocess Communications",
W. Richard Stevens

> I believe that sockets might be the answer. Is there a good resource to
> learn about these sockets with C++ in mind?

Whole internet itself is communication between processes.
There are lot of different ways to communicate under
different scenarios. Word "sockets" is ambiguous as there
are too lot of sockets. Learning tiny subpart of
possibilities will take years.

For example when the programs that communicate are designed
*always* to run on same system that happens to support
shared memory and/or memory mapped files then those
(shared memory and memory mapped files) are most efficient
in good hands but among most disastrous in clumsy hands. ;)
When additionally both programs happen to be written in C++
then Boost.Interprocess is relatively decent library that
supports it.

[see.my....@gmail.com]

> What is the right
> way to do this?

I have noticed that all answers so far focused on the lowest possible level, which is a raw communication channel. Sockets can do that. Pipes can do that. Shared memory can do that, too.

But just as you design your program using high-level concepts like objects, classes or containers, instead of raw memory blocks, you might as well raise the level of treatment for interprocess communication. In this analogy, sockets are like raw memory blocks and using sockets is like using malloc. Nobody would suggest using malloc if you need an object-oriented design pattern. It is just not the level of abstraction that is typically needed.

I would suggest going for a message-oriented library, because the level of abstraction is higher. Yes, such libraries very likely use sockets internally, but that's not relevant (just as it is not relevant that a particular design pattern ultimately relies on some malloc calls). What is relevant is that you can send *messages* between programs.

There are many high-level libraries, some of them multi-platform, some of them multi-language.
If you need object-oriented ones, look for CORBA (although some consider it to be outdated) or ICE (https://zeroc.com/products/ice). If you are not obsessive about object-orientation, then HTTP is also an option, with multiple high-level libraries for both C++ and Python.
As a shameless plug, I will also recommend YAMI4 (http://www.inspirel.com/yami4), which happens to support both your programming languages.

The advantage of all such libraries is that you can be several steps ahead in your development thanks to the high-level of abstraction that is offered. Otherwise, you might end up reinventing the wheels (and bugs) that were already implemented (and debugged) in those libraries.

Unless, of course, all you need is to send a single flag, once, between two programs. Which is most likely not the case.

--
Maciej Sobczak * http://www.inspirel.com

[Melzzzzz]

On 2020-07-29, Felix Palmen <fe...@palmen-it.de> wrote:
> * i...@und.com <i...@und.com>:
>> I'm trying to understand interprocess communication in a better way.
>> Suppose that I had some C++ program running, and perhaps some python or
>> rust or whatnot program running that is to pass information to the C++
>> program, and receive information from the C++ program. What is the right
>> way to do this?
>
> There are numerous ways for processes to communicate with each other,
> for example (in no specific order and probably incomplete)
>
> * local/Unix sockets
> * TCP sockets
> * SYSV/POSIX message queues
> * SYSV/POSIX shared memory
> * pipes
> * named pipes
> * even signals...

memory mapped files...

[Jorgen Grahn]

On Wed, 2020-07-29, Jorgen Grahn wrote:
> On Wed, 2020-07-29, <i...@und.com> wrote:
>> I'm trying to understand interprocess communication in a better way.
>> Suppose that I had some C++ program running, and perhaps some python or
>> rust or whatnot program running that is to pass information to the C++
>> program, and receive information from the C++ program. What is the right
>> way to do this?
>
> Sockets are not the solution to all IPC problems. Others in this
> thread listed many different technologies, but the thing about these
> is they are applicable in different situations, for different
> problems.

And reading closer, Felix Palmen pointed that out already:

> What's the "right way" to do it depends on the scenario [...]

> and of course, the actual design and purpose of your programs).

[IOP]

Felix Palmen <fe...@palmen-it.de> wrote:
> There are numerous ways for processes to communicate with each other,
> for example (in no specific order and probably incomplete)
>
> * local/Unix sockets
> * TCP sockets
> * SYSV/POSIX message queues
> * SYSV/POSIX shared memory
> * pipes
> * named pipes
> * even signals...
>
> What's the "right way" to do it depends on the scenario (target
> operating systems, languages involved, and of course, the actual design
> and purpose of your programs).

Thank you. This thread has been very informative. Other posters have
referenced the book UNIX Network Programming by Stevens, and it seems
like this might also give me an idea of a lot of the content that you
mention here. I intend to read that book (so far, I've only read the
TOC). But if you had any other suggestions that you thought I should
look at, I'd be interested.

Thanks - IOP

[IOP]

Paavo Helde <ees...@osa.pri.ee> wrote:
> In principle this does not require multiple processes. One can embed
> Python interpreter in a C++ program, and vice versa, you can implement
> new Python modules in C++. This would not be trivial though, and might
> mean too tight encapsulation.

I've heard this before, but I suppose I don't really understand what
means (or how to go about) embedding a python interpreter in C++. In the
past I have written some python modules in C++ and C, but that's the
only sort of thing I've done in that direction.

> HTTP connections over network sockets (and REST API-s in particular)
> is indeed a popular way for interprocess communication. It allows for
> extra flexibility in that the processes may reside on different
> computers (or in different VM-s or containers, nowadays).
>
> In C++ you can implement both an the server and client sides via the
> Boost.ASIO library (https://en.wikipedia.org/wiki/Asio_C%2B%2B_library).
> There is some chance it will be included into the C++ standard at some
> point in the future.

Thank you! I'll examine Boos.ASIO too.

- IOP

[IOP]

Juha Nieminen <nos...@thanks.invalid> wrote:
> One prominent example of interprocess communication is what happens
> between an UCI chess engine and a chess GUI.
>
> An UCI chess engine is its own independent command-line program that
> does nothing else than read commands in human-readable ascii form
> from stdin, and output results to stdout. In fact, you can run
> such an engine directly and write commands to it by hand, and it
> will respond by printing whatever you asked or it's doing.
>
> A chess GUI program that supports UCI engines will run such a program as a
> background process and connect itself to its stdin and stdout streams,
> and communicate with it in that manner.
>
> The advantage of this is that such a GUI program can support any UCI chess
> engine and, indeed, you can usually freely add new engines to it, and they
> will work with it. You can even write your own UCI chess engine and use it
> with that GUI if you want. No need for plugins or dlls or whatever.

This sounds like a very interesting example. I looked into the UCI spec
a bit and it seems like a lot is passed around using plaintext directly
through stdin and stdout. This sounds very interesting, and this is
definitely something that I would like to better understand.
Fortunately, google shows a wealth of resources concerning UCI and these
engines, so perhaps I'll find something that makes sense to me.

Thank you - IOP

[Jorgen Grahn]

Eric S. Raymond
The Art of Unix Programming
chapter 7: Multiprograming

http://www.catb.org/esr/writings/taoup/html/multiprogramchapter.html

Note that he's defending what he thinks are classic Unix paradigms,
so a lot of people probably disagree with him.