Data Sharing with Non go application

[R Srinivasan]

In my app, I would like to use golang for a data aggregator subsystem. Would like a fast and efficient way of sharing this data with a consumer subsystem developed in C++. (Both running in the same system).

Solution I am pursuing is a shared memory but I am stuck at a somewhat platform independent way of controlling access to this shared memory. Semaphores, Mutexes that I search for in the go world lead to intra process examples.

Platforms of interest - Linux and Windows.

Looking for ideas.

Thanks, srini

[Shulhan]

On Sat, 3 Apr 2021 03:42:03 -0700 (PDT)
R Srinivasan <s...@srin.me> wrote:

> In my app, I would like to use golang for a data aggregator
> subsystem. Would like a fast and efficient way of sharing this data
> with a consumer subsystem developed in C++. (Both running in the same
> system).
>
> Solution I am pursuing is a shared memory but I am stuck at a
> somewhat platform independent way of controlling access to this
> shared memory. Semaphores, Mutexes that I search for in the go world
> lead to intra process examples.
>

You should not pursue that, it may be possible, but it will lead to
complicated code and hard-to-debug program. My suggestion is put the
message bus/broker in between the producer (Go program) and consumer(s)
(C++), even thought both running in the same system. It will ease
your mind, now and later. My preference for message bus/broker is Nats
[1].

> Platforms of interest - Linux and Windows.
>
> Looking for ideas.
>
> Thanks, srini
>

[1] https://nats.io

[Brian Candler]

Have a look at nanomsg.   This should abstract away some of the underlying complexity and let you switch between TCP, IPC and other transports.

Also look at serialization formats where the data can be consumed without unpacking it: flatbuffers / flexbuffers, capnproto

(These suggestions are on the assumption that you expect the overhead of message passing to be significant compared to the processing done on each message.  If that's not true, then you might as well just stick to gRPC)

[R Srinivasan]

Great suggestions. Thanks. I will investigate the options. srini