Is it possible to construct messages directly in shared memory for IPC?

[谢骐]

Hi, friends.

I'm writing a small system consisting of two processes: a publisher and a subscriber, by C++. I use Protocol Buffers and Arena.

In this small system, publisher will directly construct some protobuf messages in the memory area (created by mmap and passed to Arena) shared by publisher and subscriber, without serialization (i.e., zero copy).

However, there are 2 (seemingly unsolvable) problems:

• the underlying data of std::string is uncontrolled, thus subscriber cannot read std::string::c_str().
• the virtual table pointer of protobuf message is an absolute address in process publisher, thus it's difficult to use reflection, and whenever I call google::protobuf::util::MessageToJsonString the system will crash.

Should I give up?  (Many of my efforts seem to be in vain.)

Thanks.

[Em Rauch]

I believe Protobuf doesn't support this usecase directly; it's an even harder case than sharing messages without serialization across .so files which is already actually very tricky to get right (and can do anything from subtle to catastrophic when you get it wrong). Generally it is preferred to simply need to serialize+parse at the IPC boundary (not least because it will also notably handle schema skew gracefully).

IIRC gRPC does support some transport path where protobuf messages are passed without serialization, but I'm unsure if that is only for same-process loopback RPCs or if gRPC may help get the performance characteristics you're looking for in under IPC communication (but it would require adoption of gRPC which may also be heavyweight compared to your usecase).

You may find other formats like FlatBuffers or Cap'n Proto are better fits if your usecase would significantly benefit from zero serialization between endpoints.

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/protobuf/9747c093-6c40-4ff0-beb1-0b733fe55905n%40googlegroups.com.