allocate google protobuf C++ objects on shared memory

[tarun kumar]

Hi protobuf owners and developers , 

To give the background we have extensively used the google protobuf 

1) for messaging between processes  (because its ability to serialize and desiralize very efficiently)

2) as well as for datamodel within application. (as its reflection capabilities and mapping files give us the ability to keep datamodel pluggable to application to high extent).

For case 2 we iterate through the mapping (where we have source as queries to database, target as generated proto objects and mapping of database columns and probuf class member variables) and create maps of protobuf objects which works as base datmodel (reference data) for our application. Thus our datamodel initialization is plugged out of source code , so far so good.

Now we want to move this entire reference data into shared memory as we are running multiple instances (80 of them :) ) of same application, and we want to have the feature where this reference data can be updated in real time (intraday , till now it was only initialized at start of the day and no change was allowed intraday) if data changes in underlined database. 

If the reference data is in shared memory one component can update the same if something is updated in underlined database and all the application will automatically get the updated data. 

Keeping these bunch of maps in shared memory is an easy task for me (using boost ipc shared memory segment manager) . But as map gets allocated into shared memory, objects inside maps and there member variables should also be allocated to the same shared memory. It it would be my own written  class as members of map the task was easy I would have used primitive data types int , float , char array and in most complex case boost:ipc:basic_string if required. 

(http://www.boost.org/doc/libs/1_63_0/doc/html/interprocess/allocators_containers.html -> example is close to what i am trying to do)

But in protobuf I do not find any way to represent strings as char array , and the only data type is string which gets generated as std::string. so I am stuck here as due to this I am not able to allocate these member strings on shared memory and thus the protbuf objects on shared memory. 

It would be great if you help me on this . My protobuf objects are pretty simple few int, float and string members (fixed byte char array will also suffice.). Thanks in advance.

Regards,

Tarun Kumar

[Adam Cozzette]

One thing that might help for your situation is to allocate protos on an Arena. Protocol buffers don't have support for C++ allocators but this is the closest thing to that. When you create an Arena you can pass in an initial block of memory to use, so you could potentially create a block of shared memory and have your protos use that memory for allocations that way. I've never heard of anyone trying to do this, though, so I can't say for sure whether this is a good idea or not. The thing with arenas is that they cannot free up any memory until they're destroyed, so you would have to be careful about that as well.

As you mentioned, the other problem is that you can't control where string fields are allocated, because the API uses std::string, and the default allocator is built into its type. Within Google we have an option ctype=STRING_PIECE (see here) that you can set on a string field so that it exposes an API based on the raw byte array instead of std::string, but unfortunately we haven't been able to open source it yet because that requires a fair amount of cleanup work to happen.

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscribe@googlegroups.com.
To post to this group, send email to prot...@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.