Reference-counting and polymorphism of protobuf message classes in C++

[John Lilley]

I have two (maybe unrelated) questions:

-- Is there a recommended way to wrap "smart pointers" around protobuf message objects in C++?

-- Is there a way to make all messages inherit from the same base class or interface?  Note I am not trying to make message definitions inherit from each other; I only want the classes generated C++ code to inherit from a class that I control.'

Thanks

john

[Adam Cozzette]

I've found that std::unique_ptr usually works well with messages in C++. Probably the only case where things get a little bit complicated is if you're using arena allocation, since in that case the arena owns the message and you must not call delete on it. Within Google we have a special pointer type that keeps track of whether a message was arena-allocated or not, and makes sure to call delete only if the message was heap-allocated.

About inheritance, there is no way to make all messages inherit from a specified class. However, all messages do inherit from google::protobuf::MessageLite (and google::protobuf::Message if you're not using the lite runtime), so that can be useful if you want to write code that operates on arbitrary message types.

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.
To post to this group, send email to prot...@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[John Lilley]

Thanks Adam!  

I'll look into arena allocation, since all I really want is destroy-after-use. Will that do it?

Regarding "polymorphism", what I'm really after is some way to manage and act upon a group of messages, something like:

1. Get incoming packet from a queue, deserialize it and create the message object.

2. Send the message object (hence the need for a base class) to a "dispatcher"

3. Dispatcher invokes the appropriate processing method which creates a response message.

4. Serialize the response and send packet on return queue.

Steps 1 and 3 are the parts that I'm grappling with, both in Java and C++.  Any tips on 

-- The best way to examine the packet and create the correct message?  I have some freedom to put framing around the packet if necessary.

-- The best way to create and invoke a message->method mapping in Java / C++?

Thanks

john

[Adam Cozzette]

I realized I missed the fact that in your subject line you mentioned reference counting. In that case it sounds like you might want to use std::shared_ptr instead of std::unique_ptr, but that should also work.

Arenas are good for performance but come at the cost of being a little more difficult to work with, so if you just want to make sure the message is destroyed after use in a simple way then I would guess arenas aren't the best option. The simplest thing would be to put your message directly on the stack so that it gets destroyed when you return from the function, but of course that doesn't work if you need the message to outlive the function call. So if putting it on the stack doesn't work for your use case, I would recommend going with std::unique_ptr or std::shared_ptr.

About your polymorphism question, I think a good solution would be to create your own class or interface and have that wrap the proto message, since that way your dispatcher could operate on multiple kinds of message without necessarily having to know the concrete message type. This polymorphism could handle  your message->method mapping since you could have a common interface with a separate implementation for each message type. Serialized protocol buffers do not indicate their type or size, so you usually have to set up your own framing to at least indicate the message size. It's common to use a varint size prefix before the message payload, though that's not the only way to do it. To indicate the type of the message, probably the easiest approach is to have a parent message that contains all possible message types inside a oneof field.

After writing all this it occurs to me that you might just want to go with an RPC framework like gRPC, since that would set up all the infrastructure you need for having service endpoints operating on protocol buffers--have you looked into gRPC by any chance?

[John Lilley]

Adam,

Thanks! all good suggestions. These confirm what I'm discovering on my own. Simple framing packet containing the message name and bytes. I've found the classes/methods for creating a concrete message given its name and byte-buffer. So now I'm trying to find a technique for generating both server-side and client-side code with request/response methods operating on concrete message type.  It seems like gRPC is the only "silver bullet", but it is tied to web services.  Meanwhile we're heading down a message-broker path, so marrying these two things together is the trick.  LMK if you think of something clever or if gRPC can be tricked into using a message-queue instead.

Cheers,

john

[Adam Cozzette]

I haven't worked with a message broker myself but I think you might find that gRPC can work well in that scenario, too. gRPC is not really tied to web services although that might be a common use case. It supports different variations of streaming requests and responses, so you're not limited to the typical send-one-request-and-wait-for-the-response model. You could stream a bunch of messages to the server without having to wait for a response for each one. I'm not sure if that necessarily helps with the message broker setup but I think there's a lot of flexibility there.