Any vs. oneof in message wrappers

[David Koch]

Our platform generates/processes different types of proto3 messages with Kafka as buffer. We currently store this type as part of the Kafka message key so message consumers know which class to use for de-serialization.

I want to add the type to the serialized message in order to not rely on the existence of a separate key. I thought about creating a wrapper message like this:

message Header {
  enum PayloadType {
    TYPE_ONE = 1;
    TYPE_TWO = 2;
  }

  int64 tmst = 1;
  string message_id = 2;
  string producer_id = 3;
  PayloadType payload_type = 4;
}


// Any payload.
message MessageWrapper {
  Header header = 1;
  Any payload = 2;
}

or alternatively:

// List different payload options.
message MessageWrapper {
  Header header = 1;
  oneof payload {
    PayloadTypeOne payload_type_one = 2;
    PayloadTypeTwo payload_type_two = 3;
  }
}

My question: Is one of the two alternatives to be preferred?

Using Any leads to more complex setting/getting in the application code but it is more generic from the message definition point of view. Dropping the Header.PayloadType field in the definition using Any would allow me to not modify the MessageWrapper definition at all when adding new payload types.

Thanks.

[Feng Xiao]

On Mon, May 16, 2016 at 11:30 AM, 'David Koch' via Protocol Buffers <prot...@googlegroups.com> wrote:

Our platform generates/processes different types of proto3 messages with Kafka as buffer. We currently store this type as part of the Kafka message key so message consumers know which class to use for de-serialization.

I want to add the type to the serialized message in order to not rely on the existence of a separate key. I thought about creating a wrapper message like this:

message Header {
  enum PayloadType {
    TYPE_ONE = 1;
    TYPE_TWO = 2;
  }

  int64 tmst = 1;
  string message_id = 2;
  string producer_id = 3;
  PayloadType payload_type = 4;
}


// Any payload.
message MessageWrapper {
  Header header = 1;
  Any payload = 2;
}

or alternatively:

// List different payload options.
message MessageWrapper {
  Header header = 1;
  oneof payload {
    PayloadTypeOne payload_type_one = 2;
    PayloadTypeTwo payload_type_two = 3;
  }
}

My question: Is one of the two alternatives to be preferred?

I don't think you need the header in either case. If you are using an Any message as payload, the payload already has the message type encoded in the type URL. If you are using an oneof, the receiver can also query which payload is set by using the oneof case API.

Comparing the two (using Any and using oneof), using Any would give your users more flexibility to use their own message types, while using oneof will require all users to define their message types in a single place (all imported by the MessageWrapper type). The API for oneof is straightforward then Any though. If you need the flexibility to allow different components to define different payload types, Any is probably better. Otherwise I would favor the simpler oenof API.

 

Using Any leads to more complex setting/getting in the application code but it is more generic from the message definition point of view. Dropping the Header.PayloadType field in the definition using Any would allow me to not modify the MessageWrapper definition at all when adding new payload types.

Thanks.

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.
To post to this group, send email to prot...@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[David Koch]

Hi Feng,

Thank you for the answer. Good point about redundant type information in the header message.

Regards,

/David