Clarification on how to determine proto format before decoding?

[Don'tJump]

I'm new to the idea of using protobufs in embedded. Biggest thing I need clarification on is how I determine which message just came in.

I have message coming in 40 bytes packets. Messages can be of various content, meant for various functions. I'll assemble them and check a CRC to make sure they all got there. Once I do that, how do I determine where to which fields struct to use? For all I know my A.proto message came in, but maybe it was a B.proto.

I see two options.

1. I make all of my protos contain a message identifier as the first element. They all contain id=SOMEBYTE. I can decode to a temporary format, check the known first byte and then switch to and decode to the proper format.

2. This may be where the oneof is used? Except that I have wildly different messages coming in. So instead of an A.proto and B.proto, am I supposed to have a unified.proto that oneof's the unique contents of A and B?

Is one of these correct? Is there a third option I'm not thinking of?
Thank You!

[jumpif...@gmail.com]

3. I need to let the message handler know the message type before attempting to decode. So that if I receive a couple packets and assemble them, I'd need some byte code meta data to inform which decode arguments to use.

None of these three seem elegant, I must be missing something. I saw an example that had message SearchResponse{...} and message SearchRequest{...} in the same proto, meaning that this must come up often.

[Petteri Aimonen]

Hi,

> I'm new to the idea of using protobufs in embedded. Biggest thing I
> need clarification on is how I determine which message just came in.

Yeah. This is a common problem, because the protobuf specification
doesn't specify how to determine message length or message type.

> 1. I make all of my protos contain a message identifier as the first
> element. They all contain id=SOMEBYTE. I can decode to a temporary
> format, check the known first byte and then switch to and decode to
> the proper format.

This works. Often the ID is placed in a separate header record along
with the message length, but it can also be the first field (tag 1) of
the message. It is possible to define a Header message type that only
contains the first field, decode the message using that, and then
re-decode using the proper type.

You can also use the macros generated by nanopb msgid option to assign
IDs and generate decoding logic for this kind of system automatically:
"option (nanopb_msgopt).msgid = 104;"

> 2. This may be where the oneof is used? Except that I have wildly
> different messages coming in. So instead of an A.proto and B.proto, am
> I supposed to have a unified.proto that oneof's the unique contents of
> A and B?

This is the option I would recommend, because it is the easiest to get
working in any protobuf library.

You can have your separate protos in multiple files, and then have a
single "container.proto" with:

#include "protoA.proto"
#include "protoB.proto"
message Container {
oneof msg {
MessageTypeA messageA = 1;
MessageTypeB messageB = 2;
...
}
}

Then when you allocate a Container struct in C code, it will
automatically have space for the largest of the submessages. You can
decode to that and then do a switch-case on container.which_msg to pass
it to a handler function. E.g:

Container container;
pb_decode(stream, Container_fields, &container);
switch (container.which_msg)
{
case Container_messageA_tag:
return handle_messageA(&container.msg.messageA);

case Container_messageB_tag:
return handle_messageB(&container.msg.messageB);
}

Main limitation with oneof approach is that it doesn't currently allow
using nanopb callbacks in decoding the submessages. If you need
callbacks, you can use the slightly more complex method shown in
examples/using_union_messages. This example has the same binary format
as the oneof above so it is easy to transition if needed. (And you don't
even need to modify the .proto except to just specify (nanopb).no_unions
= true).

> 3. I need to let the message handler know the message type before
> attempting to decode. So that if I receive a couple packets and
> assemble them, I'd need some byte code meta data to inform which
> decode arguments to use.

Yeah, this is also sometimes done by e.g. putting a 4-byte uint32_t in
front of each message to identify the type. Not very elegant, but works.

--
Petteri