Help parsing multiple levels of nested submessages

[ryankj...@gmail.com]

Hi, I'm trying to determine the most efficient way to send submessages in a multilevel hierarchy of message types. I've been using the union message example as a reference.

Say that I have something like

message TopMessage {
optional CommandMessage cmd = 1;
optional DataMessage data = 2;
}

message CommandMessage {
optional Command1Msg cmd1 = 1;
optional Command2Msg cmd2 = 2;
}

message DataMessage {
required bytes dataBlock = 1;
}

message Command1Msg {
required uint32 command1Stuff1 = 1;
required uint32 command1Stuff2 = 2;
}

message Command2Msg {
required bool command2Stuff1 = 1;
required uint32 command2Stuff2 = 2;
}

Now if I want to just send a Command1Msg message, I can construct it using the submessage method used in the union example by looking in the parent's fields (in this case CommandMessage):

bool encode_message( pb_ostream_t *stream, const pb_field_t messagetype[], const void *message )
{
const pb_field_t *field;
for( field = CommandMessage_fields; field->tag != 0; field++ )
{
if (field->ptr == messagetype)
{
/* This is our field, encode the message using it. */
if (!pb_encode_tag_for_field(stream, field))
return false;

return pb_encode_submessage(stream, messagetype, message);
}
}

/* Didn't find the field for messagetype */
return false;
}

...
Command1Msg cmd1Msg;
encode_message( &stream, Command1Msg_fields, &cmd1Msg );

Now my question is on the receiving end. The receiver needs to look for a TopMessage. Is the best way to perform the decode to start at topMessage_fields and traverse down into each subtype until I find a match? Wouldn't I need unique tags for all submessage keys since a match is determined by field->tag?

Still pretty green with how everything is encoded and with the API, so apologies if I'm thinking about things incorrectly.
Thanks,
-- Ryan

[Petteri Aimonen]

Hi,

> Now my question is on the receiving end. The receiver needs to look
> for a TopMessage. Is the best way to perform the decode to start at
> topMessage_fields and traverse down into each subtype until I find a
> match? Wouldn't I need unique tags for all submessage keys since a
> match is determined by field->tag?

Well, do the union thing once to detect whether it is command message or
data message. Then do the union thing again to detect the subtype.

--
Petteri

[RyanJ]

Well, do the union thing once to detect whether it is command message or
data message. Then do the union thing again to detect the subtype.

--
Petteri

Hmm, so I guess I need to encode it twice, correct? Once as a subtype in CommandMessage_fields and then again in TopMessage_fields? Because if not, say I encode a Command2Msg only as a subtype to its parent, CommandMessage. That means its tag will be 2.Then if on the decode I start looking for a TopMessage, a tag value of 2 means a DataMessage submessage, not a CommandMessage, and so I would decode it as the wrong subtype. Correct?

Thanks,
-- Ryan

[Petteri Aimonen]

Hi,

> Hmm, so I guess I need to encode it twice, correct? Once as a subtype in
> CommandMessage_fields and then again in TopMessage_fields? Because if not,
> say I encode a Command2Msg only as a subtype to its parent, CommandMessage.
> That means its tag will be 2.Then if on the decode I start looking for a
> TopMessage, a tag value of 2 means a DataMessage submessage, not a
> CommandMessage, and so I would decode it as the wrong subtype. Correct?

Yes. If it is a nested submessage in the .proto file, it has to be
nested submessage in encoding/decoding also. You can also put everything
inside TopMessage in .proto, without nesting, if you want to.

--
Petteri

[RyanJ]

Thanks Petteri, that makes sense.

So I was playing around with the union example. If rather than constructing just a submessage as in

/* Send message of type 3 */
MsgType3 msg = {3, 1415};
status = encode_unionmessage(&stream, MsgType3_fields, &msg);

instead I build from a top-level UnionMessage as follows:

UnionMessage uMsg = UnionMessage_init_zero;
uMsg.has_msg3 = true;
uMsg.msg3.value1 = 3;
uMsg.msg3.value2 = 1415;
status = pb_encode( &stream, UnionMessage_fields, &uMsg );

in both cases the data written to the stream (and the length) is the same:

Wrote 7 bytes:
1a 05 08 03 10 87 0b

Is that generally true? If so, is there a benefit to building just a submessage as opposed to a top-level with the optional parts filled in? Or is that the point of the example: to show that they are equivalent :)

Thanks again,
-- Ryan

[Petteri Aimonen]

Hi,

> UnionMessage uMsg = UnionMessage_init_zero;
> uMsg.has_msg3 = true;
> uMsg.msg3.value1 = 3;
> uMsg.msg3.value2 = 1415;
> status = pb_encode( &stream, UnionMessage_fields, &uMsg );

Yes, this is how you would normally use nanopb.

> Or is that the point of the example: to show that they are equivalent

Exactly :)

The point is that the uMsg variable can get pretty large if you have
many message types, and the union example shows how to avoid that memory
usage. But if you have enough memory, there is no point to it.

--
Petteri

[RyanJ]

Ah, of course. The over-the-wire size is the same in both cases, but you don't necessarily need the RAM burden of holding full messages.

Thanks for helping it to click for me!
-- Ryan

[RyanJ]

Hi Petteri, I have a followup question for you.

Is it possible to use the submessage encoding method used in the union example with extensions? For example, if I modify unionproto.proto as such:

message UnionMessage
{
    optional MsgType1 msg1 = 1;
    optional MsgType2 msg2 = 2;
    optional MsgType3 msg3 = 3;
    extensions 100 to 255;
}

extend UnionMessage {
    optional MsgType4 msg4 = 100;
}

message MsgType4 {
    required int32 value = 1;
}

Is there a way to take a MsgType4 message and encode it as a submessage of UnionMessage the way the example does with MsgType1-3?

Thanks,
Ryan

[Petteri Aimonen]

Hi,

> Is it possible to use the submessage encoding method used in the union
> example with extensions? For example, if I modify unionproto.proto as such:
>

> Is there a way to take a MsgType4 message and encode it as a submessage of
> UnionMessage the way the example does with MsgType1-3?

Well the example doesn't directly work, because it cannot find
MsgType4_fields automatically. You would have to pass it as a parameter.
There is no way to list all possible extensions, because they could come
from multiple files.

--
Petteri

[RyanJ]

Ok, that's what I thought, and your reason is pretty much the problem I'm trying to solve.

Basically I'm working on an interface where there would be one global message which is composed entirely of optional submessages. Some of those submessages would be defined in the same .proto as the top level message, while others would be in a separate file (separating 'my' messages from customer messages). Because of the number of submessages, the size of the top level message would be quite large, hence me trying to find ways to use just submessages. :)

Is there a method you would suggest? I'm assuming that the 'global top level message' way is really the only way for the receiving side to receive any potential submessage type which was sent. So far it's looking like the easiest thing is to not use extensions or sub-submessages, but to keep everything flat and in the same .proto file.

Thanks again,
Ryan