Understanding msgid usage for encoding

[Gabriel Harrison]

Hi,

I'm looking at using protobuf in an environment where smaller IOT devices will hopefully use nanopb and the servers will likely use the standard libraries. 

I'd planned to take advantage of option (nanopb_msgopt).msgid after reading the various options particularly because it has such a small overheard and partly because it means we can stream out data on the fly without needed to know the length before transmission like you need with using proto.any. I'm planning on taking advantage of callbacks too.

Firstly, am I correct that nanopb_msgopt is the only sensible built-in nanopb option if I want to stream out before knowing the total message length?

Secondly, one of the dev partners has asked if we can use a solution that doesn't require inclusion of nanopb definitions on the server-side.  So I was looking at achieving the same thing with our own options but I'm relatively new to protobuf and am completely new to proto options.

 - If I create a similar option with a different name, is it possible to use the nano.options file to link it into to nanopb and take advantage of the extra code generation? 

- Is there something special about the field number 1010 or is it an arbitrary number big enough to not clash with other fields?

- Is the option (nanopb_msgopt).msgid field guaranteed to be at the beginning of the message? I think I read in the main protobuf reference that order is not guaranteed to be the same between implementations.

Thank you,

Gabriel

[Petteri Aimonen]

Hi,

> Firstly, am I correct that nanopb_msgopt is the only sensible built-in
> nanopb option if I want to stream out before knowing the total message
> length?

An alternative is to use plain oneofs instead:

message Container {
oneof {
MsgType1 msg1 = 1;
MsgType2 msg2 = 2;
...
}
}

Nanopb will then generate defines like "Container_msg1_tag" that you can
use for identifying the message type. This avoids needing to reference nanopb.proto,
but I'm not sure how easily the tag number is accessible in other languages.

In either case, for streaming output (bigger messages than the available
memory buffer), you cannot place the message length in front of the message.
So while streaming decode of a "Container" message works fine, streaming encode
of it doesn't.

> - If I create a similar option with a different name, is it possible to
> use the nano.options file to link it into to nanopb and take advantage of
> the extra code generation?

No, currently nanopb generator will only consider options coming from
the .options file or defined with nanopb.proto.

> - Is there something special about the field number 1010 or is it an
> arbitrary number big enough to not clash with other fields?

It's the identifier assigned to nanopb:
https://chromium.googlesource.com/chromium/src/+/master/third_party/protobuf/docs/options.md

> - Is the option (nanopb_msgopt).msgid field guaranteed to be at the
> beginning of the message? I think I read in the main protobuf reference
> that order is not guaranteed to be the same between implementations.

The option will not be in the message at all. It is just compile time
generation, you can then encode it yourself as you wish.

--
Petteri

[Gabriel Harrison]

Thank you, that's actually massively helped my understanding of pb option extension and the way the example works in tests/msgid/encode_msgid.c.