Minimal Nanopb_Cpp use for repeating message

[Jonathan Bennett]

Hey, I'm working on the Meshtastic project, written in C++, and we use Nanopb to turn our fairly complicated protobufs into C headers. One message in particular has a repeating message inside it, which has other messages inside that. Our current codebase uses a max_count option so we get this repeating message as a static array.

We're trying to get rid of that max_count option, and I came across Nanopb_Cpp as a potential way to nicely turn the repeated message into a std::vector. The problem I'm running into is that the documentation for Nanopb_Cpp seems to say that I need to do a full conversion of the entire protobuf. I would love to just get a std::vector of our existing structs. Am I missing a simple way to use Nanopb_Cpp to just convert a repeating message into a vector?

Thanks!

Jonathan Bennett

[Petteri Aimonen]

Hi,

> We're trying to get rid of that max_count option, and I came across
> Nanopb_Cpp as a potential way to nicely turn the repeated message into a
> std::vector. The problem I'm running into is that the documentation for
> Nanopb_Cpp seems to say that I need to do a full conversion of the entire
> protobuf. I would love to just get a std::vector of our existing structs.
> Am I missing a simple way to use Nanopb_Cpp to just convert a repeating
> message into a vector?

I'm not very familiar with nanopb_cpp, but in main nanopb you could do
something like this (untested):

.proto:

option (nanopb_fileopt).include = "<vector>";

message MyMessage
{
repeated SubMessage myfield = 1 [(nanopb).callback_datatype="std::vector<SubMessage>"];
}

.cxx:

bool MyMessage_myfield_callback(pb_istream_t *istream, pb_ostream_t *ostream, const pb_field_iter_t *field)
{
std::vector<SubMessage> *vec = (std::vector<SubMessage>*)field->pData;

for (SubMessage &item: vec)
{
pb_encode_tag_for_field(ostream, field);
pb_encode_submessage(ostream, SubMessage_fields, &item);
}
}

I'm not sure if there will be some complexity about putting a C++ object inside
the C struct, because it needs some logic to call the constructors and destructors
correctly. I think it should work, because the struct construction will be in your
own C++ code, and the nanopb C side only sees offsets to fields.
A pointer to a vector would be an alternative.

--
Petteri

[Jonathan Bennett]

That is far more to work with than I had. I'll give this approach a shot!

[Jonathan Bennett]

Hey, I've been working through this, and a couple questions have popped up. First, this callback function is a bool. What does that return value signify, just success in encoding?

This callback seems to run when encoding the value back into a protobuf. Do I need a second callback to handle decoding a protobuf into a vector?

Thanks!

On Sunday, March 17, 2024 at 3:20:53 PM UTC-5 Petteri Aimonen wrote:

[Petteri Aimonen]

Hi,

> Hey, I've been working through this, and a couple questions have popped up.
> First, this callback function is a bool. What does that return value
> signify, just success in encoding?

Yeah. You should pass through any false return values from pb_encode_submessage() etc.
so that encoding terminates if there is IO error or buffer full.

> This callback seems to run when encoding the value back into a protobuf. Do
> I need a second callback to handle decoding a protobuf into a vector?

The name bound callbacks get "istream" when decoding and "ostream" when
encoding, so you can do something like:

if (istream) { ... decode .. }
if (ostream) { .. encode ..}

If there are multiple callback fields in same message, check field->tag also.

Relevant docs:
https://jpa.kapsi.fi/nanopb/docs/concepts.html#function-name-bound-callbacks

--
Petteri

[Jonathan Bennett]

I'm almost there. I'm successfully decoding protobuf data into a scratch buffer inside the callback function. The only thing I haven't managed is actually pushing the submessages back into the vector, to use them. Is that accessible through the pb_field_iter_t that gets passed into the callback?

I currently have:

bool meshtastic_DeviceState_callback(pb_istream_t *istream, pb_ostream_t *ostream, const pb_field_iter_t *field)

{

if (ostream) {

std::vector<meshtastic_NodeInfoLite> *vec = (std::vector<meshtastic_NodeInfoLite> *)field->pData;

for (auto item : *vec) {

if (!pb_encode_tag_for_field(ostream, field))

return false;

pb_encode_submessage(ostream, meshtastic_NodeInfoLite_fields, &item);

}

}

if (istream) {

meshtastic_NodeInfoLite node; // this gets good data

std::vector<meshtastic_NodeInfoLite> *vec = (std::vector<meshtastic_NodeInfoLite> *)field->pData; // this is probably wrong

while (istream->bytes_left) {

pb_decode(istream, meshtastic_NodeInfoLite_fields, &node);

}

vec->operator[](0) = node; // this crashes

}

return true;

}

[Petteri Aimonen]

Hi,

> std::vector<meshtastic_NodeInfoLite> *vec = (std::vector<
> meshtastic_NodeInfoLite> *)field->pData; // this is probably wrong

I think it looks correct. If you are unsure, you could check in debugger
or printf() the pointer value of the vector before you call the outer pb_decode(),
and compare with the one you get here.

> vec->operator[](0) = node; // this crashes

If the vector is empty, I think you would want to do a vec->push_back(node);
instead?

--
Petteri

[Jonathan Bennett]

That was it. I thought this was an already resized vector, and it was actually size 0. Thank you so much for the help!