Is protobuf/nanobp suitable for my embedded applications?

[pozz...@gmail.com]

I usually work on embedded applications, from 8- to 32-bits MCUs. Lastly I'm using Cortex-Mx devices.

I often have the need to exchange some data between MCUs (on a point-to-point serial link or on a multi-drop bus network topology), or between one MCU and an application running on a full OS (Linux, Windows) written in a high-level language (C++, Python).

In both cases I think protobuf has some advantages (over my custom solutions), mainly because it is future-proof and more fields can be added without breaking compatibility with older version of messages.
When the messages are managed by a Python application, it will be very easy to decode and encode it.

Of course the firmware running on one MCU can't use dynamic memory allocation (malloc) and the RAM memory is often limited. So I looked at nanopb implementation of protobuf.

With my custom solution, when I have to serialize some data I write:

serialize_u8(get_data1());
serialize_str(get_data2());
etc

serialize_...() functions directly write the encoded data to an output buffer, mostly a transmit FIFO buffer of the UART peripheral.

With nanopb I have to create the instance of the message in RAM and only after that, encode it:

MyMessage msg;
msg.data1 = get_data1();
msg.has_data1 = true;
strcpy(msg.data2, get_data2());
msg.has_data2 = true;

pb_encode(&ostream, &msg);

In this example, I made the assumption data2 field is a fixed-length string.
If the message contains many fields, during serialization I need an additional memory space to temporarily save the message instance (msg variable).
Of course, it could be an automatic variable allocated on the stack by the compiler (so not consuming the RAM for global variables), but a stack overflow could occur if the message contains many fields.

Moreover, I understood you can't define a field as required in proto3, so every field is doubled with its has_... boolean field counterpart.

Is there a way to avoid this additional memory requirement?

[Petteri Aimonen]

Hi,

> With nanopb I have to create the instance of the message in RAM and
> only after that, encode it:

Your understanding is correct, and this does require a bit more RAM than
the function-driven encoding method. The reason I chose this kind of API
for nanopb is ease of use. On the other hand, nanopb supports directly
encoding and decoding from a serial port stream, so you do not need to
allocate RAM for the encoded message.

There are a few protobuf libraries that use the kind of API you referred
to:
https://github.com/squidfunk/protobluff
https://github.com/cloudwu/pbc/

I'm not sure if they are otherwise suitable for embedded systems.

> Moreover, I understood you can't define a field as required in proto3,
> so every field is doubled with its has_... boolean field counterpart.

This part is now fixed thanks to berni155's pull request #216. In
proto3 the has_ fields are not actually required. This fix will be
included in the next release once I find a moment of time to test it out
a bit.

--
Petteri

[pozz...@gmail.com]

Il giorno domenica 23 ottobre 2016 10:02:46 UTC+2, Petteri Aimonen ha scritto:
> Hi,
>
> > With nanopb I have to create the instance of the message in RAM and
> > only after that, encode it:
>
> Your understanding is correct, and this does require a bit more RAM than
> the function-driven encoding method.

Just to alleviate the RAM space needed when encoding/decoding, I'm thinking to group the fields of a long message in a few sub-messages.

message LongMessage {
message Group1 {
...
}
message Group2 {
...
}
Group1 grp1 = 1;
Group2 grp2 = 2;
}

Is it possible to encode/decode one submessage at a time?

Group1 grp1;
grp1.field1 = get_data...();
...
pb_encode(&ostream, Group1_fields, &grp1);

Group2 grp2;
grp2.field1 = get_data...();
...
pb_encode(&ostream, Group2_fields, &grp2);

I tried something, but it seems it doesn't work as expected.

[Petteri Aimonen]

Hi,

> Is it possible to encode/decode one submessage at a time?

It can be made to work, but you need to decode the outer structure
yourself. Check examples/using_union_messages, it does a similar thing
for a similar purpose.

You can also make partial messages, if you don't mind having to maintain
the match manually. For example:

message Complete {
optional uint32 field1 = 1;
optional uint32 field2 = 2;
optional uint32 field3 = 3;
}

message Partial1 {
optional uint32 field1 = 1;
}

message Partial2 {
optional uint32 field2 = 2;
optional uint32 field3 = 3;
}


Then you can decode partial messages as if they were the complete one,
because extra fields are ignored. And append them when encoding, to
yield a complete message.

--
Petteri