Protocol Buffers on an embedded system

[Dale Peterson]

I am collecting data from sensors with an ARM Cortex-M4 running and RTOS, and logging the data in binary format to an micro SD card, which is then periodically transferred to a PC for analysis.  I am using the FatFS [0] FAT32 filesystem library along with the ChibiOS RTOS [1].  After a few revisions of my data format, I realized that this problem is directly solved by Google Protocol Buffers and that I should be using something like it to manage the serialization/deserialization of my data.  However, it seems there are some challenges to get my firmware to build.  The two issues I'm facing are:

1) pthreads isn't available for my target.

2) Target device is an ARM Cortex-M4 with the armv7m-e instruction set.

3) ostream does not work out of the box in the embedded environment (though I might be able to serialize to a string)

Here are my questions:

1) Has anybody successfully used protocol buffers in an identical or very similar environment?  If so, can you point me to your code so I can see how you made it work, or offer any advice on what is needed?

2) I am tempted to use another project called nanopb [2] because it seems like it might be easier to get working on my embedded system.  Has anybody here used it who can report on how well it worked when inter-operating with GPB?  nanopb should be compatible with GPB but I just want to make sure the process is actually smooth sailing.  All my embedded code is in C++ so I would prefer to use GPB over nanopb if possible, but if it is too much to make GPB work on the embedded system then I guess I would have to use both (nanopb on firmware and GPB on PC that decodes data from the SD card).

Thank you for any insights or thoughts.

Dale

[0] -- http://elm-chan.org/fsw/ff/00index_e.html

[1] -- http://chibios.org/dokuwiki/doku.php

[2] -- http://koti.kapsi.fi/jpa/nanopb/

[Trepi Dacious]

1) I'm guessing the STM32F4 MCU? In that case yes, I've used protobuf in an identical environment (chibios, etc.).

2) Yes, I've used nanopb extensively, and it works very well. I've only run into one issue with inter-operation with the Java version of protobuf (and also protoc) - nanopb perfectly reasonably doesn't check that the enum values you use are actually in the enum, whereas some other implementations do. The result of this is not terrible - if you encode with nanopb and decode with such an implementation, the decoder will just ignore the invalid enum values, possibly adding them as unknown fields. In addition you might want to validate your own enums on the MCU before encoding and after decoding, if the results of an invalid value are bad...

In general it's really no harder to use nanopb on the firmware and GPB on the PC than to use the same on both ends, since they both follow the protobuf spec., although it's a little different for me since I'm using Java on the PC so I'm automatically not using the same libraries. I've also tried with a ruby protobuf library, which was also happy talking to nanopb.

As a general protobuf point, I've found it very helpful to stick with the "standard" delimited form for multiple messages, where the data has a varint encoded length for each message in the data. The Google Java protobuf library supports reading this directly, and encoding it with nanopb is also pretty easy.

I actually found this post looking for anyone with some code to wrap a fatfs file as a nanopb stream, I'll continue my search!

[Dale Peterson]

On Thursday, August 22, 2013 6:27:08 AM UTC-7, Trepi Dacious wrote:

1) I'm guessing the STM32F4 MCU? In that case yes, I've used protobuf in an identical environment (chibios, etc.).

You guessed it. Yeah, Olimex STM32-H407, ChibiOS/RT, fatfs, and nanopb. It was *extremely* helpful to be able to log intermediate data as optional fields without breaking backwards compatibility.
 

2) Yes, I've used nanopb extensively, and it works very well. I've only run into one issue with inter-operation with the Java version of protobuf (and also protoc) - nanopb perfectly reasonably doesn't check that the enum values you use are actually in the enum, whereas some other implementations do. The result of this is not terrible - if you encode with nanopb and decode with such an implementation, the decoder will just ignore the invalid enum values, possibly adding them as unknown fields. In addition you might want to validate your own enums on the MCU before encoding and after decoding, if the results of an invalid value are bad...

In general it's really no harder to use nanopb on the firmware and GPB on the PC than to use the same on both ends, since they both follow the protobuf spec., although it's a little different for me since I'm using Java on the PC so I'm automatically not using the same libraries. I've also tried with a ruby protobuf library, which was also happy talking to nanopb.

Yeah, I agree. I've been looking at the nanopb+stm32+fatfs generated data on my PC using the C++ Google protobuf library. Works very well. I originally implemented things in Python, but it is silly slow compare to C++. I could just be doing something stupid but I saw 100x speedups by using the C++ API directly.
 

As a general protobuf point, I've found it very helpful to stick with the "standard" delimited form for multiple messages, where the data has a varint encoded length for each message in the data. The Google Java protobuf library supports reading this directly, and encoding it with nanopb is also pretty easy.

I will probably do this next time. What I ended up doing was writing 2-byte (LSB, then MSB) message size delimiters prior to each message. In my case, it probably wasn't worth the extra work and I should have gone with the varint apprach. The nice thing about my approach is that I only had to encode the data once and it gets encoded directly into the memory buffer, so there is no double encoding or copying of the message after it is encoded. On the flip side, I had to make my buffers a bit bigger and handle the case where the message "overflows", and copy the overflowing bytes into the next buffer. You can check it out here in case you are interested:

https://github.com/hazelnusse/robot.bicycle/blob/master/firmware/src/sample_buffer.cpp#L42

Such optimizing was probably not necessary for my application, but if you were trying to push the hardware harder, it might be.

 

I actually found this post looking for anyone with some code to wrap a fatfs file as a nanopb stream, I'll continue my search!

Let me know if you find something like that, it would be really cool to have.

Luke