Writing multiple messages to a stream

[Dale Peterson]

I am modifying some firmware that logged data directly into a C array of structs.  Once the array was halfway filled, I would call fwrite() to write the first half of the array to file in a binary format, and then continue populating the second half.  Once the second half was filled I would then begin writing it to file and start logging data into the first half, and so on.

I have read through the examples and various posts on the mailing list but am unclear how I should be using pb_ostream_from_buffer() and pb_encode() when writing multiple messages to the same stream.  A few questions:

1) If I call pb_encoder() on the same stream more than once, does it take care of incrementing a pointer to the correct location in the buffer passed to pb_ostream_from_buffer()?

2) Can I simply accumulate stream.bytes_written after each call to pb_encode(), and then trigger a filesystem write with that number of bytes when I get "close" to the end of the first half of the buffer?

3) Do I need to close the stream in any way before I call pb_ostream_from_buffer() again?

4) Do I need to do anything in between each call to encode as far as separating messages go?  Or will the decode functionality just be able to detect where each message starts and ends?  In my previous code, all my structs were fixed size so I didn't have to deal with this issue, I'm hoping nanopb works the similarly in that regard.

Thanks,

Dale

[Petteri Aimonen]

Hi,

> Once the second half was filled I would then begin writing it to file
> and start logging data into the first half, and so on.

So a double buffer. Yeah, seems quite normal.

> 1) If I call pb_encoder() on the same stream more than once, does it take
> care of incrementing a pointer to the correct location in the buffer passed
> to pb_ostream_from_buffer()?

Yeah, the stream remembers where it left off.

> 2) Can I simply accumulate stream.bytes_written after each call to
> pb_encode(), and then trigger a filesystem write with that number of bytes
> when I get "close" to the end of the first half of the buffer?

Yes.

Defining 'close' is difficult of course. You could either hardcode
something, or implement your own pb_ostream callback that would handle
the switch at the exact byte position when the first buffer fills up.

> 3) Do I need to close the stream in any way before I call
> pb_ostream_from_buffer() again?

No. Just don't use the old stream anymore after you begin a new one.

> 4) Do I need to do anything in between each call to encode as far as
> separating messages go?

Yes.

If you use nanopb for decoding also, you can simply write a '0' byte
between the messages, e.g. pb_write(&stream, (const uint8_t*)"\0", 1);

However, other decoders don't usually support zero-delimited messages.
So for them you may need to do something more complex, like writing the
length of the message before each message.

If your purpose is to write a log file in nanopb and read it in
something else, I would probably do this:

message LogMessage {
time, date, whatever..
}

message LogFile {
repeated LogMessage messages = 1;
}

Thanks to the way the protocol buffers format works, if you encode
multiple LogFiles after each other without separators, they'll all be
combined into a single message. It's actually fully specified operation
and is called 'merging' in Google's protobuf documentation.

--
Petteri

[Dale Peterson]

However, other decoders don't usually support zero-delimited messages.
So for them you may need to do something more complex, like writing the
length of the message before each message.

Is there a recommended byte order if message length prefixes exceed one byte?

 

If your purpose is to write a log file in nanopb and read it in
something else, I would probably do this:

message LogMessage {
    time, date, whatever..
}

message LogFile {
    repeated LogMessage messages = 1;
}

Thanks to the way the protocol buffers format works, if you encode
multiple LogFiles after each other without separators, they'll all be
combined into a single message. It's actually fully specified operation
and is called 'merging' in Google's protobuf documentation.

Just to confirm, you did mean multiple LogFiles, not multiple LogMessages, correct? So something like:

Logfile f;

... // populate f.messages

 pb_encode(&stream, LogFile_fields, &f);

... // change f.messages

pb_encode(&stream, LogFile_fields, &f);

And then on the receiving side, do exactly what is done in the tutorial for reading the message [0]?  I guess consideration would need to be made about the max size of the repeated field or whether dynamic memory allocation is ok on the embedded side?

Thanks,

Luke

[0] -- https://developers.google.com/protocol-buffers/docs/cpptutorial

--
Petteri

[Petteri Aimonen]

Hi,

> Is there a recommended byte order if message length prefixes exceed one
> byte?

Often people use varints, i.e. pb_encode_varint(). However, if you use
the below method you probably don't need to separately encode the size.

> Just to confirm, you did mean multiple LogFiles, not multiple LogMessages,
> correct? So something like:
>
> Logfile f;
> ... // populate f.messages
> pb_encode(&stream, LogFile_fields, &f);
> ... // change f.messages
> pb_encode(&stream, LogFile_fields, &f);

Yes. You can add (nanopb).max_count = 1 (or whatever) so that you get a
static array that is easy to populate.

> And then on the receiving side, do exactly what is done in the tutorial for
> reading the message [0]? I guess consideration would need to be made about
> the max size of the repeated field or whether dynamic memory allocation is
> ok on the embedded side?

On the receiving side, you would read only one LogFile, and the messages
array will contain all the messages in proper order.

--
Petteri