How to transport protobuffers messages?

[mrlu...@gmail.com]

Hi, I am really new to pb.

I was reading about pb and most of the examples transport the messages over data integrity streams layers (streams, tcp/ip..etc.. )

Considering nanopb for embedded systems, lets say I want to exchange information via serial port.. What would be the best way to make a data integrity protocol to exchange pb data?
Can it be embedded in the pb struct or should be another exchange layer (that can be described also with pb) ?

Thanks for your ideas.

[Petteri Aimonen]

Hi,

> I was reading about pb and most of the examples transport the messages
> over data integrity streams layers (streams, tcp/ip..etc.. )

That is correct, protocol buffers itself does not define any
encapsulation format. This is both an advantage (flexibility) as well as
a drawback (you have to invent your own).

> Considering nanopb for embedded systems, lets say I want to exchange
> information via serial port.. What would be the best way to make a
> data integrity protocol to exchange pb data?

If you trust your serial port not to corrupt data, you could use one of
these simple methods:

1) Encode length as varint, then write the message.
2) Write the message, write 0 byte to terminate.

Otherwise, you'll need some checksum method. I have used the following,
but it is just one possible way. There is no standard:

1. Length (2 bytes = N)
2. Packet type (1 byte)
3. Protobuf message (N bytes)
4. Checksum (1 byte)

where checksum is just 0xFF XOR all bytes that come before it.

--

Extra rambling about the theoretically best way:

The problem with encoding length first is that you need to know it
before you write out the message. In practise this means that either you
need a memory buffer for the whole message or have to compute the length
separately.

I have been contemplating to use a Consistent Overhead Byte Stuffing
method. It would allow small memory usage when used together with
nanopb streams. However the plain COBS uses block size of 255, which is
already quite a lot for many smaller controllers.

However this is all a bit useless from practical point of view, because
if you make the packet format very complicated, everyone else will have
hard time decoding it.

--
Petteri

[Dave Jackson]

I'm using an extra, non-protobuf layer that sends a checksum after the proto buffer message has been transmitted. However, I've also defined my messages as follows:

message RealMessage1 {

    // Fields for a type 1 message

}

message RealMessage2 {

    // Fields for a type 2 message

}

message RootMessage {

    repeated RealMessage1 msg1 = 1;

    repeated RealMessage2 msg2 = 2;

}

RootMessage is a wrapper. It only exists to allow me to stream multiple RealMessages of either type 1, type 2, or mixed. I can choose to send only one RealMessage inside the RootMessage wrapper. 

Note that this requires you to know the size and contents of each RealMessage up front, however you could send a stream of RealMessages and not know how many are in the stream. Also, each RealMessage can be a different size (based upon optional fields or different encoding for different integer values).

I have avoided using 0 to separate messages in a stream as the Google protocol buffer code that runs under Java does not allow messages that end in a 0.

I could also see putting a checksum inside RootMessage as follows:

message RootMessage {

    repeated RealMessage1 msg1 = 1;

    repeated RealMessage2 msg2 = 2;

    uint32 checksum = 3;

}

-Dave