formatting for sending/receiving nanopb encoded data over serial

[Mike T]

I'm writing a few programs to send and receive data between two MCUs using the MBED platform. A colleague compiled all the libraries, and it's been my task to to handle the MCU portion. I had success encoding and decoding data using the simple encoding and decoding example. My question is regarding how to expand this example to send and receive this data over serial. 

I based what I'm doing largely on this example with a few modifications. I think my send module is working, though it's hard to verify because I haven't been able to receive data and decode it on another MCU. Here's the send code:

Serial pc(PA_2, PA_3);
//some declarations and functions omitted for brevity
int main()

{

    LIS3DH_config();

    while(1)

    {

     //pc.printf("here\r");

     //do accelerometer stuff

     LIS3DH_read_3axis();

     LIS3DH_tilt_angle();    

           

    /* Encode our message */

    {

        /* Allocate space on the stack to store the message data.

         *

         * Nanopb generates simple struct definitions for all the messages.

         * - check out the contents of simple.pb.h! */

        SensorData message;

        

        /* Create a stream that will write to our buffer. */

        pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));

        

        /* Fill in the tilt angle from p (p is a tilt angle) */

        message.tilt_angle = p;

                

        /* Now we are ready to encode the message! */

        status = pb_encode(&stream, SensorData_fields, &message);

        message_length = stream.bytes_written;

        pc.printf("%s\r", buffer);

        //}

        

        /* Then just check for any errors.. */

        if (!status)

        {

            pc.printf("Encoding failed: %s\n", PB_GET_ERROR(&stream));

            return 1;

        }

        /* Now we could transmit the message over network, store it in a file or

        

      }

    wait(.1);

    }

}

 

My main question here is on sending the encoded information. It gets loaded into the buffer using the pb_ostream_from_buffer function, but is pc.printf("%s\r", buffer); an acceptable way to send out the encoded information over serial?

FYI, the data output being sent over serial looks something like this:

K.i...-[.

.&p...-[.

..m...-[.

..m...-[.

3vW...-[.

.QQ...-[.

..Q...-[.

..f...-[.

".h...-[.

WlY...-[.

.(k...-[.

So it looks like some data that is changing is being sent, which is good, as I'm sending a tilt angle from an accelerometer. 

Similarly, for the receive side, here is my code:

#include "mbed.h"

#include <stdio.h>

#include <pb_encode.h>

#include <pb_decode.h>

#include "KV1KV2Comms.pb.h"

#include <string> 

 

Serial pc(PA_2, PA_3);

Serial device(PA_9, PA_10);  // tx, rx

uint8_t buffer[128];

size_t message_length;

bool status;

int i = 0;

int main() {

    /* Allocate space for the decoded message. */

    SensorData message;

    char c;

    

    while(1) {     

        if(device.readable()) {

            /* But because we are lazy, we will just decode it immediately. */

            int i = 0;

            while ((c = device.getc()) != '\r'){

                buffer[i] = c;

                //pc.printf("%c", c);

                

                if(++i > sizeof(buffer)) break;

                

                }

            if (c == '\r'){  

                for(int y = 0; y < sizeof(buffer); y++){

                    pc.printf("%02x ", buffer[y]);

                }

                pc.printf("\r");

                        

                /* Create a stream that reads from the buffer. */

                pb_istream_t stream = pb_istream_from_buffer(buffer, message_length);

        

                /* Now we are ready to decode the message. */

                status = pb_decode(&stream, SensorData_fields, &message);

        

                /* Check for errors... */

                if (!status)

                {

                    pc.printf("Decoding failed: %s\n", PB_GET_ERROR(&stream));

                    return 1;

                }

        

                /* Print the data contained in the message. */

                pc.printf("Your lucky number was %f!\n", message.tilt_angle);

                

            } 

        }

    }

}

This snippet has not successfully read in and decoded data. I am using

 for(int y = 0; y < sizeof(buffer); y++){

                    pc.printf("%02x ", buffer[y]);

                }

to check to see what the data I am receiving and loading into the buffer looks like, and it looks something like this:

5b 1d 5b 1d 1d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 

5b 1d 5b 1d 1d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 

5b 1d 5b 1d 1d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 

5b 1d 5b 1d 1d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 

But when I go through the decoding,

/* Now we are ready to decode the message. */

                status = pb_decode(&stream, SensorData_fields, &message);

        

                /* Check for errors... */

                if (!status)

                {

                    pc.printf("Decoding failed: %s\n", PB_GET_ERROR(&stream));

                    return 1;

                }

        

                /* Print the data contained in the message. */

                pc.printf("Your lucky number was %f!\n", message.tilt_angle);

Nothing happens; the result I see is this:

Your lucky number was 0.000000!

My questions for the receive side are:

1. What is an appropriate way to read data in from the serial port and load it into a buffer so that it plays nicely with the decode function? I didn't follow the examples in the input streams section of the Concepts page

2. Why am I seeing the same data being loaded into the buffer over and over?

3. Why is the data seemingly not being decoded (why am I seeing 0.000000 as my tilt angle)? 

Thanks for the help,
Mike

[Petteri Aimonen]

Hi,

> My main question here is on sending the encoded information. It gets loaded
> into the buffer using the pb_ostream_from_buffer function, but is pc.printf("%s\r",
> buffer); an acceptable way to send out the encoded information over serial?

No, because it is binary data. I.e. it may contain \r inside the data,
so you cannot use it as separator.

Protobuf does not specify a framing protocol, because requirements for
such vary a lot. For example, you may want to add CRC and maybe re-sync
after frame loss.

Here are some common choices:
- Encode as HEX (useful if you want other ASCII data on same port)
- Prefix with length (no error-checking at all)
- Prefix with length, suffix with checksum (error-checking but no
recovery)
- Do byte stuffing to allow for frame delimiters. E.g. HDLC style.
Detects errors and recovers after framing failures.

> size_t message_length;

> pb_istream_t stream = pb_istream_from_buffer(buffer,
> message_length);

I don't see you initializing message_length anywhere. So it will be 0,
and therefore pb_decode() will read 0 bytes from the buffer.

> follow the examples in the input streams section of the Concepts

> <http://koti.kapsi.fi/jpa/nanopb/docs/concepts.html#input-streams> page

Don't worry about it for now, the streams are just a way to conserve RAM
in case of big messages. You can get started with buffers just fine.

--
Petteri

[Mike T]

Petteri,

Thanks for the response. I'm now trying to encode the data and then send it as hex. However, when I do so, The data in buffer doesn't appear to be changing at all. 

SensorData message;

        

        /* Create a stream that will write to our buffer. */

        pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));

        

        /* Fill in the tilt angle from p */

        message.tilt_angle = p;
        //debug so we know what the data is before being encoded

        pc.printf("message contains %f\n", message.tilt_angle);

        

        /* Now we are ready to encode the message! */

        status = pb_encode(&stream, SensorData_fields, &message);

        message_length = stream.bytes_written;

        
        //print out buffer to serial in hex

        pc.printf("%X\r", buffer);

And here is what I'm seeing.

message contains 1.234718

200001D8

message contains 1.252614

200001D8

message contains 1.159460

200001D8

message contains 1.179667

200001D8

message contains 1.251465

200001D8

message contains 1.291126

200001D8

message contains 1.332516

200001D8

message contains 1.294451

200001D8

Clearly, the data being encoded (message.tilt_angle) is changing, but for some reason when I print out buffer in Hex, I just keep getting 200001D8. Any idea why this would be? Thanks again for the help.

-Mike

[Petteri Aimonen]

Hi,

> //print out buffer to serial in hex
> pc.printf("%X\r", buffer);

This is printing out the address of the buffer, not the buffer contents.

--
Petteri

[Mike T]

Thanks for that!

For completeness

        SensorData message;

        

        /* Create a stream that will write to our buffer. */

        pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));

        

        /* Fill in the tilt angle from p */

        message.tilt_angle = p;

        pc.printf("message contains %f\n", message.tilt_angle);

        

        /* Now we are ready to encode the message! */

        status = pb_encode(&stream, SensorData_fields, &message);

        message_length = stream.bytes_written;

        

        //pc.printf("%u\r", status);

        

        //if (pb_encode(&stream, SensorData_fields, &message)) { //encode message

            //new test code:

        //    message_length = stream.bytes_written;

            for (int i = 0; i < sizeof(buffer); i++) {

            pc.printf("%X", buffer[i]);

            }

            pc.printf("\r");

gives me this

message contains 1.112535

D8C678E3F101F2D5B1D0830D9B48040000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

message contains 1.132095

D7BE8903F101F2D5B1D0830D9B48040000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

message contains 1.165736

DDA36953F101F2D5B1D0830D9B48040000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

message contains 1.139276

DCED3913F101F2D5B1D0830D9B48040000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

I'll try the receiver side now.