Decode from stream in non-blocking mode

[Christian Baumberger]

Hello

I use nanopb on an embedded platform, where I have one single main-loop (no OS). I am decoding data with nanopb which arrives with tcp packets (lwip). Currently I am using the pb_istream_t functionality with a own defined read-callback where I decode data from the last tcp packet. But what if my encoded profobuf messages are split over multiple tcp packets. My main-loop should not be blocked until all the data is available in the streaming mode. Can I somehow break the current decoding, and continue with other stuff in the main-loop and restart the decoding when new tcp data arrives?

Thanks in advance.

Christian

[Petteri Aimonen]

Hi,

> Can I somehow break the current decoding, and continue with other
> stuff in the main-loop and restart the decoding when new tcp data
> arrives?

Unfortunately no. This would require keeping track of the decoding
state, which is currently done implicitly on the C stack.

With varying level of hackiness, you could break the decoding between
fields in the top-level message. This still requires having enough RAM
to buffer the encoded representation of a single field, so it quite much
reduces the advantage you get from decoding from a stream.

So the possible "solutions" that come to mind are:

- Use RTOS with threads, so you can give own stack to the thread doing
nanopb processing.
- Buffer the whole message or atleast one field in RAM before decoding.
- Call back to your main loop from your IO stream callback.
This will allow other things to be done while waiting, but will
consume more stack.

--
Petteri

[Christian Baumberger]

Hi Petteri

Thank you very much for your response. I think we will just use the blocking mode and not process other things in the meantime. There is one thing that is not clear to me: 

Lets say we have the following protobuf message:

message SampleMessage {

    optional bytes val_bytes = 1;

}

and we dont know how large this repeated field is, therefore I define a field-callback for SampleMessage.val_bytes. On the otherhand I defined a stream-callback like the following: 

pb_istream_t pb_istream_from_socket(int fd)

{

    pb_istream_t stream = {&read_callback, (void*)(intptr_t)fd, SIZE_MAX};

    return stream;

}

static bool read_callback(pb_istream_t *stream, uint8_t *buf, size_t count)

{

    int fd = (intptr_t)stream->state;

    int result;

    

    result = recv(fd, buf, count, MSG_WAITALL);

    

    if (result == 0)

        stream->bytes_left = 0; /* EOF */

    

    return result == count;

}

How is nanopb now handling this situation. Is my stream-callback read_callback called multiple times to collect all the data for all repeated fields and finally the field-callback ist called. Or is the stream-callback just called to collect some data, then the field-callback is called to process this data, then again the stream-callback is called to collect more data, then again the field-callback and so forth. 

Which approach is correct? This is important to me, because I cannot store for example 8MB in my RAM, I would like to process this blockwise.

Thank your for your response.

Best,

Christian

[Petteri Aimonen]

Hi,

> and we dont know how large this repeated field is, therefore I define a

> *field-callback* for SampleMessage.val_bytes.

So, something like:
static bool field_callback(pb_istream_t *stream,
const pb_field_t *field, void **arg);

> On the otherhand I defined a *stream-callback* like the following:

> pb_istream_t stream = {&read_callback, (void*)(intptr_t)fd, SIZE_MAX};

You will be given this stream in your field callback (look at the args).
Except for one thing: the bytes_left value will be modified by nanopb
to the length of the field data, i.e. the field callback cannot read
past the end of the field.

> Or is the stream-callback just called to collect some data, then the
> field-callback is called to process this data, then again the
> stream-callback is called to collect more data, then again the
> field-callback and so forth.

It actually will depend on your field callback. It reads the data by
calling pb_read(). If it doesn't read all of it, it will be called again
by nanopb until all data is consumed. But protobuf arrays are not
necessarily contiguous, so it may be called multiple times even if it
does read all the available data.

--
Petteri