nanopb google proto buf compatibility

[maheshvenk...@gmail.com]

Hi,
I am trying to serialize in nanopb and deserialize in google protobuf python with the addressbook example.

here is my proto file
import "nanopb.proto";

message Person {
required string name = 1 [(nanopb).max_size = 40];
required int32 id = 2;
optional string email = 3 [(nanopb).max_size = 40];

enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}

message PhoneNumber {
required string number = 1 [(nanopb).max_size = 40];
optional PhoneType type = 2 [default = HOME];
}

repeated PhoneNumber phone = 4 [(nanopb).max_count = 5];
}

// Our address book file is just one of these.
message AddressBook {
repeated Person person = 1;
}

and the C program.
Person person = {"test", 1, true, "te...@test.com"};

uint8_t buffer[512];
pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
/* Now encode it and check if we succeeded. */
if (pb_encode(&stream, Person_fields, &person))
{
fwrite(buffer, 1, stream.bytes_written, stdout);
return 0; /* Success */
}
else
{
fprintf(stderr, "Encoding failed: %s\n", PB_GET_ERROR(&stream));
return 1; /* Failure */
}

protoc -I. -I/home/maheshv/nanopb/nanopb/generator -I/usr/include -I../../../gpb/protobuf-2.5.0/src/ -onanodbmahesh.pb nanopb_mahesh.proto
python /home/maheshv/nanopb/nanopb/generator/nanopb_generator.py nanodbmahesh.pb
cc -ansi -Wall -Werror -I. -I.. -g -O0 --coverage -fstack-protector-all -c -o nanopb_pack.o nanopb_pack.c
cc -DPB_BUFFER_ONLY -ansi -Wall -Werror -I. -I.. -g -O0 --coverage -fstack-protector-all -pedantic -Wextra -Wcast-qual -Wlogical-op -Wconversion -c -o nanodbmahesh.pb.o nanodbmahesh.pb.c
cc --coverage nanopb_pack.o pb_decode.o pb_encode.o nanodbmahesh.pb.o -o ./a.out

I run a.out > e

xxd ./e
0000000: 0a04 7465 7374 1001 1a0d 7465 7374 4074 ..test....test@t
0000010: 6573 742e 636f 6d est.com

This file does not get parsed correctly from the python side.

bash-4.1$ python list_people.py ./e
Traceback (most recent call last):
File "list_people.py", line 35, in <module>
address_book.ParseFromString(f.read())
File "/usr/lib/python2.6/site-packages/protobuf-2.5.0-py2.6.egg/google/protobuf/message.py", line 182, in ParseFromString
self.MergeFromString(serialized)
File "/usr/lib/python2.6/site-packages/protobuf-2.5.0-py2.6.egg/google/protobuf/internal/python_message.py", line 795, in MergeFromString
if self._InternalParse(serialized, 0, length) != length:
File "/usr/lib/python2.6/site-packages/protobuf-2.5.0-py2.6.egg/google/protobuf/internal/python_message.py", line 827, in InternalParse
pos = field_decoder(buffer, new_pos, end, self, field_dict)
File "/usr/lib/python2.6/site-packages/protobuf-2.5.0-py2.6.egg/google/protobuf/internal/decoder.py", line 526, in DecodeRepeatedField
raise _DecodeError('Unexpected end-group tag.')
google.protobuf.message.DecodeError: Unexpected end-group tag.

Comparison with another file that parsed correctly

bash-4.1$ xxd ./e
0000000: 0a04 7465 7374 1001 1a0d 7465 7374 4074 ..test....test@t
0000010: 6573 742e 636f 6d est.com

bash-4.1$ xxd ./y
0000000: 0a17 0a04 7465 7374 1001 1a0d 7465 7374 ....test....test
0000010: 4074 6573 742e 636f 6d @test.com

Notice the two extra bytes 0x17 in the y file.

Question : why is the tag not getting generated in nanopb?
Appreciate anh help.
Mahesh

[Petteri Aimonen]

Hi,

Comparing the C program:

> Person person = {"test", 1, true, "te...@test.com"};

> if (pb_encode(&stream, Person_fields, &person))

vs. the Python program:

> File "list_people.py", line 35, in <module>
> address_book.ParseFromString(f.read())

it seems you are writing out a "Person" message, but attempting to read
in an "AddressBook" message.

--

Perhaps part of the confusion is that by default nanopb will generate a
callback field for the AddressBook.person, because it is an unbounded
repeated field. It may not be immediately obvious how to use a callback
field. So for an easy test program, you'll want to make an .options
file with something like:

AddressBook.person max_count:8

which will give you a nice static array that is simple to assign into.

Further reading on callbacks and .options files:
http://koti.kapsi.fi/~jpa/nanopb/docs/concepts.html#field-callbacks
http://koti.kapsi.fi/~jpa/nanopb/docs/reference.html#defining-the-options-in-a-options-file

--
Petteri

[maheshvenk...@gmail.com]

Hi,

Thanks for the quick reply.
I am redirecting the output of the nanopb program to a file and reading it
from python google protobuf.

./a.out > e
python list_people.py ./e

I noticed that the second byte in the google protobuf generated addressbook
(17) in 0x0a17 is the size of the data (23 bytes).
Not sure what 0x0a is.

Are you saying that dumping the output to a file is causing this issue?

[Petteri Aimonen]

Hi,

> Are you saying that dumping the output to a file is causing this issue?

Nope.

The problem is that your C program writes out a "Person" message and the
Python program expects to receive "AddressBook" message. Wrong message
type causes the error.

> I noticed that the second byte in the google protobuf generated addressbook
> (17) in 0x0a17 is the size of the data (23 bytes).
> Not sure what 0x0a is.

0x0a is the field tag: field index 1, wire type 2 (PB_WT_STRING). This
is what begins the field "person" in the AddressBook message.

Because you are writing out only the "Person" message, you get the same
data but without the field tag in the beginning.

Change the C program to encode an "AddressBook" message instead and it
should work.

--
Petteri

[maheshvenk...@gmail.com]

Thanks.

Pardon the dumb question but Is this how it is done?

Person person = {"test", 1, true, "te...@test.com"};

AddressBook address_book ;
memcpy(&address_book.person, &person , sizeof(Person));

[maheshvenk...@gmail.com]

I tried the callback method.

This compiled and ran but I still don't get the tag :(

bool encode_person(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
{
char* name = "test";
char* email = "te...@test.com";

if (!pb_encode_tag_for_field(stream, field)) {
printf(" %s returning false \n", __FUNCTION__);
return false;
}

pb_encode_string(stream, (uint8_t *)name, strlen(name));
pb_encode_varint(stream, 1);
pb_encode_string(stream, (uint8_t *)email, strlen(email));
return true;
}

main() {

AddressBook address_book ;

uint8_t buffer[1024];

pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));

address_book.person.funcs.encode = &encode_person;

/* Now encode it and check if we succeeded. */

if (pb_encode(&stream, AddressBook_fields , &address_book))

{
fwrite(buffer, 1, stream.bytes_written, stdout);
return 0; /* Success */
}
else
{
fprintf(stderr, "Encoding failed: %s\n", PB_GET_ERROR(&stream));
return 1; /* Failure */
}

}

./a.out > e
xxd ./e

0000000: 0a04 7465 7374 010d 7465 7374 4074 6573 ..test..test@tes
0000010: 742e 636f 6d t.com

[Petteri Aimonen]

Hi,

> I tried the callback method.

I'd prefer if you would get the static method working first (i.e. create
the .options file). It is simpler.

> This compiled and ran but I still don't get the tag :(
> bool encode_person(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
> {

> pb_encode_string(stream, (uint8_t *)name, strlen(name));
> pb_encode_varint(stream, 1);
> pb_encode_string(stream, (uint8_t *)email, strlen(email));
> return true;
> }

You should make a Person message and call pb_encode_submessage().
Because the person field is a message, and not just a bunch of separate
strings.

--
Petteri

[Mahesh V]

hi,

I am confused.

do you have a working simple example that I can use?

thanks

Mahesh

--
Petteri

--
You received this message because you are subscribed to a topic in the Google Groups "nanopb" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nanopb/cIhC7iZnBa8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nanopb+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Mahesh V]

bool encode_person(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
{

#if 0

    char* name = "test";
    char* email = "te...@test.com";

#endif

    Person person = {"test", 1, true,  "te...@test.com"};

    if (!pb_encode_tag_for_field(stream, field)) {

        printf(" %s returning false \n", __FUNCTION__);
        return false;
    }

#if 0

    pb_encode_string(stream, (uint8_t *)name, strlen(name));
    pb_encode_varint(stream, 1);
    pb_encode_string(stream, (uint8_t *)email, strlen(email));

#endif

    if (!pb_encode_submessage(stream, Person_fields, &person))
            return false;

    return true;
}

worked thanks a lot !

[Petteri Aimonen]

> do you have a working simple example that I can use?

http://kapsi.fi/~jpa/stuff/other/addressbook.tar.gz

--
Petteri