Nanopb and protocol buffers 3.0

[Petteri Aimonen]

Hi,

Google has released the first alphas of protobuf version 3.

There are some quite significant changes they are planning for the protocol.

It appears that some amount of interoperability between proto2 and proto3 will be possible. Nanopb in general favors stability, so proto2 support should remain unaffected for a long time to come. However, proto3 support may be added in parallel, just like in Google's protobuf library.

So here are my thoughts on the changes, and how they will affect nanopb:

1) Removal of field presence logic for primitive value fields, removal of required fields

This seems like a bit strange decision. Suddenly there is no way to know whether a message contained any field at all or not. Instead, any missing scalar field will be initialized to 0. Also zero fields are not encoded in output messages.

It is easy enough to implement, but seems like it may not be so pleasant to use. Submessages will still be detected as present/missing, so maybe they will be used to group optional fields together.

2) removal of default values. 

Default values always was a bit strange feature. Useful, but I'm not sad to see it go.

3) Removal of unknown fields.

Never implemented in nanopb anyway, as storing these would require unbounded RAM.

4) Removal of extensions, which are instead replaced by a new standard type called Any.

Not sure how the "Any" type will work, it does not seem to be implemented yet.

5) Fix semantics for unknown enum values.

No idea what this means. Currently nanopb does not check whether enum values are known or not, while protobuf does.

6) Addition of maps.

Will be interesting to figure out how to map this to C, i.e. what kind of map implementation to choose.

7) Addition of a small set of standard types for representation of time, dynamic data, etc.

I hope these will be expressed in .proto format, so that they can parsed as regular submessages also.

8) A well-defined encoding in JSON as an alternative to binary proto encoding.

This might be useful, if only for printing the messages for debugging purposes.

--

Petteri

[Wink Saville]

So Google recently announced alpha2 for proto3, have you made any headway on support proto3 in Nanobp?

-- Wink

[Petteri Aimonen]

Hi,

> So Google recently announced alpha2 for proto3

> <https://github.com/google/protobuf/releases/tag/v3.0.0-alpha-2>, have you

> made any headway on support proto3 in Nanobp?

No, not so far.

If people have any features in proto3 that they would especially like to
see in nanopb, please reply here or add feature requests on the tracker.
There is always a lot of features to implement, but I tend to focus on
those that are actually wanted by someone.

Seems like a half of the "new features" in proto3 are just removal of
old features :)

--
Petteri

[csaba....@gmail.com]

Removing default values is a big loss, by the way...

In embedded environment it was a killer feature. A very nice way of compressing configuration files. Let's say you have a big configuration which you need to store in the EEPROM. There are many variants, but there is only a very little difference between the variants... In the EEPROM you only need to store the difference using protobuf...

Second killer feature is the upward and backward compatibility which was possible to achieve with the smart usage of default values.

Basically default values and the key-value serialization idea are the two features we are bonded to protocol buffers.

[Petteri Aimonen]

Hi,

> Removing default values is a big loss, by the way...

Hmm, maybe a reasonable long term plan would be to:

1) Switch default value implementation in nanopb to only use the
MyMessage_init_default #defines. This way there is no runtime
cost to the support.

2) Add a new nanopb-specific option for defining default values
when using proto3 format.

Of these 1) is something that has been in planning for some time,
as issue 114. 2) is pretty easy to implement, so no reason not to
do it as default values can indeed be useful sometimes.

--
Petteri

[csaba....@gmail.com]

Thanks for reply, 2) sounds like a solution.

[Csaba Purszki]

Hi Petteri,

I migrating from 0.2.3 to 0.3.3.

Look at this message, I have a compiler error.

Message Foobar {

optional int32 a = 2 [default = 12];

}

<Filename> : error C2065: 'Foobar_a_default' : undeclared

identifier

Compiling with Visual Studio 2008. I'm compiling nanopb from source.

Do you have any idea? Maybe a MACRO to set in the build-system?

Regards,

Csaba

--
You received this message because you are subscribed to a topic in the Google Groups "nanopb" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nanopb/1di-Ua50N3Q/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nanopb+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Petteri Aimonen]

Hi,

> <Filename> : error C2065: 'Foobar_a_default' : undeclared
> identifier

Please attach the generated .pb.h and .pb.c files and the actual
error message with file name and line number and I'll take a look.

--
Petteri

[Petteri Aimonen]

Hi,

> > <Filename> : error C2065: 'Foobar_a_default' : undeclared
> > identifier

This was resolved off the list, was due to custom python script.

--
Petteri

[James Parker]

Petteri et al,

Do we know what the JSON wire format specs are yet?

I have experimented (and got a working version of) NanoPB with JSON encoding/decoding which I would like to share with everyone but it may become out of date /less useful if the "official" format is different.

One the problems you face when using a "trivial" JSON mapping (as I had done) is that repeated fields were appearing as JSON arrays - and this means there is no way to know how long that field may get when you begin parsing. This can make processing and allocating memory during deserialisation problematic - repeatedly calling realloc is not good! The main technique I used to ensure this was running well was to force all repeated field to have a max count - then we just have a static length arrays in all structs.

For those on *very* tight ROM constraints JSON is a pain because each new message necessarily has to store the field names - not just the tag ID's - in the message descriptions. But it's not too bad. I'm mostly working with 512kB - 1MB ROM and 64kB - 128kB RAM.

For what, it's worth I had also made some good headway into a lightweight C++ wrapper for nanopb for anyone who might be interested - very cool for maintaining optional fields and repeated field counts automatically :) I tried to keep the generated C code identical but wrap the structs in a class that could manipulate and stream them.

If there is enough interest I'm happy to start a branch - this work has lay dormant for about a year now. I think it was based on 2.7 or 2.6.

Anyone doing a web-based interface on a low powered device might find this useful :)

Thanks,

James.

[Petteri Aimonen]

Hi,

> Do we know what the JSON wire format specs are yet?

Last time I checked I couldn't find any specs on protobuf 3.0.

> deserialisation problematic - repeatedly calling realloc is not good

Well, it is somewhat slow but it is not particularly bad if you are
reallocing the latest allocated memory block. If it happens to already
be at the beginning of a big enough free area, it can just grow; if it
isn't, it'll move around until it is. Anyway no fragmentation trouble
like when reallocing several blocks alternately.

> For those on *very* tight ROM constraints JSON is a pain because each new
> message necessarily has to store the field names - not just the tag ID's -
> in the message descriptions. But it's not too bad. I'm mostly working with
> 512kB - 1MB ROM and 64kB - 128kB RAM.

Yeah, storing field names is indeed required for JSON. #ifdefing them
out should be enough.

> For what, it's worth I had also made some good headway into a lightweight
> C++ wrapper for nanopb for anyone who might be interested - very cool for
> maintaining optional fields and repeated field counts automatically :) I
> tried to keep the generated C code identical but wrap the structs in a
> class that could manipulate and stream them.

I would be interested to take a look. It's something that is worth
doing, but also something that would be nice to get right before it
comes a compatibility and maintainability burden.

--
Petteri

[Michael Haberler]

> Am 17.04.2015 um 11:13 schrieb James Parker <jamespa...@gmail.com>:
> ...

> For what, it's worth I had also made some good headway into a lightweight C++ wrapper for nanopb for anyone who might be interested - very cool for maintaining optional fields and repeated field counts automatically :) I tried to keep the generated C code identical but wrap the structs in a class that could manipulate and stream them.
>
> If there is enough interest I'm happy to start a branch - this work has lay dormant for about a year now. I think it was based on 2.7 or 2.6.

I would be interested in this as well!

regards

Micahel

>
> Anyone doing a web-based interface on a low powered device might find this useful :)
>
> Thanks,
> James.
>
>
>

> --
> You received this message because you are subscribed to the Google Groups "nanopb" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to nanopb+un...@googlegroups.com.

[James Parker]

Hi guys,

Just revisiting my JSON and C++ changes to nanopb that I wanted resurrect:

C++ changes are all handled in the generator python script so easy to switch on or off - it will sit alongside existing generated C code.

However, the JSON stuff requires a little more refactoring in the code.

Up until now, I have just had a copy of the nanopb source files and have modified them for my own project but to share this meaningfully with you all I will need a bit of time to tidy this up. 

I could really do with help getting my environment into a place where I can build your distributions - I just don't seem to be able to get bbfreeze or any of that stuff in a good working order.

Can you recommend any forum post? or drop me some hints?

Thanks!

James.

[Petteri Aimonen]

Hi,

> I could really do with help getting my environment into a place where I can
> build your distributions - I just don't seem to be able to get bbfreeze or
> any of that stuff in a good working order.
> Can you recommend any forum post? or drop me some hints?

Hmm yeah, that's a bit though. And something I should document, if only
for myself :)

Linux is the simplest:
1) Install the protobuf-compiler, python and scons.
2) Install bbfreeze using "easy_install bbfreeze"
3) Go to top nanopb directory and run:
tools/make_linux_package.sh

Windows differs only in installation of tools:
1) I have installed protoc from the binary packages distributed by
Google.
2) Python from python.org, I think scons and bbfreeze were installed
with easy_install.
3) Git bash or MSYS bash is required to run the script:
tools/make_windows_package.sh
4) The windows packaging script hardcodes some dll paths, this may or
may not cause trouble..

Mac OS X was a bit more problematic for me, though I cannot remember
details:
1) Protobuf-compiler etc. from homebrew. Installing homebrew required
digging up some Xcode command line tools which somehow didn't work
on first install for me. Well, google gave some hints and eventually
it worked.
2) Pyinstaller from somewhere, probably installed from source.
3) Script should run fine under Mac's bash:
tools/make_mac_package.sh

These were from memory, so probably a lot missing. Please report back if
you try it out, I'll probably remember the solutions once you remind me
of the problems :)

If it helps, you can see the working version console outputs here:
http://lakka.kapsi.fi:50140/job/nanopb-binary-windows/102/console
http://lakka.kapsi.fi:50140/job/nanopb-binary-linux/98/console

--
Petteri

[James Parker]

If it helps, you can see the working version console outputs here:
http://lakka.kapsi.fi:50140/job/nanopb-binary-windows/102/console
http://lakka.kapsi.fi:50140/job/nanopb-binary-linux/98/console

--
Petteri

Thanks - I'll give that a go! 

[Wink Saville]

So what's the current state of Proto3 support?

[ni...@automatic.com]

On Thursday, October 8, 2015 at 12:00:26 PM UTC-7, Wink Saville wrote:
> So what's the current state of Proto3 support?

I came to ask exactly the same question!

protoc has gone beta1 as far as I can tell - I'm starting to add in hacks to support these new proto files now (happy to contribute them back of course, but I fear I don't have time to fully grok everything to make them viable patches).

Cheers,

N

[Petteri Aimonen]

Hi,

> So what's the current state of Proto3 support?

Nanopb builds and works fine against protobuf3.0 compiler,
but none of the proto3 new features have been added, and
none of the old features have been removed.

--
Petteri

[Wink Saville]

I'm looking at designing a new system in C and I thought I'd use proto3 support to start on the "right track". Should we create a branch and start working on supporting proto3 and start reviewing Nick's patches?

[Petteri Aimonen]

Hi,

> I'm looking at designing a new system in C and I thought I'd use proto3
> support to start on the "right track". Should we create a branch and start
> working on supporting proto3 and start reviewing Nick's patches?

Could work. It would be great if proto3 support can be added
step-by-step, for example:

1) First making sure that all the common features already in proto2 work
also with proto3.

2) Figuring out what to do about the new default values model in proto3.
Probably also proto2 default values should be decoupled from the decoder
(issue #114). Then the decoder would always just initialize fields to
zero, and initialization to proto2 default values would go through the
generated initializer values (MyMessage_init_default).

3) Adding support for Map types. Need to find a reasonable C
representation for a map. I think that ordered list and binary search is
probably the most appropriate, as many arrays in embedded use will be
relatively small.

4) Adding support for Any types, seems to be work-in-progress in
official protobuf also. Basically very similar how extensions work now.

5) Adding support for json encoding and decoding. This should probably
be a separate module, pb_json.c or something. It would be useful for
proto2 also.

--
Petteri