Protobuf Buffers v3.0.0-alpha-1

[Feng Xiao]

Hi all,

I just published protobuf v3.0.0-alpha-1 on our github site:

https://github.com/google/protobuf/releases/tag/v3.0.0-alpha-1

This is the first alpha release of protobuf v3.0.0. In protobuf v3.0.0, we will add a new protobuf language version (aka proto3) and support a wider range of programming languages (to name a few: ruby, php, node.js, objective-c). This alpha version contains C++ and Java implementation with partial proto3 support (see below for details). In future releases we will add support for more programming languages and implement the full proto3 feature set. Besides proto3, this alpha version also includes two other new features: map fields and arena allocation. They are implemented for both proto3 and the old protobuf language version (aka proto2).

We are currently working on the documentation of these new features and when it's ready it will be updated to our protobuf developer guide. For the time being if you have any questions regarding proto3 or other new features, please post your question in the discussion group.

CHANGS

=======

Version 3.0.0-alpha-1 (C++/Java):

  General

  * Introduced Protocol Buffers language version 3 (aka proto3).

    When protobuf was initially opensourced it implemented Protocol Buffers

    language version 2 (aka proto2), which is why the version number

    started from v2.0.0. From v3.0.0, a new language version (proto3) is

    introduced while the old version (proto2) will continue to be supported.

    The main intent of introducing proto3 is to clean up protobuf before

    pushing the language as the foundation of Google's new API platform.

    In proto3, the language is simplified, both for ease of use and  to

    make it available in a wider range of programming languages. At the

    same time a few features are added to better support common idioms

    found in APIs.

    The following are the main new features in language version 3:

      1. Removal of field presence logic for primitive value fields, removal

         of required fields, and removal of default values. This makes proto3

         significantly easier to implement with open struct representations,

         as in languages like Android Java, Objective C, or Go.

      2. Removal of unknown fields.

      3. Removal of extensions, which are instead replaced by a new standard

         type called Any.

      4. Fix semantics for unknown enum values.

      5. Addition of maps.

      6. Addition of a small set of standard types for representation of time,

         dynamic data, etc.

      7. A well-defined encoding in JSON as an alternative to binary proto

         encoding.

    This release (v3.0.0-alpha-1) includes partial proto3 support for C++ and

    Java. Items 6 (well-known types) and 7 (JSON format) in the above feature

    list are not implemented.

    A new notion "syntax" is introduced to specify whether a .proto file

    uses proto2 or proto3:

      // foo.proto

      syntax = "proto3";

      message Bar {...}

    If omitted, the protocol compiler will generate a warning and "proto2" will

    be used as the default. This warning will be turned into an error in a

    future release.

    We recommend that new Protocol Buffers users use proto3. However, we do not

    generally recommend that existing users migrate from proto2 from proto3 due

    to API incompatibility, and we will continue to support proto2 for a long

    time.

  * Added support for map fields (implemented in C++/Java for both proto2 and

    proto3).

    Map fields can be declared using the following syntax:

      message Foo {

        map<string, string> values = 1;

      }

    Data of a map field will be stored in memory as an unordered map and it

    can be accessed through generated accessors.

  C++

  * Added arena allocation support (for both proto2 and proto3).

    Profiling shows memory allocation and deallocation constitutes a significant

    fraction of CPU-time spent in protobuf code and arena allocation is a

    technique introduced to reduce this cost. With arena allocation, new

    objects will be allocated from a large piece of preallocated memory and

    deallocation of these objects is almost free. Early adoption shows 20% to

    50% improvement in some Google binaries.

    To enable arena support, add the following option to your .proto file:

      option cc_enable_arenas = true;

    Protocol compiler will generate additional code to make the generated

    message classes work with arenas. This does not change the existing API

    of protobuf messages and does not affect wire format. Your existing code

    should continue to work after adding this option. In the future we will

    make this option enabled by default.

    To actually take advantage of arena allocation, you need to use the arena

    APIs when creating messages. A quick example of using the arena API:

      {

        google::protobuf::Arena arena;

        // Allocate a protobuf message in the arena.

        MyMessage* message = Arena::CreateMessage<MyMessage>(&arena);

        // All submessages will be allocated in the same arena.

        if (!message->ParseFromString(data)) {

          // Deal with malformed input data.

        }

        // Must not delete the message here. It will be deleted automatically

        // when the arena is destroyed.

      }

    Currently arena does not work with map fields. Enabling arena in a .proto

    file containing map fields will result in compile errors in the generated

    code. This will be addressed in a future release.

=======

Thanks,

Feng

[chai2010]

Feng Xiao,

I have some questions:

1. does protobuf3 will include golang compiler?

2. does protobuf3 have a spec doc (link?) ?

Thanks.

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.
To post to this group, send email to prot...@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

--

https://github.com/chai2010

[Vladimir Agafonkin]

Really awesome!

One question about maps — how are they encoded in terms of packed size? How does it compare to just using a repeated message with key/value pairs?

[Feng Xiao]

On Wed, Dec 10, 2014 at 9:19 PM, chai2010 <chais...@gmail.com> wrote:

Feng Xiao,

I have some questions:

1. does protobuf3 will include golang compiler?

Go protobuf is in its own repository and proto3 will supported there. See:

https://github.com/golang/protobuf

 

2. does protobuf3 have a spec doc (link?) ?

We are working on the documentation right now and when it's ready we will publish it on the protobuf developer guide:

https://developers.google.com/protocol-buffers/docs/overview

[Feng Xiao]

On Thu, Dec 11, 2014 at 9:14 AM, Vladimir Agafonkin <agaf...@gmail.com> wrote:

Really awesome!

One question about maps — how are they encoded in terms of packed size? How does it compare to just using a repeated message with key/value pairs?

Map fields are encoded as a repeated message field. The following two definitions generate the same wire format:

message A {

  map<string, string> values = 1;

}

message B {

  message MapEntry {

    option map_entry = true;

    string key = 1;

    string value = 2;

  }

  repeated MapEntry values = 1;

}

If you add map fields to a proto file, languages that haven't supported map fields will still be able to generate code for this proto file and can also use it to communicate with other languages that have maps implemented.

 

--

[Michael Haberler]

Hallo Feng,

> Am 11.12.2014 um 05:51 schrieb Feng Xiao <xiao...@google.com>:
>
> Hi all,
>
> I just published protobuf v3.0.0-alpha-1 on our github site:
> https://github.com/google/protobuf/releases/tag/v3.0.0-alpha-1

a question on structuring web applications further downstream:


you mention node.js and JSON (de)serialisation

reading between the lines this would suggest to me a typical protobuf application talking to a web client would talk JSON-serialized protobuf (maybe over a websocket stream)

I've used this scheme and while JSON is easy for browser js engines, there are downsides; for instance, (de)serializing doubles from/to JSON usually creates conversion/rounding fuzz - that precludes signing a protobuf object in binary representation because the signature generally wont be valid after JSON conversion. Looser type checking is another drawback.

That is why I found an end-to-end binary protobuf transfer and client-side js bindings along the lines of https://github.com/dcodeIO/ProtoBuf.js more robust

what's the grand vision here - how are web apps going to talk to protobuf API's server-side?

thanks in advance,

Michael

[Vladimir Agafonkin]

Oh, bummer. I was hoping for a more compact map packing. Currently I'm using the following to do this:

repeated uint32 properties = 1; // key/value index pairs

repeated string keys = 2; // unique keys

repeated string values = 3; // unique values

[Feng Xiao]

On Thu, Dec 11, 2014 at 12:02 PM, Michael Haberler <mai...@mah.priv.at> wrote:

Hallo Feng,


> Am 11.12.2014 um 05:51 schrieb Feng Xiao <xiao...@google.com>:
>
> Hi all,
>
> I just published protobuf v3.0.0-alpha-1 on our github site:
> https://github.com/google/protobuf/releases/tag/v3.0.0-alpha-1

a question on structuring  web applications further downstream:


you mention node.js and JSON (de)serialisation

reading between the lines this would suggest to me a typical protobuf application talking to a web client would talk JSON-serialized protobuf (maybe over a websocket stream)

I've used this scheme and while JSON is easy for browser js engines, there are downsides; for instance, (de)serializing doubles from/to JSON usually creates conversion/rounding fuzz - that precludes signing a protobuf object in binary representation because the signature generally wont be valid after JSON conversion. Looser type checking is another drawback.

That is why I found an end-to-end binary protobuf transfer and client-side js bindings along the lines of https://github.com/dcodeIO/ProtoBuf.js more robust

what's the grand vision here - how are web apps going to talk to protobuf API's server-side?

+liujisi, who is more qualified to answer this than me.

I think your reading is correct. We'll publish an protobuf implementation for node.js but not for the Javascript language in general (like the ProtoBuf.js you linked), while JSON format support will be added to all protobuf implementations. Web apps would talk JSON to its server although the server can support both JSON format and protobuf wire format.

As far I know, protobuf wire format does not have much of an advantage over JSON format on web apps because the payload is usually small enough and encoding/decoding protobuf wire format with Javascript does not necessarily have a better performance than the built-in JSON encoder/decoder. As most web apps are already using JSON as the data exchange format, supporting JSON format on the sever side so it can talk with JSON clients seems a natural choice here.

[Jeremy Swigart]

Does the arena allocator also get used by messages allocated as children of the root message?

[Feng Xiao]

On Fri, Dec 12, 2014 at 5:56 AM, Jeremy Swigart <jswi...@gmail.com> wrote:

Does the arena allocator also get used by messages allocated as children of the root message?

Yes.

[Vladimir Agafonkin]

Hi Feng,

 

As far I know, protobuf wire format does not have much of an advantage over JSON format on web apps because the payload is usually small enough and encoding/decoding protobuf wire format with Javascript does not necessarily have a better performance than the built-in JSON encoder/decoder.

Protobuf has a big advantage over JSON for some web apps. In particular, when you are transferring big amounts of data. In our use case (client-side WebGL vector maps, with lots of numeric data on the wire), Protobuf provides about 2-3 smaller gzipped sizes compared to JSON. In addition, it is much faster to decode — `JSON.parse` parses the data all at once due to the arbitrarily nested nature of JSON, while with Protobuf, you can parse data sequentially, chunk by chunk.

[Sumit Kumar]

Does Arena allocator support custom memory pool override ? Can NUMA aware memory pools be used instead ?

Any change to the set_allocated ?

Regards,

Sumit Kumar

[Feng Xiao]

On Mon, Dec 15, 2014 at 8:33 PM, Sumit Kumar <Kumar...@hotmail.com> wrote:

Does Arena allocator support custom memory pool override ? Can NUMA aware memory pools be used instead ?

No. We can consider adding support for custom memory pools, but I think it might be very hard. The current arena code has its built-in memory allocation mechanism and is highly optimized for that. Supporting custom memory pools will probably require a rewrite of arena code.

 

Any change to the set_allocated ?

We added a new generated method "unsafe_arena_set_allocated" which behaves the same way as the old "set_allocated". The new "set_allocated" implementation will check whether the passed-in object is on the same arena of the containing message and if it's not, a copy will be made.

[Arjun Satish]

Hey guys,

Thanks for all the hard work!

I have a question regarding the decision to drop support for default values. Fields which are set to their default values are not serialized. I noticed that in the new code (3.0.0-alpha-1 for Java),  this condition still holds true. But the default values used are the standard ones (0 for int64/int32 etc) and cannot be specified in the .proto file. In some of my code, I had reasons to use non-zero default values (-1 for some integers, 1024 for some others, 3.14 for some doubles etc). Using the old protocol buffers, this was trivial to implement. This was a great feature as we could save atleast 2 bytes for every "untouched" field (which comes in handy when we persist the data :-)).

Is there any way we can retain specification of default values in the .proto files and using them in the generated encoders/decoders?

Thanks very much!

Looking forward to the 3.0 release!

Best,
Arjun Satish

[Arjun Satish]

Did anyone get a chance to look at this request?

[Feng Xiao]

+liujisi, who should have a better idea of why default value is dropped from proto3 and what alternatives users can rely on.

Internally the design of proto3 has been discussed among a group of people for quite a long time, but most of them haven't subscribed this forum though...

--

[Arjun Satish]

Would it be possible to re-introduce this feature in a subsequent release? It seems like you are still using it under-the-hood. And because of the benefits I mentioned above, I strongly feel that it will only help the community.

Best,

[Feng Xiao]

On Mon, Jan 12, 2015 at 10:40 PM, Arjun Satish <arjun....@gmail.com> wrote:

Would it be possible to re-introduce this feature in a subsequent release? It seems like you are still using it under-the-hood.

In C++/Java/Python where we support both proto2 and proto3, default values will continue to exist. In new languages (e.g., ruby) though, the support for non-zero default values will be dropped completely.

 

And because of the benefits I mentioned above, I strongly feel that it will only help the community.

As far as I know, the decision is final. Internally a lot Google projects have already adopted the new syntax and so far we have not heard problems caused by disallowing default values. It's unlikely this will be changed in the future. The omission of this feature (and other features) is to make the language simpler and to allow more idiomatic implementations in a wider range of languages. It's believed this decision will help the protobuf community (both protobuf maintainers and protobuf users) and we expect proto3 to be a version that can be more easily adopted than proto2 by new users due to these simplifications. For existing users who rely on removed features, they can continue to use proto2 and that will be supported for a long time (if not forever). Currently we generally do not recommend migrating existing proto2 projects to proto3 because of incompatibility issues (e.g., extensions are dropped in proto3) and only recommend new users to use proto3.

[Arjun Satish]

Feng,

What do you mean when you say "In C++/Java/Python where we support both proto2 and proto3, default values will continue to exist"? When I run protoc (v3) with the syntax="proto3" tag, it shows an error "Explicit default values are not allowed in proto3." and exits (no code is generated). This does not let me use other proto 3 features if my proto definition file contain default values.

Thanks for your timely responses! Highly appreciate it!

[Feng Xiao]

On Tue, Jan 13, 2015 at 3:05 PM, Arjun Satish <arjun....@gmail.com> wrote:

Feng,

What do you mean when you say "In C++/Java/Python where we support both proto2 and proto3, default values will continue to exist"?

What I meant is that you can still find its traces in the implementation but the feature itself is not exposed publicly in proto3 (i.e., we are still using it under-the-hood).

[Arjun Satish]

Since this feature is never going to be exposed, what advice do you have for people who are using this feature and want to migrate to v3? Also, what can we do as users to skip encoding certain fields with the new library?

Thanks,

[Jeremy Swigart]

That sounds like a poor design decision, and one easily readded without breaking anything. If a field doesn't have an explicit default, you use 0 or whatever, thereby not breaking anyone not using them, but if an explicit default is provided that is used instead. I am using that feature as well.

[Alex Antonov]

I fully second that opinion.  We rely a lot on being able to set explicit defaults that are not language defaults (Java 0, "", false, etc).  It puzzles me to even think as to why someone might want to take that feature away!!!

[Feng Xiao]

On Tue Jan 13 2015 at 3:17:26 PM Arjun Satish <arjun....@gmail.com> wrote:

Since this feature is never going to be exposed, what advice do you have for people who are using this feature and want to migrate to v3? Also, what can we do as users to skip encoding certain fields with the new library?

We are aware of the migration difficulties from v2 to v3 due to removed features. As such we do not encourage existing users to migrate and commit to continue the support for proto2 syntax.

You could have both proto2 and proto3 syntax files in your project and they are allowed to import each other despite the syntax differences.

[Feng Xiao]

On Thu Jan 15 2015 at 7:01:33 AM Alex Antonov <aan...@gmail.com> wrote:

I fully second that opinion.  We rely a lot on being able to set explicit defaults that are not language defaults (Java 0, "", false, etc).  It puzzles me to even think as to why someone might want to take that feature away!!!

The decision is made to make protobuf easier to implement efficiently in a lot of other languages that we want to support and for some features like JSON that we are going to add. I think we can all agree that default value is a useful feature, but it's not a mandatory feature to protobuf and creates real problems in the implementation of some languages (For example some languages may use a simple struct to represent a protobuf message and there isn't a way to specify default values. Or in the case of JSON, a client might have no way to know the non-0 default values.)

 

On Wednesday, January 14, 2015 at 6:50:37 AM UTC-6, Jeremy Swigart wrote:

That sounds like a poor design decision, and one easily readded without breaking anything. If a field doesn't have an explicit default, you use 0 or whatever, thereby not breaking anyone not using them, but if an explicit default is provided that is used instead. I am using that feature as well.

--

[Arjun Satish]

Feng,

Would it not be better to throw errors/exceptions when you try to serialize to JSON (from languages like C++ or Java) or when code is generated for these particular languages, rather than completely remove the feature across the board?

--
You received this message because you are subscribed to a topic in the Google Groups "Protocol Buffers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/protobuf/ZRpcfmeGK6s/unsubscribe.
To unsubscribe from this group and all its topics, send an email to protobuf+u...@googlegroups.com.

[V.B.]

Can I ask for more details about why presence logic was removed (e.g. hasFoo() ) for primitives? This has been a very useful feature for us.

[Feng Xiao]

The reason for dropping field presence is more of the same with dropping default values. Basically we want to simplify protobuf and make it easier to implement efficiently in more languages. We are preparing the proto3 documentation and will share more information about the trade-offs we have made.

--

[Feng Xiao]

On Thu Jan 15 2015 at 1:10:10 PM Arjun Satish <arjun....@gmail.com> wrote:

Feng,

Would it not be better to throw errors/exceptions when you try to serialize to JSON (from languages like C++ or Java) or when code is generated for these particular languages, rather than completely remove the feature across the board?

My gut feeling is "no, it would not make it better".

Proto3 is designed as a new protobuf language version with support for a wider range of programming languages. A consistent feature set is important for the interchangeability between languages. I don't think we'll make the proto3 support in C++/Java have more features than other languages (or vice versa).

[V.B.]

I suppose what I'm really wondering is:
a) How does it simplify the language implementations exactly?
b) Why was that not the case for non-primitives, which still have presence logic?

[elodg]

Interesting changes in proto3. I checked out alpha-1, and tried building it in VS2013. Gtest is missing from this release, I had to copy it from 2.6.0, but managed to build it in the end. Finally each project has its own build folder.
The following 3 tests fail:
[  FAILED  ] BootstrapTest.GeneratedDescriptorMatc
[  FAILED  ] TextFormatTest.Basic
[  FAILED  ] TextFormatExtensionsTest.Extensions

It might have been asked before, but is writeDelimitedTo for c++ slated to be included in 3.0.0? Also, is there a release roadmap for the new version and documentation?

Cheers,
Elod

On Thursday, December 11, 2014 at 6:51:01 AM UTC+2, Feng Xiao wrote:

Hi all,

I just published protobuf v3.0.0-alpha-1 on our github site:

https://github.com/google/protobuf/releases/tag/v3.0.0-alpha-1

This is the first alpha release of protobuf v3.0.0. In protobuf v3.0.0, we will add a new protobuf language version (aka proto3) and support a wider range of programming languages (to name a few: ruby, php, node.js, objective-c). This alpha version contains C++ and Java implementation with partial proto3 support (see below for details). In future releases we will add support for more programming languages and implement the full proto3 feature set. Besides proto3, this alpha version also includes two other new features: map fields and arena allocation. They are implemented for both proto3 and the old protobuf language version (aka proto2).

We are currently working on the documentation of these new features and when it's ready it will be updated to our protobuf developer guide. For the time being if you have any questions regarding proto3 or other new features, please post your question in the discussion group.

CHANGS

=======

Version 3.0.0-alpha-1 (C++/Java):

  General

  * Introduced Protocol Buffers language version 3 (aka proto3).

    When protobuf was initially opensourced it implemented Protocol Buffers

    language version 2 (aka proto2), which is why the version number

    started from v2.0.0. From v3.0.0, a new language version (proto3) is

    introduced while the old version (proto2) will continue to be supported.

    The main intent of introducing proto3 is to clean up protobuf before

    pushing the language as the foundation of Google's new API platform.

    In proto3, the language is simplified, both for ease of use and  to

    make it available in a wider range of programming languages. At the

    same time a few features are added to better support common idioms

    found in APIs.

    The following are the main new features in language version 3:

      1. Removal of field presence logic for primitive value fields, removal

         of required fields, and removal of default values. This makes proto3

         significantly easier to implement with open struct representations,

         as in languages like Android Java, Objective C, or Go.

=======

Thanks,

Feng

[Sumit Kumar]

Which forum was this discussed (dropping of default) ? May be reading it would give some insights into the details.

Regards,

Sumit Kumar

[Sumit Kumar]

Specially makes difficult for adoption in financial applications, the has_field was one of the key reasons to migrate over to protofuf.

Financial applications need differentiation in-between 0 value set and not set. Eg: Limit order with 0 price is valid but with no price set is invalid. Likewise market order with no price set is valid and with any other price set is invalid (including the 0 value). And there are many other cases, but anyway if the decision is made then not much value discussing it.

Regards,

Sumit Kumar

[Feng Xiao]

The proto3 design discussion has lasted for more than half a year and all of them happened as an internal process. We have a lot of design docs, email exchanges and weekly design meetings, but they are not available for public consumption. Currently we are preparing public documentations for proto3 and it's targeted to be publicized late this quarter.

[gtw....@gmail.com]

As a new user, we still have need (C++ target) for default values, as well as  required fields (e.g., message headers for embedded target).

1) It appears that you are completely removing the default/required capability from the proto3 language.  Is this correct ?
2) Will the proto2 language be maintained going forward ?  If so, then a simple generation addition may work for the envisioned application.  This is to generate non-static default message content so that the copy operator works simply, enabling applications to populate
    changed fields with minimal coding effort.  I looked into this possibility before I saw this forum thread.

Thank you for your effort and reply.

Gilbert T Williams.

[Feng Xiao]

On Thu Jan 22 2015 at 4:53:53 PM <gtw....@gmail.com> wrote:

As a new user, we still have need (C++ target) for default values, as well as  required fields (e.g., message headers for embedded target).

1) It appears that you are completely removing the default/required capability from the proto3 language.  Is this correct ?

Yes.

 

2) Will the proto2 language be maintained going forward ?

Yes. Proto2 will continue to be supported for C++/Java/Python. The new languages that we are adding may only support proto3 though.

[Ryan Gaudon]

As a new user it's encouraged to begin implementation with PB3.0+ opposed to 2.6. With that being said, since I'm beginning integration now and replacing XStream with this, where would you suggest I start? There is very little / no documentation available at this time, and there hasn't been an update since. 

Specifically I'm curious how to go about tackling serialization of objects that inherit properties and methods from a parent class. IIRC, this was handled with extensions and .setExtension() in older version, however I was unable to figure out how to use the new 'any' type and the compiler threw errors when attempting to work with it blindly. 

Is there anything you have available to get those started off with protobuf 3?

On Thursday, December 11, 2014 at 1:21:01 AM UTC-3:30, Feng Xiao wrote:

Hi all,

I just published protobuf v3.0.0-alpha-1 on our github site:

https://github.com/google/protobuf/releases/tag/v3.0.0-alpha-1

This is the first alpha release of protobuf v3.0.0. In protobuf v3.0.0, we will add a new protobuf language version (aka proto3) and support a wider range of programming languages (to name a few: ruby, php, node.js, objective-c). This alpha version contains C++ and Java implementation with partial proto3 support (see below for details). In future releases we will add support for more programming languages and implement the full proto3 feature set. Besides proto3, this alpha version also includes two other new features: map fields and arena allocation. They are implemented for both proto3 and the old protobuf language version (aka proto2).

We are currently working on the documentation of these new features and when it's ready it will be updated to our protobuf developer guide. For the time being if you have any questions regarding proto3 or other new features, please post your question in the discussion group.

CHANGS

=======

Version 3.0.0-alpha-1 (C++/Java):

  General

  * Introduced Protocol Buffers language version 3 (aka proto3).

    When protobuf was initially opensourced it implemented Protocol Buffers

    language version 2 (aka proto2), which is why the version number

    started from v2.0.0. From v3.0.0, a new language version (proto3) is

    introduced while the old version (proto2) will continue to be supported.

    The main intent of introducing proto3 is to clean up protobuf before

    pushing the language as the foundation of Google's new API platform.

    In proto3, the language is simplified, both for ease of use and  to

    make it available in a wider range of programming languages. At the

    same time a few features are added to better support common idioms

    found in APIs.

    The following are the main new features in language version 3:

      1. Removal of field presence logic for primitive value fields, removal

         of required fields, and removal of default values. This makes proto3

         significantly easier to implement with open struct representations,

         as in languages like Android Java, Objective C, or Go.

=======

Thanks,

Feng

[Feng Xiao]

On Mon Jan 26 2015 at 5:38:22 PM Ryan Gaudon <ry...@gaudon.ca> wrote:

As a new user it's encouraged to begin implementation with PB3.0+ opposed to 2.6. With that being said, since I'm beginning integration now and replacing XStream with this, where would you suggest I start? There is very little / no documentation available at this time, and there hasn't been an update since.

Most of the documentation on our developer guide site still applies to proto3. We don't have any documentation for proto3 specific features though.

 

Specifically I'm curious how to go about tackling serialization of objects that inherit properties and methods from a parent class. IIRC, this was handled with extensions and .setExtension() in older version, however I was unable to figure out how to use the new 'any' type and the compiler threw errors when attempting to work with it blindly. 

Sorry, but "Any" is not added in this alpha version. In proto3, "Any" will be a pre-defined message type. Its definition looks like this:

package google.protobuf;

message Any {

  string type_url = 1;

  bytes value = 2;

}

You could define such a message yourself and use it as a temporary solution. In the next alpha version we'll provide these types and utility functions to work with an Any message.

--

[Walter Schulze]

Will the extensions in the descriptor.proto also be changed to Any types?

[Feng Xiao]

On Tue Jan 27 2015 at 5:42:34 AM Walter Schulze <awalter...@gmail.com> wrote:

Will the extensions in the descriptor.proto also be changed to Any types?

No. That will continue to be supported. descriptor.proto is the only proto that will allow extensions in proto3.

 

--

[Walter Schulze]

I have declared quite a few file, message, field, etc. extensions on the descriptor.

https://github.com/gogo/protobuf/blob/master/gogoproto/gogo.proto

These extensions are used to modify the code that is generated.

This results in a user being able to create a proto file like this

https://github.com/gogo/protobuf/blob/master/test/example/example.proto

There are file, field and message extensions used in the proto, which protoc can parse.

Can I use this style in proto3 syntax?

I guess I should be able to if the descriptor does not offer me any other way to do this, like Any?

[Troy Lee]

Feng, 

Version 3 removes presence logic. 

How do we exam whether a field is exist or not?

Thanks,
troylee

Feng Xiao於 2014年12月11日星期四 UTC+8下午12時51分01秒寫道：

[Feng Xiao]

On Wed Jan 28 2015 at 12:06:21 PM Troy Lee <lee...@gmail.com> wrote:

Feng, 

Version 3 removes presence logic. 

How do we exam whether a field is exist or not?

This is no possible for singular primitive fields. For singular message fields, the has methods will still be generated. Basically with proto3 you'll need to write your code without depending on these dropped features. It's believed that most users don't use the field presence logic much and for those who need this feature adding a bool field is an easy workaround.

 

--

[Jeremy Swigart]

I don't understand. If a message is a simple struct then the generated wrapper code would populate it with the default as defined by the proto it was compiled with wouldn't it? Are you suggesting that the implementation on different platforms would lack the wrapper objects generated by protobuf? As long as you have that you have the default value. This rationale doesn't make sense.

[Alfred Kwan]

To implement the has_boo() in 3.0 implies one boolean per each truly optional field, which means additional maintenance is now required, e.g. matching naming scheme for the pool together with the optional struct, also should we group all booleans together or should they sit right next to the to corresponding optional structures...

With the uncertainty of how "any" replaces "extensions" plus the removal of has_boo(), it seems like new adopters (I'm one of them) should pick V2 over 3.0.

[Feng Xiao]

The union types are obsoleted by oneof:

https://developers.google.com/protocol-buffers/docs/proto#oneof

[Feng Xiao]

On Sat, Feb 7, 2015 at 4:31 AM, Jeremy Swigart <jswi...@gmail.com> wrote:

I don't understand. If a message is a simple struct then the generated wrapper code would populate it with the default as defined by the proto it was compiled with wouldn't it? Are you suggesting that the implementation on different platforms would lack the wrapper objects generated by protobuf?

There may be languages whose protobuf implementation would not be able to efficiently support these features. Note that these decisions are not made based on the current languages that we support, but based on that we are going to support a much wider range of languages.

 

As long as you have that you have the default value. This rationale doesn't make sense.

[Alfred Kwan]

Thanks for pointing me to "oneof". I gave it a try and I have two questions:

1) I see the has_() being generated for all the fields inside oneof. Is this kind of has_() function here to stay throughout subsequent V3 releases?

2) Since oneof does not allow a repeated field in both V2/3, is there pro/cons in case I create one extra layer of structure i.e.:

message manyMsg{ // workaround: wrap the repeating message
   repeated oneMsg = 1;
}
message unionMsg{
   oneof testOneof{
      manyMsg msg = 1; // doesn't allow repeated
      uint32 foo = 2;
   }
}

[Kevin Baker]

Hi,

Thanks for all your work with protobuf. I am excited about the changes with proto3 that will reduce errors (no forgetting to set has_* in nanopb, yay!) and will make mapping into new languages much simpler, helping our interop case a lot.

My question is: We are currently using protobuf pretty extensively and it looks like we will not be impacted by any changes in proto3 in our proto files (all fields being present, removal of required, default values, etc.) Does this mean our existing proto2 applications are compatible on-the-wire with proto3? How upwards-compatible is proto3 with proto2?

Of course I will test this as well but I was wondering if there are any planned breakages of the wire format or if they will be compatibly phased in.

Thanks,
Kevin

[Feng Xiao]

On Thu, Feb 26, 2015 at 11:55 PM, Kevin Baker <kba...@gmail.com> wrote:

Hi,

Thanks for all your work with protobuf. I am excited about the changes with proto3 that will reduce errors (no forgetting to set has_* in nanopb, yay!) and will make mapping into new languages much simpler, helping our interop case a lot.

My question is: We are currently using protobuf pretty extensively and it looks like we will not be impacted by any changes in proto3 in our proto files (all fields being present, removal of required, default values, etc.) Does this mean our existing proto2 applications are compatible on-the-wire with proto3?

Yes.

 

How upwards-compatible is proto3 with proto2?

Proto3 uses the same wire-format as proto2. A proto2 application should be able to parse the output of a proto3 server using the same .proto definition (only differing in syntax version). It's also true vice versa.

 

Of course I will test this as well but I was wondering if there are any planned breakages of the wire format or if they will be compatibly phased in.

There is no planned wire-format changes for proto3.

[Kevin Baker]

That's great news, thanks!!! Looking forward to proto3!

Kevin

[Kostiantyn Shchepanovskyi]

Hi,

3. Removal of extensions, which are instead replaced by a new standard type called Any.

Extensions are used for custom options definition in proto2.

import "google/protobuf/descriptor.proto";

extend google.protobuf.MessageOptions {
  optional string my_option = 51234;
}

message MyMessage {
  option (my_option) = "Hello world!";
}

How it will look in proto3?

[Feng Xiao]

Extensions to these options are explicitly allowed in proto3.

[Jonathan Briggs]

I just ran into this while trying to write a gRPC service that can handle forward and back compatibility.

So say that a new service is running with new fields in its proto3 message and it returns this to an older client. The older client is supposed to receive it, modify one or two fields and then send it back to update the data.

It seems that with proto3 the new fields are just being erased when the older client sends back an older message.

What is the solution to this?

[Josh Gargus]

When arenas are used, do they capture all dynamic allocation, or is there other memory allocated at some time?  For example, is there any static initialization done by protobufs?  What about when you modify a parsed proto by using e.g. clear_data()/add_data()/clear()/append()? Etc.

Thanks,

Josh

      2. Removal of unknown fields.

      3. Removal of extensions, which are instead replaced by a new standard

         type called Any.

=======

Thanks,

Feng

[Feng Xiao]

On Thu, Apr 9, 2015 at 8:34 PM, Josh Gargus <jj...@google.com> wrote:

When arenas are used, do they capture all dynamic allocation, or is there other memory allocated at some time?  For example, is there any static initialization done by protobufs?  What about when you modify a parsed proto by using e.g. clear_data()/add_data()/clear()/append()? Etc.

+Chris, could you help answer these questions?

 

--

[wora]

The recommended way to solve this problem is to use google.protobuf.FieldMask. The client should just send a request with 1 or 2 fields plus the FieldMask, and server will perform partial update on it. protobuf will provide better support on FieldMask, which the server code can utilize to perform the partial update.

[Nikolay Mladenov]

I am also evaluating proto2 vs proto3 and even though it seems proto3 should be the way to go I really miss the has_** functionality in proto3.

It seems the following proto pattern may be a workaround:

message M{
   oneof optional_value{
     int32 value = 1;
   }
}

It does generate value(), set_value(), clear_value() and has_value() methods (C++) but unfortunately the has_value is private.
Is there a reason such a useful and short method is declared private (it implementation only uses public functionality as well)

[Feng Xiao]

On Wed, Apr 29, 2015 at 6:21 AM, Nikolay Mladenov <nikolay....@gmail.com> wrote:

I am also evaluating proto2 vs proto3 and even though it seems proto3 should be the way to go I really miss the has_** functionality in proto3.

It seems the following proto pattern may be a workaround:

message M{
   oneof optional_value{
     int32 value = 1;
   }
}

It does generate value(), set_value(), clear_value() and has_value() methods (C++) but unfortunately the has_value is private.
Is there a reason such a useful and short method is declared private (it implementation only uses public functionality as well)

You can use "optoinal_value_case()" to check which oneof is set.

[Alfred Kwan]

Nikolay, the has_foo being private is actually intentional, see this closed issue.

Alfred

[Nikolay Mladenov]

I understand it was an intentional design decision, I just fail to understand the reasoning for it.
I see this statement.

>>"In proto3, this is an intentional design decision: has_...() methods go away except for message fields. For oneofs, the idiomatic API pattern is to use get_oneof_case() to check which field in the oneof is set."

It is obvious for me that regular non-message fields should not have has_.. in proto3.

but for oneof has_... makes perfect sense.

It is still defined because it is convenient for the "internal" code.

But it is also convenient for other uses, so hiding it is superficial.

I now have to redefine all such method for my needs and they have exactly the same definitions as the private members,

but are just a hassle to write/update and use....

--
You received this message because you are subscribed to a topic in the Google Groups "Protocol Buffers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/protobuf/ZRpcfmeGK6s/unsubscribe.
To unsubscribe from this group and all its topics, send an email to protobuf+u...@googlegroups.com.

[Feng Xiao]

On Thu, Apr 30, 2015 at 12:52 PM, Nikolay Mladenov <nikolay....@gmail.com> wrote:

I understand it was an intentional design decision, I just fail to understand the reasoning for it.
I see this statement.

>>"In proto3, this is an intentional design decision: has_...() methods go away except for message fields. For oneofs, the idiomatic API pattern is to use get_oneof_case() to check which field in the oneof is set."

It is obvious for me that regular non-message fields should not have has_.. in proto3.

but for oneof has_... makes perfect sense.

It is still defined because it is convenient for the "internal" code.

But it is also convenient for other uses, so hiding it is superficial.

I now have to redefine all such method for my needs and they have exactly the same definitions as the private members,

but are just a hassle to write/update and use....

The intention is to make the public API simpler and consistent. As there is already a way to check for field presence for oneof fields, we decide to not provide has_ methods.

[Mike McElligott]

Hi
   I have a collection of message types which all share a common set of fields. Each message type has an additional number of attributes particular to that message type. Two choices I am considering on how to structure our schemas

Option A - import and compose
========================
common.proto
syntax = "proto3";

message Common
{
                string fruit = 1;
                string vegetable = 2;
}

messageA.proto
syntax = "proto3";
import "common.proto";

message messageA
{
                Common shared = 1;
                string cereal = 2;
}


Option B - Use Any in the common part
============================
common.proto
syntax = "proto3";

message Common
{
                string fruit = 1;
                string vegetable = 2;
                Any  messageX = 3;
}

messageA.proto
syntax = "proto3";
import "common.proto";

message messageA
{
        string cereal = 2;
}


Any advice or opinions on which choice is preferable? Would the Any route require an extra deserializing step that the composition route does not and be less efficient? It seems that after deserializing in the client, we still have to pull the any field, check the type and then additionally unpack it. Would the Any route require me to compile twice and generate two separate sets of files for each client - one for the Common.proto and one for the messageX.proto? Option A would require just a compile against messageX.proto.


Cheers
Mike

[Chris]

Thanks for your great work.

I'm considering migrating to version 3. Is existing proto2 data (i.e. serialized messages) transparently loadable in a proto3-aware application ?

[Feng Xiao]

I would only resort to Any if normal protobuf constructs cannot do the job. In your case I don't see any benefit from the using of Any. To me a strictly better approach then option B is:

message Common {

  string fruit = 1;

  string vegetable = 2;

  oneof other_ingredients {

    MessageA a = 3;

  }

}

message MessageA {

  string cereal = 2;

}

That's not to say I prefer this to option A though. I believe both of them are commonly used patterns. The difference is whether you have a single type to hold any data or different types to hold different data. If you need to send the data over the wire, using a single root type might be easier.

For Any, a typical use case is that you are the server and want to allow clients to attach arbitrary data to the message. Your use case doesn't seem similar to this though.

[Mikhail Nikonov]

First of all, thanks for the great work; I've been using protobufs for a while, and it's good to see them evolving.

One question - if map field is internally emulated by repeated field type, is it packed (e.g. [packed=true]) by default, and is there a way to pack it? As a related question, was there a thought about making repeated fields packed by default, now that .proto language is changing in backward-incompatible manner?

[Feng Xiao]

On Tue, May 12, 2015 at 5:58 PM, Mikhail Nikonov <michael....@gmail.com> wrote:

First of all, thanks for the great work; I've been using protobufs for a while, and it's good to see them evolving.

One question - if map field is internally emulated by repeated field type, is it packed (e.g. [packed=true]) by default, and is there a way to pack it?

No. Only repeated primitive fields can be packed but map fields are represented as repeated message fields on the wire.

 

As a related question, was there a thought about making repeated fields packed by default, now that .proto language is changing in backward-incompatible manner?

Yes, we decided to make repeated primitive fields always packed in proto3. It will be updated soon.

[Darin Gordon]

Hi Feng

What would you say is a reasonable ETA for the protobuf 3 full release?  Are you going through a series of RC's first?

[Feng Xiao]

On Fri, Jun 12, 2015 at 12:40 PM, Darin Gordon <dar...@gmail.com> wrote:

Hi Feng

What would you say is a reasonable ETA for the protobuf 3 full release?  Are you going through a series of RC's first?

We will have a series of alpha releases followed by a series of betas. The 3 full release may be in Q4 this year.

 

--

[James Hugard]

On Fri, Jun 12, 2015 at 12:40 PM, Darin Gordon <dar...@gmail.com> wrote:

Hi Feng

What would you say is a reasonable ETA for the protobuf 3 full release?  Are you going through a series of RC's first?

We will have a series of alpha releases followed by a series of betas. The 3 full release may be in Q4 this year.

While the Google implementation of protobuf3 is still alpha, what is the status of the proto3 specification?  Are the language, binary and json serialization formats, and wire format all locked?

Thank you!

James

[Feng Xiao]

Yes, proto3 design is finalized.

 

Thank you!

James

[The Nguyen Xuan]

Does this version support object type in C# ?

ex:

[ProtoMember(1)]

public object A {get;set;}

thank.

Vào 11:51:01 UTC+7 Thứ Năm, ngày 11 tháng 12 năm 2014, Feng Xiao đã viết:

[Jon Skeet]

That looks like you're expecting a protobuf.net-style approach - to which the answer is "no" and will continue to be "no".

The C# support will continue to be based on generated code, but there's a new code generator and runtime now in the master branch. The main changes from the previous code are:

- proto3-only support (no proto2 at all)

- mutable generated types rather than the Java-style builders and immutable messages

Jon

[Feng Xiao]

The decision is not to support proto2 in C# (and probably also for all other languages that are new in v3.0.0+).

On Fri, Oct 9, 2015 at 10:42 AM, Teddy Zhang <losti...@gmail.com> wrote:

Will the C# implementation support proto2 message as well?

What is the compatibility story between proto2 and proto 3? I assume the wire format is compatible as long as no proto 3 exclusive features are used.

[Walter Schulze]

In other words C# and those new languages won't be able to serialize the descriptor?

--
You received this message because you are subscribed to a topic in the Google Groups "Protocol Buffers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/protobuf/ZRpcfmeGK6s/unsubscribe.
To unsubscribe from this group and all its topics, send an email to protobuf+u...@googlegroups.com.

[Feng Xiao]

On Fri, Oct 9, 2015 at 10:51 AM, Walter Schulze <awalter...@gmail.com> wrote:

In other words C# and those new languages won't be able to serialize the descriptor?

descriptor.proto is an exception. It's allowed to be imported by proto3 files to support custom options. I.e., the following is explicitly allowed:

syntax = "proto3";

package test;

import "google/protobuf/descriptor.proto";

extend google.protobuf.FieldOptions {

  string my_field_option = 123456789;

}

message TestMessage {

  int32 value = 1 [(my_field_option) = "Some extra option data"];

}

However, the ability to use these descriptor info at run-time may be limited (or not present) in certain languages. It depends on what API is provided by the specific language. +Jon Skeet could probably say more about how to use/access descriptors in C#.

[Teddy Zhang]

Will the C# implementation support proto2 message as well?

What is the compatibility story between proto2 and proto 3? I assume the wire format is compatible as long as no proto 3 exclusive features are used.

On Tuesday, August 4, 2015 at 5:43:36 AM UTC-7, Jon Skeet wrote:

[Teddy Zhang]

Will the C# implementation support proto2 messages?

What is the capability story? I assume the wire format is compatible if no proto3 exclusive features are used?

On Tuesday, August 4, 2015 at 5:43:36 AM UTC-7, Jon Skeet wrote:

[Jon Skeet]

Yes, just to expand on this - the descriptor protos aren't directly accessible, but the information contained within them is - via the public types in the Google.Protobuf.Reflection namespace.

We're able to do this because we've looked (very carefully!) at how descriptor.proto works in terms of proto3 - basically we generate code as if it were proto3 syntax, and have workarounds for couple of issues that throws up.

Jon

[minghu...@nutanix.com]

Hi Feng,

I have a question: looks like there is no copy constructor that takes arena pointer. I got compilation error on this:

  feature_package::MyFeatureMessage* arena_message =

       Arena::CreateMessage<feature_package::MyFeatureMessage>(arena, m);

I need to rewrite this in 2 steps:

  feature_package::MyFeatureMessage* arena_message =

       Arena::CreateMessage<feature_package::MyFeatureMessage>(arena);

  arena_message->CopyFrom(m);

My question is that would this be less efficient than a copy constructor with arena pointer if there were such one?

Thanks,

Minghui

[le liu]

If it possible to compile into DLL? I have not find the directory about "vsproject" which is exists in v2.0. Thanks

在 2014年12月11日星期四 UTC+8下午12:51:01，Feng Xiao写道：

[minghu...@nutanix.com]

Is there a plan to also arena allocate string fields?

[Feng Xiao]

On Wed, Oct 21, 2015 at 1:39 PM, <minghu...@nutanix.com> wrote:

Is there a plan to also arena allocate string fields?

In the future we will introduce different ctypes for string fields and if you specify ctype=STRING_PIECE on a string field it will be stored as a StringPiece and can then be arena-allocated. There is no timeline for when this will be added though.

 

--

You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.

[Freelancer Casper]

Надеюсь,что вы не будете против моего вступления в вашу группу.Тема очень интересна.

четверг, 11 декабря 2014 г., 7:51:01 UTC+3 пользователь Feng Xiao написал:

[Manivannan s]

For Enums it is not possible - It's possible to import proto2 message types and use them in your proto3 messages, and vice versa. However, proto2 enums cannot be used in proto3 syntax.

https://developers.google.com/protocol-buffers/docs/proto3 

On Friday, 16 January 2015 01:21:04 UTC+5:30, Feng Xiao wrote:

On Tue Jan 13 2015 at 3:17:26 PM Arjun Satish <arjun....@gmail.com> wrote:

Since this feature is never going to be exposed, what advice do you have for people who are using this feature and want to migrate to v3? Also, what can we do as users to skip encoding certain fields with the new library?

We are aware of the migration difficulties from v2 to v3 due to removed features. As such we do not encourage existing users to migrate and commit to continue the support for proto2 syntax.

You could have both proto2 and proto3 syntax files in your project and they are allowed to import each other despite the syntax differences.

 

Thanks,

On Tue, Jan 13, 2015 at 3:10 PM, Feng Xiao <xiao...@google.com> wrote:

On Tue, Jan 13, 2015 at 3:05 PM, Arjun Satish <arjun....@gmail.com> wrote:

Feng,

What do you mean when you say "In C++/Java/Python where we support both proto2 and proto3, default values will continue to exist"?

What I meant is that you can still find its traces in the implementation but the feature itself is not exposed publicly in proto3 (i.e., we are still using it under-the-hood).

 

When I run protoc (v3) with the syntax="proto3" tag, it shows an error "Explicit default values are not allowed in proto3." and exits (no code is generated). This does not let me use other proto 3 features if my proto definition file contain default values.

Thanks for your timely responses! Highly appreciate it!

On Tue, Jan 13, 2015 at 10:35 AM, Feng Xiao <xiao...@google.com> wrote:

On Mon, Jan 12, 2015 at 10:40 PM, Arjun Satish <arjun....@gmail.com> wrote:

Would it be possible to re-introduce this feature in a subsequent release? It seems like you are still using it under-the-hood.

In C++/Java/Python where we support both proto2 and proto3, default values will continue to exist. In new languages (e.g., ruby) though, the support for non-zero default values will be dropped completely.

 

And because of the benefits I mentioned above, I strongly feel that it will only help the community.

As far as I know, the decision is final. Internally a lot Google projects have already adopted the new syntax and so far we have not heard problems caused by disallowing default values. It's unlikely this will be changed in the future. The omission of this feature (and other features) is to make the language simpler and to allow more idiomatic implementations in a wider range of languages. It's believed this decision will help the protobuf community (both protobuf maintainers and protobuf users) and we expect proto3 to be a version that can be more easily adopted than proto2 by new users due to these simplifications. For existing users who rely on removed features, they can continue to use proto2 and that will be supported for a long time (if not forever). Currently we generally do not recommend migrating existing proto2 projects to proto3 because of incompatibility issues (e.g., extensions are dropped in proto3) and only recommend new users to use proto3.

 

Best,

On Wed, Jan 7, 2015 at 8:01 PM, Feng Xiao <xiao...@google.com> wrote:

+liujisi, who should have a better idea of why default value is dropped from proto3 and what alternatives users can rely on.

Internally the design of proto3 has been discussed among a group of people for quite a long time, but most of them haven't subscribed this forum though...

On Wed, Jan 7, 2015 at 18:52 Arjun Satish <arjun....@gmail.com> wrote:

Did anyone get a chance to look at this request?

On Wednesday, December 17, 2014 3:54:12 PM UTC-8, Arjun Satish wrote:

Hey guys,

Thanks for all the hard work!

I have a question regarding the decision to drop support for default values. Fields which are set to their default values are not serialized. I noticed that in the new code (3.0.0-alpha-1 for Java),  this condition still holds true. But the default values used are the standard ones (0 for int64/int32 etc) and cannot be specified in the .proto file. In some of my code, I had reasons to use non-zero default values (-1 for some integers, 1024 for some others, 3.14 for some doubles etc). Using the old protocol buffers, this was trivial to implement. This was a great feature as we could save atleast 2 bytes for every "untouched" field (which comes in handy when we persist the data :-)).

Is there any way we can retain specification of default values in the .proto files and using them in the generated encoders/decoders?

Thanks very much!

Looking forward to the 3.0 release!

Best,
Arjun Satish

--