Is protobuf3 a superset of protobuf2?

[ajcurtis84]

Hello,

I have not found anything in the documentation that explicitly says this. The only indication is that the proto3 documentation refers to proto2. Things like extensions are available in the proto3 syntax?

thanks

[Adam Cozzette]

Neither one is really a superset of the other. Proto3 does not support extensions but uses the Any type as a replacement.

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscribe@googlegroups.com.
To post to this group, send email to prot...@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[Ilia Mirkin]

The main thing that proto3 lacks and keeps coming up every time I use
it is field availability information. All the has_* API is gone for
basic types, so you have to supply that information separately, or
restructure your logic to not rely on it.

-ilia

> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to protobuf+u...@googlegroups.com.

[Josh Humphries]

To clarify, proto3 supports extensions in that you can define custom options or create extensions for proto2 messages (since it is valid to import proto2 files). But it does not support defining an extendable message (e.g. no extension ranges can be defined with proto3).

Proto3 is almost more like a subset of proto2:

1. You cannot define extension ranges in proto3
2. You cannot define required fields in proto3
3. You cannot define default values in proto3. Default values are always the zero value for a type.
4. You cannot define groups in proto3
5. Proto3 has no has* methods because, by design, an absent value is indistinguishable from its default (e.g. zero) value.
6. You cannot define an enum whose first value does not have the numeric value of zero.
7. Up until 3.5, proto3 did not preserve unrecognized fields. Even as of 3.5, this support is still not complete in all languages (such as Go), even though it is supported for proto2 (for the languages/runtimes that have proto2 support anyway).

All of the "new" features ascribed to proto3 are really features of the protoc compiler, starting with v3.0, not of proto3. The new features include the following:

1. Map types
   
2. Streaming RPCs
3. Standard JSON representation
4. Well-known types (including Any)
5. New language-level: proto3 syntax

So it's important not to conflate "protobuf v3" with bullet 5 above, "proto3". All of those other features can still be used with proto2 source files (aside, possibly, from using JSON format with a message with extensions).

----
Josh Humphries
jh...@bluegosling.com

[ajcurtis84]

Thank you everyone for all the great input!

Based on this discussion, what is the advantage of using proto3? It appears that proto2 is more feature rich. JSON isn't a compelling reason....

Thanks

[Josh Humphries]

I think proto3 was intended to be simpler -- an evolution of protobuf in a direction that is more refined and elides superfluous features. Eventually (though not likely any time soon), support for proto2 will go away.

The main omission in proto3 that I personally felt strongly about was the lack of preserving unknown fields. But that functionality has been restored for many languages/runtimes in 3.5 (and coming soon, hopefully, for Go and the others).

All of the other omissions in proto3 have alternate solutions/work-arounds that aren't baked into the language (like wrapper types and Any), which shows that these features didn't truly need to be language features to begin with. So it allows the language itself to remain simple, which means libraries/tooling are simpler to build on top.

So a big advantage of proto3 is simplicity (and for some languages/runtimes, that will translate into improved performance). And another advantage is future compatibility (since, in all likelihood, proto2 will eventually go away.)


If creating new files, it makes sense to use proto3. If you already have a corpus of proto2 files, a reasonable migration path forward is to start evolving them to proto3 (or, if the languages/runtimes you use don't yet preserve unrecognized fields, wait for that functionality to be fully implemented everywhere and then start evolving).

• Make required fields optional. Required fields have always been dangerous. If a field is required in your protocol, it can be enforced in application code instead of in the de-serialization logic of the protobuf runtime.
• Use Any instead of extensions. An extension range can generally be replaced with a repeated Any field. Extensions with scalar types will have to be "boxed" into single-field messages.
• Use wrapper types for scalar fields where absence must be distinguishable from zero.
• Usages of default values can be changed to use wrapper types and application logic that decides non-zero values to use when the field is not present.
  
• Groups should be replaced with nested messages.

Once a file is no longer using any proto2 features, it can be changed to proto3 syntax. Unfortunately, there is not yet a way to incrementally move to proto3. Once you change the syntax of the file, the generated code will change, and all clients of that generated code must be changed atomically in order to compile (at least for languages/runtimes where generated code for the two syntaxes is materially different/incompatible). I would hope that this is something that will be addressed in future versions of protobuf, to facilitate migrations away from proto2.

----
Josh Humphries
jh...@bluegosling.com

--

[Adam Cozzette]

Actually we have no plans to deprecate proto2 and we are still actively developing it, so you can really choose either one without having to worry about support going away.

[Josh Humphries]

Adam,

I understand there is no current plan to deprecate proto2. But is it really expected to be supported forever?

If that's the case, I suppose the only reason to choose proto3 over proto2 (if you happen to want the features and semantics of proto2) might be that not all languages/runtimes have support for proto2. Though I thought this was a short/medium-term issue. If all languages/runtimes eventually support both, it seems like a strange decision to continue supporting both indefinitely.

----
Josh Humphries
jh...@bluegosling.com

[Adam Cozzette]

Yes, we are planning on supporting proto2 pretty much forever. Within Google we have a huge amount of code using proto2 and for the most part we're not attempting to migrate existing code to proto3. Language support is one reason to go with proto3, but it's also simpler and more convenient for working with JSON.

[Luciano Perezzini]

This is still true, Adam? Google would not deprecate Proto2, and Proto2 and Proto3 are just different flavors of Protocol Buffers? Me and my team are about to start a brand new project using Protocol Buffers and the lack of missing value check properties in Proto3 is a BIG concern.

Thanks in advance, and I wait for your response.

Regards!

----
Josh Humphries
jh...@bluegosling.com

----
Josh Humphries
jh...@bluegosling.com

To unsubscribe from this group and stop receiving emails from it, send an email to prot...@googlegroups.com.

To post to this group, send email to prot...@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prot...@googlegroups.com.

[Adam Cozzette]

Right, we are still maintaining both proto2 and proto3 and plan to keep supporting both flavors indefinitely.

To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/3e078cd3-ba3f-4ab0-a9a4-e8852f502045%40googlegroups.com.

[Luciano Perezzini]

Thanks for your quick response Adam!

Regards!

To unsubscribe from this group and stop receiving emails from it, send an email to prot...@googlegroups.com.