Status of protobuf-java 2.x / 3.x compatibility

[Chris Thunes]

I'm looking at options for moving some applications that currently depend on protobuf-java 2.5.0 to a more recent version. This is made complicated by the fact that we have a mixure of internal and external dependencies (Hadoop & HBase) which depend on protobuf-java. My understanding is that this will require these dependencies to move to a 3.x release sychronously (i.e. regenerate using a 3.x protoc and update protobuf-java to a corresponding release).

However, looking through release notes and protobuf source code it seems like some attempts have been made to address the source and binary compatibility issues between 2.5/2.6.1 and 3.x. Specifically,

• The 3.0.0-beta-4 release notes mention runtime updates "to be compatible with 2.5.0/2.6.1 generated protos".
• A number of classes have "V3" variants where the non-V3 variants appear to exist solely in an attempt to maintain binary compatibility with pre-3.x generated code.

Running the compatibility tests in java/compatibility_tests/v2.5.0 it appears that source and binary incompatibilities still exist.

I'm curious if anyone can shed some light on this effort and its status or provide suggestions for migrating to a recent protobuf release under these circumstances.

Thanks,

Chris

[Feng Xiao]

Protobuf 2.5.0/2.6.1 should be compatible with 3.0.0-beta-4 if you only uses protobuf public APIs. That's unfortunately not the case with Hadoop & HBase though. They introduced a class into com.google.protobuf package with the sole purpose to access protobuf package private classes. As such there is no way you can upgrade to protobuf 3.x if the version of Hadoop & HBase you use still depends on protobuf 2.5/2.6 private symbols.

Thanks,

Chris

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscribe@googlegroups.com.
To post to this group, send email to prot...@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[Chris Thunes]

Thanks Feng. It seems like the GeneratedMessage / GeneratedMessageV3 split introduced between 3.0.0-beta-4 and the 3.0.0 final release caused the java/compatability_tests suite to start failing when run against 2.5.0 / 2.6.1 ("java/compatibility_tests/v2.5.0/test.sh 2.5.0" passes at git tag v3.0.0-beta-4 but fails at v3.0.0). In 3.0.0 and newer, what is the role of the non-V3 variants of GeneratedMessage, SingleFieldBuilder, etc? Are these classes simply vestigial at this point or do they still provide some benefit to compatibility (even if not 100%).

Thanks again,

Chris

[Feng Xiao]

On Tue, Dec 5, 2017 at 1:33 PM, Chris Thunes <cth...@brewtab.com> wrote:

Thanks Feng. It seems like the GeneratedMessage / GeneratedMessageV3 split introduced between 3.0.0-beta-4 and the 3.0.0 final release caused the java/compatability_tests suite to start failing when run against 2.5.0 / 2.6.1 ("java/compatibility_tests/v2.5.0/test.sh 2.5.0" passes at git tag v3.0.0-beta-4 but fails at v3.0.0).

This is expected. Only 3.0.0-beta-4 is intended to be compatible with 2.5.0/2.6.1. The final 3.0.0 release is not compatible with 2.5.0/2.6.1. 3.0.0 release is compatible with 3.0.0-beta-4 though. So a migration path could be:

1. Move individual components to depend on 3.0.0-beta-4 instead of 2.5.0/2.6.1. Since 3.0.0-beta-4 is compatible with 2.5.0/2.6.1 you don't need to upgrade everything at once but can do it incrementally.

2. Once every component is on 3.0.0-beta-4, upgrade the dependency to 3.x.x.

This is how we do the migration inside google, but I guess it's hard to do the same in the opensource world because you don't have control over your dependencies.

 

In 3.0.0 and newer, what is the role of the non-V3 variants of GeneratedMessage, SingleFieldBuilder, etc? Are these classes simply vestigial at this point or do they still provide some benefit to compatibility (even if not 100%).

They are still needed to be compatible with older beta releases like 3.0.0-beta-3 etc.

[Igor Gatis]

Slightly off-topic but... Can one mix classes generated by protoc 2.6.1 and protoc 3.x? What about runtime?

[Adam Cozzette]

In Java it should generally be OK to mix generated classes from different versions. Actually though, 2.6.1 may be old enough to be problematic. Your best bet is probably to make sure everything is from at least 3.0 or higher. The other important rule is that you can't have generated code newer than the runtime, so if you update any generated code to a newer version, make sure to update the runtime as well.

On Fri, Jun 28, 2019 at 5:06 AM Igor Gatis <igor...@gmail.com> wrote:

Slightly off-topic but... Can one mix classes generated by protoc 2.6.1 and protoc 3.x? What about runtime?

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.

To post to this group, send email to prot...@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.

To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/f9420d89-89c9-4183-8e1b-6093d422743c%40googlegroups.com.