Using grpc in a project with protobuf-java 2.5.0 dependencies

[skat...@root.gg]

I'm willing to use grpc in some projects that have transitive dependencies to protobuf-java 2.5.0 in some hadoop/hbase related dependencies :

org.hbase:asynchbase:jar:1.6.0

org.apache.hbase:hbase-client:jar:0.98.6-hadoop2
org.apache.hadoop:hadoop-hdfs:jar:2.6.0

Maven seems to resolve conflicts by omitting protobuf-java 2.5.0 and I end up with this kind of errors :

Exception in thread "main" java.lang.VerifyError: class org.hbase.async.generated.ClientPB$GetRequest overrides final method getParserForType.()Lcom/google/protobuf/Parser;

at java.lang.ClassLoader.defineClass1(Native Method)

at java.lang.ClassLoader.defineClass(ClassLoader.java:760)

at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)

at java.net.URLClassLoader.access$100(URLClassLoader.java:73)

at java.net.URLClassLoader$1.run(URLClassLoader.java:368)

at java.net.URLClassLoader$1.run(URLClassLoader.java:362)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:361)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at org.hbase.async.HBaseClient.ensureTableFamilyExists(HBaseClient.java:1036)

at org.hbase.async.HBaseClient.ensureTableExists(HBaseClient.java:1061)

at com.me.compute.utils.AsyncHBase.<init>(AsyncHBase.java:52)

at com.me.compute.Compute.start(Compute.java:62)

From what I understand the protobuf-2.5.0 generated classes in asynchbase are not usable anymore by protobuf-java 3.0.0-beta1.
Do someone have an idea of how to set up grpc in a project with such dependencies to protobuf-java 2.x ?

May be useful :

I generated classes from my grpc proto files using this command and the grpc server is doing fine :

protoc --plugin=protoc-gen-grpc-java=$GRPC_JAVA_HOME/compiler/build/binaries/java_pluginExecutable/protoc-gen-grpc-java --grpc-java_out="../java" --proto_path="." grpc.proto

This are the interesting parts of my pom.xml :

<dependencies>
<!-- ASYNCHBASE -->
<dependency>
<groupId>org.hbase</groupId>
<artifactId>asynchbase</artifactId>
<version>1.6.0</version>

<!-- GRPC -->
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-all</artifactId>
<version>0.9.0</version>
</dependency>

<!-- PROTOBUF-JAVA -->
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>3.0.0-beta-1</version>
</dependency>
</dependencies>

<build>
<plugins>
<!-- We specify the Maven compiler plugin as we need to set it to Java 1.8 -->
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>

<!-- Make mvn package produce a jar-with-depenencies -->
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<archive>
<manifest>
<mainClass>com.me.compute.Compute</mainClass>
</manifest>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id> <!-- this is used for inheritance merges -->
<phase>package</phase> <!-- bind to the packaging phase -->
<goals>
<goal>attached</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>

[skat...@root.gg]

Looks like shading slove the problem nicely.

https://www.elastic.co/blog/to-shade-or-not-to-shade

[Adam Lugowski]

We have the same problem, and also solved it with relocation. Shading is when you put all the classes into one big .jar, but some shading plugins also support relocation, where you can rename packages. Maven's shading plugin is particularly good at this; it can also relocate .class files it doesn't have the source for. I haven't found the same support in the Gradle shade plugin, but that might have changed since I last looked.

I really wish the protobuf guys would not use the same package name for incompatible pieces of code, but that's their choice.

Anyway, what I did is build a very small Maven project (two files) that builds our grpc .proto, includes the bits of grpc that use protobuf, and includes a small adapter class. If there's interest I can see if I can get it open sourced. But for now, here's what you need to do:

Make a grpc-yourcompany Maven project with the shade plugin.
Dependencies:
 - protobuf-java
 - grpc-protobuf
 - grpc-stub

 - add your grpc .proto and use the grpc protoc as described in the grpc-java docs.

 - Add a shade+relocation rule similar to the one at the end of this post.

 - To use your stubs, you need to pass them protobuf3 ByteStrings. That would mean you'd have to import the protobuf package in your code, and then you're back to the package conflict. The way I solved that is to make a small adapter class that just forwards byte arrays into protobuf3 ByteStrings. I'm attaching my ByteStringContainer.java, it's best to only use the static copyFrom() function so you don't have to create junk intermediaries. What this does is it makes this one class do the protobuf3 import, and since it's in your relocatable project then its import will also be relocated. That means your actual project doesn't have to import protobuf3 directly.

- Delete anything protobuf from the grpc-all jar so it doesn't get picked up by the class loader and override your old protobuf:
zip -d lib/grpc-all-0.10.0-SNAPSHOT.jar 'io/grpc/protobuf/*'

Then you add your grpc-yourcompany.jar and your modified grpc-all.jar into your project's classpath, and everything works honky dory.

Shading+relocation rule I use:

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.4</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <includes>
                                        <include>com/turn/**</include>
                                        <include>com/google/protobuf/**</include>
                                        <include>io/grpc/protobuf/**</include>
                                    </includes>
                                </filter>
                            </filters>
                            <!---
                            Relocation. Rename the com.google.protobuf package so it doesn't clash with
                            the protobuf2 package in the Turn repo.
                             -->
                            <relocations>
                                <relocation>
                                    <pattern>com.google.protobuf</pattern>
                                    <shadedPattern>com.google.protoshadebuf3</shadedPattern>
                                    <!--- Relocation supports exclusion. We don't need it, but it looks like this:
                                    <excludes>
                                        <exclude>org.codehaus.plexus.util.xml.pull.*</exclude>
                                    </excludes>
                                    -->
                                </relocation>
                            </relocations>

                            <createSourcesJar> true </createSourcesJar>
                            <shadeSourcesContent> true </shadeSourcesContent>

                        </configuration>
                    </execution>
                </executions>
            </plugin>