Hadoop guava dependencies conflict with gobblin guava jar

[Zhiying Zhang]

Hi,

Has anyone met the problem that gobblin guava jar version conflicts with hadoop library guava dependencies? The exception is as follows:

java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:134)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:747)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:132)
        ... 7 more
Caused by: java.lang.NoSuchMethodError: com.google.common.collect.Sets.newConcurrentHashSet()Ljava/util/Set;
        at gobblin.configuration.SourceState.<clinit>(SourceState.java:54)
        at gobblin.runtime.mapreduce.MRJobLauncher$TaskRunner.<init>(MRJobLauncher.java:551)
        ... 12 more

I'm using HDP 2.3.2.0-2950 which uses hadoop 2.7.1, and it seems to use guava-11.0 while gobblin uses guava-18.0. I find in gobblin-mapreduce.sh, it actually sets

export HADOOP_USER_CLASSPATH_FIRST=true

export HADOOP_CLASSPATH=$GOBBLIN_DEP_JARS:$HADOOP_CLASSPATH

also:

-D mapreduce.user.classpath.first=true \

-D mapreduce.job.user.classpath.first=true \

-libjars $LIBJARS \

and I'm running gobblin like this:

 ./bin/gobblin-mapreduce.sh start --jars ./lib/kafka-gobblin-hdfs-test.jar,./lib/* --conf ./job_config_dir/*.job

So, why would it still gives the above exception? Could anyone give some tips on what is wrong?

Thanks,

Zhiying

[Zhiying Zhang]

And also I can run standalone mode perfectly well with my settings.

[George]

You are sure that gobblin uses 18.0, because here I see here  https://github.com/linkedin/gobblin/blob/master/build.gradle 15.0? you can get an overview over the needed dependecies with ./gradlew dependencies...

[Issac Buenrostro]

We actually downgraded to 15.0 on August, previously it was using v 18.0.

https://github.com/linkedin/gobblin/commit/e22a768d907bdc999df41578b6e69249c12ca9a1

--
You received this message because you are subscribed to the Google Groups "gobblin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gobblin-user...@googlegroups.com.
To post to this group, send email to gobbli...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gobblin-users/71e6883b-f08c-4440-bdce-3ebd280e56ae%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Zhiying Zhang]

Hi Issac,

I have just tried to use the latest code, and with the 15.0 guava, have got the same exception. Could you please help point out what I'm doing wrong?

Thanks,

Zhiying

[Mark]

I ran into similar issues some time ago (September 10).  Please have a look at my workaround here:  https://github.com/mark1900/druid-sandbox/blob/master/kafka-gobblin-hdfs-test/README.md

vim bin/gobblin-mapreduce.sh
 # Update Guava Library
 # Jars Gobblin runtime depends on
LIBJARS=$USER_JARS$separator$FWDIR_LIB/gobblin-metastore.jar,$FWDIR_LIB/gobblin-metrics.jar,\
$FWDIR_LIB/gobblin-core.jar,$FWDIR_LIB/gobblin-api.jar,$FWDIR_LIB/gobblin-utility.jar,\
$FWDIR_LIB/guava-15.0.jar,$FWDIR_LIB/avro-1.7.7.jar,$FWDIR_LIB/metrics-core-3.1.0.jar,\
$FWDIR_LIB/gson-2.3.1.jar,$FWDIR_LIB/joda-time-2.8.1.jar,$FWDIR_LIB/data-1.15.9.jar


./bin/gobblin-mapreduce.sh --jars ../kafka-gobblin-hdfs-test-0.0.0.jar,lib/guava-15.0.jar,lib/* --conf ../kafka-gobblin-hdfs-test.pull --workdir hdfs://server1:8020/gobblin/work

Related the Gobblin issues:

• https://github.com/linkedin/gobblin/pull/280
  
• https://github.com/linkedin/gobblin/pull/366