How to resolve the conflict between 11.0.2 and 16.0 of guava when using yarn, spark and spark-cassandra-connector?

[Dillon Peng]

my yarn's version is hadop-2.4.0.x, spark is spark-1.5.1-bin-hadoop2.4 and spark-cassandra-connector is spark-cassandra-connector_2.10-1.5.0-M2, when I executed the following command:

bin/spark-shell --driver-class-path $(echo lib/*.jar | sed 's/ /:/g')  --master yarn-client 
--deploy-mode client --conf spark.cassandra.connection.host=192.21.0.209 
--conf spark.cassandra.auth.username=username --conf spark.cassandra.auth.password=password --conf spark.sql.dialect=sql 
--jars lib/guava-16.0.jar,spark-cassandra-connector_2.10-1.5.0-M2.jar,lib/cassandra-driver-core-2.2.0-rc3.jar

After starting, I input the following scala under the prompt:

import org.apache.spark.sql.cassandra.CassandraSQLContext
import org.apache.spark.sql.{DataFrame, SaveMode}
import org.apache.spark.{Logging, SparkConf, SparkContext}
import org.joda.time.{DateTime, Days, LocalDate}
val cc = new CassandraSQLContext(sc)

val rdd: DataFrame = cc.sql("select user_id,tag_models,dmp_province," +
"zp_gender,zp_age,zp_edu,stg_stage,zp_income,type " +
"from user_center.users_test") 

I got the classic error:

Caused by: java.lang.NoSuchMethodError:  
com.google.common.util.concurrent.Futures.withFallback
(Lcom/google/common/util/concurrent/ListenableFuture;
Lcom/google/common/util/concurrent/FutureFallback;
Ljava/util/concurrent/Executor;)
Lcom/google/common/util/concurrent/ListenableFuture;

After search this error in google and stackoverflower, I know that the conflict between the different versions of guava caused this error, and found hadoop 2.4 use guava-11.0.2 but spark-cassandra-connector_2.10-1.5.0-M2 use guava-16.0.1.

How to resolve this kind of error, any advice will be appreciated!

[Dillon Peng]

hi, all
Is there any one has the same question? I need your help or advice, Thank you!

Dillon

[matty...@gmail.com]

Hi Peng,

Just forwarded you a thread that shows how to resolve this.

I will post on the FAQ shortly as well

  Original Message  
From: Dillon Peng
Sent: Saturday, March 26, 2016 12:31 AM
To: DataStax Spark Connector for Apache Cassandra
Reply To: spark-conn...@lists.datastax.com
Subject: Re: How to resolve the conflict between 11.0.2 and 16.0 of guava when using yarn, spark and spark-cassandra-connector?

--
You received this message because you are subscribed to the Google Groups "DataStax Spark Connector for Apache Cassandra" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spark-connector-...@lists.datastax.com.

[Dillon Peng]

Hi Matt
Thank you very much!
I can't wait to know the resolution!

Dillon Peng

[Dillon Peng]

hi, Matt

    Several days past, I did not get your resolution and FAQ link.  I don't know whether you can give me the resolution at your free time, Thanks a lot!

Dillon

[matty...@gmail.com]

‎Check your junk mail maybe? I forwarded you an email a couple days ago. I'll Re forward now too 

[vincent gromakowski]

Can you share it for everybody please ?

[matty...@gmail.com]

Done

[Dillon Peng]

hi, Matt
I received the mail two times, I am sorry for carelessly ignoring it!
I will check the resolution in the next week, thank you very much!

[Varun Kumar]

Hello All,

I am still not able to find the solution mail. Can someone please be kind enough to pass it on to me again. 

Thanks!

--
You received this message because you are subscribed to the Google Groups "DataStax Spark Connector for Apache Cassandra" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spark-connector-...@lists.datastax.com.

--

Varun Kumar

Developer

ReadyPulse Inc. | 8429 154th Ave NE, Redmond WA 98052

www.ReadyPulse.com  

c: +1-206-310-7665

NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information.  Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.

[Alexander Kasper]

When using spark-submit, I passed --conf spark.driver.userClassPathFirst=true which solved this issue for me.

[Dillon Peng]

hi, all
I am so sorry for testing long time!

Now, for spark-submit, I tested this resolution [Making Hadoop 2.6 + Spark-Cassandra Driver Play Nice Together](http://arjon.es/2015/10/12/making-hadoop-2-dot-6-plus-spark-cassandra-driver-play-nice-together/) successfully under my test yarn cluster

On Tuesday, March 15, 2016 at 2:28:42 PM UTC+8, Dillon Peng wrote:

[Benyi Wang]

You can resolve this guava conflict by add '--conf spark.driver.extraClassPath' and '--conf spark.executor.extraClassPath', and you don't have to compile something or shade them.

I wrote a blog to describe the detail. http://ben-tech.blogspot.com/2016/04/how-to-resolve-spark-cassandra.html

[Krish]

On Friday, April 15, 2016 at 2:10:56 PM UTC-4, Benyi Wang wrote:
> You can resolve this guava conflict by add '--conf spark.driver.extraClassPath' and '--conf spark.executor.extraClassPath', and you don't have to compile something or shade them.
>
> I wrote a blog to describe the detail. http://ben-tech.blogspot.com/2016/04/how-to-resolve-spark-cassandra.html

Do we know, how to handle this when using Jupyter Notebook, instead of spark-submit?

[Russell Spitzer]

On the recent releases you should be able to just use the SparkPackages distribution, so --packages still 

[Krish]

Thanks Russell. I am not sure if I got your full message.

I am using Jupyter notebook from Azure's HDInsight Spark cluster. I tried to connect to DSE which is on Azure as well. When I use SBT to build the FAT Jar I am able to run a Spark Streaming application both in local and yarn mode. But when I run the same code from Jupyter Notebook, I get the java.lang.NoSuchMethodError:
com.google.common.util.concurrent.Futures.withFallback issue. I am not sure how I can avoid the classpath conflicts with Jupyter. BTW - I have a ticket open with Microsoft Azure team.

Thanks

Krish

[Russell Spitzer]

when i start jupyter I do it via the pyspark command, so there you just add --packages datastax.... Not sure how the azure version works

--

Russell Spitzer
Software Engineer

https://github.com/datastax/spark-cassandra-connector/blob/master/doc/FAQ.md
http://spark-packages.org/package/datastax/spark-cassandra-connector

[Mario Gerardo Fonseca]

You could package your project in a "fat jar" whith the guava 14(or higher) dependencies "shaded" with the package overwritten to avoid the conflict. If you are using maven, you could use the Shade Plugin. In the following example I will overwrite the package "com.google" with "hidden.google". Then you could run your fat jar in Hadoop. Add this lines to your pom file in Maven:

<build>
<finalName>nameoffatjar</finalName>
<plugins>
<!-- Maven Shade Plugin -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<executions>
<!-- Run shade goal on package phase -->
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<!-- add Main-Class to manifest file -->
</transformers>
<relocations>
<relocation>
<pattern>com.google</pattern>
<shadedPattern>hidden.google</shadedPattern>
</relocation>
</relocations>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>