Exception in thread "main" java.lang.ExceptionInInitializerError at com.datastax.spark.connector.cql.DefaultConnectionFactory$.clusterBuilder(CassandraConnectionFact35)

[parag]

Error
*****
16/03/11 18:48:02 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/03/11 18:48:03 WARN Utils: Your hostname, diotist01 resolves to a loopback address: 127.0.1.1; using 10.88.23.12 instead (on interface eth0)
16/03/11 18:48:03 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
16/03/11 18:48:04 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set

Exception in thread "main" java.lang.ExceptionInInitializerError
at com.datastax.spark.connector.cql.DefaultConnectionFactory$.clusterBuilder(CassandraConnectionFact35)
....
...
...
..
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.IllegalStateException: Detected Guava issue #1635 which indicates that a version of Guaan 16.01 is in use. This introduces codec resolution issues and potentially other incompatibility issues inr. Please upgrade to Guava 16.01 or later.
at com.datastax.driver.core.SanityChecks.checkGuava(SanityChecks.java:62)
at com.datastax.driver.core.SanityChecks.check(SanityChecks.java:36)
at com.datastax.driver.core.Cluster.<clinit>(Cluster.java:67)

Versions -
**************
Cassandra - 2.2.4 on cluster
Spark -1.5.2 on cluster
Kafka - kafka_2.10-0.8.2.1
Spark cassandra driver -

spark submit -
*****************
parag@diotist01:~/dev/scripts/test1$ spark-submit --class "SimpleApp" --master spark://10.88.23.13:7077 --jars lib/spark-streaming-kafka_2.10-1.5.2.jar,lib/kafka_2.10-0.8.2.1.jar,lib/kafka-clients-0.8.2.1.jar,lib/metrics-core-2.2.0.jar,lib/spark-cassandra-connector-assembly-1.5.0.jar target/scala-2.10/test1_2.10-1.0.jar
.
fat jar was created for driver

Please guide how to resolve this issue

[Matt Black]

This worked for me with connector version 1.6.0-M1, spark 1.6.0, and cassandra 2.2.3:

Changed your project/assembly.sbt to look like below line so that you use Version 0.14 of sbt:

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.2")

Change your build.sbt to shade com.google.guava (need sbt 0.14 for this) and then have a merge strategy that takes the latest of each.

Add this to the end of your build.sbt (not sure if it matters where):

// There is a conflict between Guava versions on Cassandra Drive and Cassandra
// Core
// Shading Guava Package
assemblyShadeRules in assembly := Seq(
ShadeRule.rename("com.google.**" -> "shadeio.@1").inAll
)

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
{
case PathList("META-INF", "MANIFEST.MF") => MergeStrategy.discard
case _ => MergeStrategy.last

[Russell Spitzer]

Hey Matt would you mind adding a PR too add instructions for that to the FAQ? I could always to it as well but I'd like to make sure you get the credit :)

--
You received this message because you are subscribed to the Google Groups "DataStax Spark Connector for Apache Cassandra" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spark-connector-...@lists.datastax.com.

--

https://github.com/datastax/spark-cassandra-connector/blob/master/doc/FAQ.md
http://spark-packages.org/package/datastax/spark-cassandra-connector

[matty...@gmail.com]

Ya, if you can verify it works for you too then sure.

Later,

Matt

From: Russell Spitzer Sent: Friday, March 11, 2016 5:53 PM To: DataStax Spark Connector for Apache Cassandra Reply To: spark-conn...@lists.datastax.com Subject: Re: Exception in thread "main" java.lang.ExceptionInInitializerError at com.datastax.spark.connector.cql.DefaultConnectionFactory$.clusterBuilder(CassandraConnectionFact35)

[parag]

Thanks Matt, I made the suggested changes, I am still getting same errors.
My build.sbt
**************

name := "test1"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.5.2" % "provided",
"org.apache.spark" %% "spark-streaming" % "1.5.2" % "provided"
)

assemblyShadeRules in assembly := Seq(
ShadeRule.rename("com.google.**" -> "shadeio.@1").inAll
)

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
{
case PathList("META-INF", "MANIFEST.MF") => MergeStrategy.discard
case _ => MergeStrategy.last
}
}

assembly.sbt
**************

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.2")

Please guide
Parag

[parag]

On Friday, March 11, 2016 at 7:01:25 PM UTC, parag wrote:

Hello
I have in my lib folder depended jars -
kafka_2.10-0.8.2.1.jar spark-cassandra-connector-assembly-1.5.0.jar
kafka-clients-0.8.2.1.jar spark-cassandra-connector-java-assembly-1.5.0.jar
metrics-core-2.2.0.jar

My build.sbt as follows -
name := "test1"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.5.2" % "provided",

"org.apache.spark" %% "spark-streaming" % "1.5.2" % "provided",
("org.apache.spark" % "spark-streaming-kafka_2.10" % "1.5.2").
exclude("org.apache.spark","spark-core_2.10")
)

assemblyShadeRules in assembly := Seq(
ShadeRule.rename("com.google.**" -> "shadeio.@1").inAll
)

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
{
case PathList("META-INF", "MANIFEST.MF") => MergeStrategy.discard
case _ => MergeStrategy.last
}
}

I am giving command sbt assembly to create jar
and then following spark-submit command -
spark-submit --class "SimpleApp" --master spark://10.88.23.13:7077 target/scala-2.10/test1-assembly-1.0.jar

I still get same errors related to guava .
Caused by: java.lang.IllegalStateException: Detected Guava issue #1635 which indicates that a version of Guava less than 16.01 is in use. This introduces codec resolution issues and potentially other incompatibility issues in the driver. Please upgrade to Guava 16.01 or later.

When i remove "provided" from build.sbt i get
Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:287)

****************
Please guide me what mistake I am doing

Thanks
Parag

[matty...@gmail.com]

Hi Parag,

I'm using spark 1.6, also 1.6 of the connector. try upgrading and see if it takes for you

  Original Message  
From: parag
Sent: Friday, March 11, 2016 6:30 PM

To: DataStax Spark Connector for Apache Cassandra
Reply To: spark-conn...@lists.datastax.com

Subject: Re: Exception in thread "main" java.lang.ExceptionInInitializerError at com.datastax.spark.connector.cql.DefaultConnectionFactory$.clusterBuilder(CassandraConnectionFact35)

[parag parandkar]

Hi Matt

Kindly let me know which version of Kafka and kafka spark client do you use

Best Regards

Parag

[matty...@gmail.com]

Hi Parag,

Not using streaming, so no Kafka spark. Still getting the same error after bumping up to 1.6?

[parag parandkar]

Hi Matt, I will try 1.6 and get back

Regards

Parag

[matty...@gmail.com]

K, sounds good. I also make a fat  jar for my whole app, you should try that too.‎ I saw fat jar mentioned in your post and thought you meant the whole thing, not just the connector.

[parag parandkar]

Thank you Mat

Your solution works on Spark 1.6

Thanks a lot

Best Regards

Parag

[matty...@gmail.com]

No problem

[matty...@gmail.com]

These worked for myself and confirmed by one other to get around conflicting versions of guava.

From: matty...@gmail.com Sent: Wednesday, March 30, 2016 8:19 AM To: Dillon Peng Subject: Fw: Exception in thread "main" java.lang.ExceptionInInitializerError at com.datastax.spark.connector.cql.DefaultConnectionFactory$.clusterBuilder(CassandraConnectionFact35)

From: matty...@gmail.com Sent: Saturday, March 26, 2016 8:41 AM To: Dillon Peng Subject: Fw: Exception in thread "main" java.lang.ExceptionInInitializerError at com.datastax.spark.connector.cql.DefaultConnectionFactory$.clusterBuilder(CassandraConnectionFact35)