Re: Compatibility issue Connector 3.0.0 and Databricks 7.3LTS

[Russell Spitzer]

Looks like you need to shade a bunch of things rather than just including the raw dependencies. There are collisions with at least TypeSafe Config but there will probably be ones with other deps as well. I would suggest using the maven package which shade a bunch of things rather than building your own jar. It's usually the wrong thing to do if you have to list out the dependencies by hand.

On Thu, Oct 15, 2020 at 12:22 PM Nishant J <ico...@gmail.com> wrote:

Is there any compatibility issue between the Connector and Databricks 7.3LTS (Spark 3.0.1, Scala 2.12). I created a cluster with spark connector (spark-cassandra-connector_2.12-3.0.0.jar) and all its transient dependencies however I get this error on a simple select statement - 

java.io.IOException: Failed to open native connection to Cassandra at {10.255.7.60:9042} :: com.typesafe.config.impl.ConfigImpl.newSimpleOrigin(Ljava/lang/String;)Lcom/typesafe/config/ConfigOrigin;

ROOT cause - 

Caused by: java.lang.NoSuchMethodError: com.typesafe.config.impl.ConfigImpl.newSimpleOrigin(Ljava/lang/String;)Lcom/typesafe/config/ConfigOrigin; at com.typesafe.config.ConfigOriginFactory.newSimple(ConfigOriginFactory.java:42)

I uploaded the connector jar and all of its dependencies to the cluster. Here is the list of dependencies that maven pulled for me - 

HdrHistogram-2.1.11.jar

commons-lang3-3.9.jar

config-1.3.4.jar

java-driver-core-shaded-4.7.2.jar

java-driver-mapper-runtime-4.7.2.jar

java-driver-query-builder-4.7.2.jar

java-driver-shaded-guava-25.1-jre-graal-sub-1.jar

javatuples-1.2.jar

jcip-annotations-1.0-1.jar

jsr305-3.0.2.jar

metrics-core-4.0.5.jar

native-protocol-1.4.10.jar

paranamer-2.8.jar

reactive-streams-1.0.2.jar

scala-library-2.12.11.jar

scala-reflect-2.12.11.jar

slf4j-api-1.7.26.jar

spark-cassandra-connector-driver_2.12-3.0.0.jar

spark-cassandra-connector_2.12-3.0.0.jar

spotbugs-annotations-3.1.12.jar

--
To unsubscribe from this group and stop receiving emails from it, send an email to spark-connector-...@lists.datastax.com.

[Nishant J]

Thanks for the reply. Not sure I follow when you say "maven package". Is it a shaded version of the 3.0 connector? If so, can you point me to the coordinates to it in a repository.

I didn't assemble the dependencies by hand, but just let maven collect all the transitive dependencies when I added the com.datastax.spark:spark-cassandra-connector_2.12:3.0.0 to my pom.

[Alex Ott]

It's answered in the another thread - check there.

--

With best wishes,                    Alex Ott
http://alexott.net/
Twitter: alexott_en (English), alexott (Russian)