Which spark-cassandra-connector should be used with DSE 6.18 ?

[Alex G]

Hello all community,

We are trying to switch from Apache Cassandra 3.11 to DSE 6.18.15

but our spark-streaming job is failing :

com.datastax.driver.core.exceptions.UnresolvedUserTypeException: Cannot resolve user type system_distributed.duration

at com.datastax.driver.core.DataTypeCqlNameParser.parse(DataTypeCqlNameParser.java:147)

Please share how to find appropriate spark-сassandra driver which would work with our code ?
Previously we have used this one (2.11/2.4.2) and everything is working fine on prod.

https://mvnrepository.com/artifact/com.datastax.spark/spark-cassandra-connector_2.11/2.4.2

<!-- https://mvnrepository.com/artifact/com.datastax.spark/spark-cassandra-connector -->

<dependency>

    <groupId>com.datastax.spark</groupId>

    <artifactId>spark-cassandra-connector_2.11</artifactId>

    <version>2.4.2</version>

</dependency>

Thank you for help.

[Alex Ott]

You may try following dependency instead (if you're running on Spark that is part of DSE):

<dependency>

<groupId>com.datastax.dse</groupId>

<artifactId>dse-spark-dependencies</artifactId>

<version>${dse.spark.version}</version>

<scope>provided</scope>

</dependency>

https://github.com/alexott/cassandra-dse-playground/blob/master/spark-dse/pom.xml#L23

I don't remember what version is in 6.8, most probably 2.4, but it could be heavily modified.

--
To unsubscribe from this group and stop receiving emails from it, send an email to spark-connector-...@lists.datastax.com.

--

With best wishes,                    Alex Ott
http://alexott.net/
Twitter: alexott_en (English), alexott (Russian)