CassandraConnectorSource not on executor classpath

[Max Pimm]

I am using a standalone cluster and have configured the connector source as per the docs. I get metrics on my driver but not on my executors. I see a ClassNotFoundException in the stderr of my job.

java.lang.ClassNotFoundException: org.apache.spark.metrics.CassandraConnectorSource

This happens before my application jar (which bundles the connector classes) is fetched from the driver.

It looks like the connector and it's dependencies need to be added to the classpath of the spark slave on launch.

Is this to be expected?

How should I add dependencies to the default spark classpath?

[Russell Spitzer]

Yes, the Connector code must be on the executor classpath,
Either using --packages 

 --packages com.datastax.spark:spark-cassandra-connector_2.10:1.5.0-M2

or 

--jars path/to/your/spark-cassandra-connector-assembly.jar

--
You received this message because you are subscribed to the Google Groups "DataStax Spark Connector for Apache Cassandra" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spark-connector-...@lists.datastax.com.

--

http://datastax.com/all/images/cs_logo_color_sm.png

[Max Pimm]

I presume that you mean as part of the spark submit command?

This does not work for me, It seems that the connector needs to be on the classpath of the worker when it launches (before the executor has been created). I've achieved this by adding it to the SPARK_CLASSPATH environmental.

[Russell Spitzer]

Hmm I remember there is something about this, Jacek do you remember?

--

http://datastax.com/all/images/cs_logo_color_sm.png

[dev.rabin...@gmail.com]

Facing the same issue,

Connector used:
https://mvnrepository.com/artifact/com.datastax.spark/spark-cassandra-connector_2.10/1.6.0

java.lang.ClassNotFoundException: org.apache.spark.metrics.CassandraConnectorSource
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:175)
at org.apache.spark.metrics.MetricsSystem$$anonfun$registerSources$1.apply(MetricsSystem.scala:174)
at org.apache.spark.metrics.MetricsSystem$$anonfun$registerSources$1.apply(MetricsSystem.scala:171)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
at org.apache.spark.metrics.MetricsSystem.registerSources(MetricsSystem.scala:171)
at org.apache.spark.metrics.MetricsSystem.start(MetricsSystem.scala:99)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:384)
at org.apache.spark.SparkEnv$.createExecutorEnv(SparkEnv.scala:217)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:183)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:69)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:68)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:68)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:148)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:250)
at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)

[Jayant Gope]

Hey, did you figure it out?