Error while using spark-cassandra connector in pyspark -> java.lang.NoClassDefFoundError: com/twitter/jsr166e/LongAdder
Pari Margu
I am getting the following error while using spark-cassandra connector in pyspark,
Create keyspace, table:
cqlsh> CREATE KEYSPACE test WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1 };
cqlsh> USE "test";
cqlsh:test> CREATE TABLE my_table(key text PRIMARY KEY, value int);
PYSPARK_PYTHON=/home/biadmin/anaconda2/bin/python pyspark --master local[4] --driver-memory 32g --executor-memory 16g --conf spark.cassandra.connection.host=aws-us-east-1-portal.11.dblayer.com --conf spark.cassandra.auth.username=scylla --conf spark.cassandra.auth.password=<PASSWORD> --conf spark.cassandra.connection.port=15600 --jars /home/biadmin/data/dbConnectorJars/spark-cassandra-connector-1.6.4-s_2.10.jar
my_table_RDD = sc.parallelize([{ "key": "key3", "value": 3 }, { "key": "key4", "value": 4 }])
df = sqlContext.createDataFrame(my_table_RDD)
df.write\
.format("org.apache.spark.sql.cassandra")\
.mode('append')\
.options(table="my_table", keyspace="test")\
.save()
java.lang.NoClassDefFoundError: com/twitter/jsr166e/LongAdder
Request you to help us to resolve the issue.
Spark version: 1.6.1
spark-cassandra jar: spark-cassandra-connector-1.6.4-s_2.10.jar
Full stack trace:
17/01/18 10:15:56 ERROR Executor: Exception in task 1.0 in stage 4.0 (TID 8)
java.lang.NoClassDefFoundError: com/twitter/jsr166e/LongAdder
at org.apache.spark.metrics.OutputMetricsUpdater$TaskMetricsSupport$class.$init$(OutputMetricsUpdater.scala:107)
at org.apache.spark.metrics.OutputMetricsUpdater$TaskMetricsUpdater.<init>(OutputMetricsUpdater.scala:151)
at org.apache.spark.metrics.OutputMetricsUpdater$.apply(OutputMetricsUpdater.scala:75)
at com.datastax.spark.connector.writer.TableWriter.writeInternal(TableWriter.scala:175)
at com.datastax.spark.connector.writer.TableWriter.insert(TableWriter.scala:163)
at com.datastax.spark.connector.writer.TableWriter.write(TableWriter.scala:150)
at com.datastax.spark.connector.RDDFunctions$$anonfun$saveToCassandra$1.apply(RDDFunctions.scala:36)
at com.datastax.spark.connector.RDDFunctions$$anonfun$saveToCassandra$1.apply(RDDFunctions.scala:36)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.twitter.jsr166e.LongAdder
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 14 more
17/01/18 10:15:56 ERROR Executor: Exception in task 3.0 in stage 4.0 (TID 10)
java.lang.NoClassDefFoundError: com/twitter/jsr166e/LongAdder
at org.apache.spark.metrics.OutputMetricsUpdater$TaskMetricsSupport$class.$init$(OutputMetricsUpdater.scala:107)
at org.apache.spark.metrics.OutputMetricsUpdater$TaskMetricsUpdater.<init>(OutputMetricsUpdater.scala:151)
at org.apache.spark.metrics.OutputMetricsUpdater$.apply(OutputMetricsUpdater.scala:75)
at com.datastax.spark.connector.writer.TableWriter.writeInternal(TableWriter.scala:175)
at com.datastax.spark.connector.writer.TableWriter.insert(TableWriter.scala:163)
at com.datastax.spark.connector.writer.TableWriter.write(TableWriter.scala:150)
at com.datastax.spark.connector.RDDFunctions$$anonfun$saveToCassandra$1.apply(RDDFunctions.scala:36)
at com.datastax.spark.connector.RDDFunctions$$anonfun$saveToCassandra$1.apply(RDDFunctions.scala:36)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
17/01/18 10:15:56 ERROR Executor: Exception in task 2.0 in stage 4.0 (TID 9)
java.lang.NoClassDefFoundError: com/twitter/jsr166e/LongAdder
at org.apache.spark.metrics.OutputMetricsUpdater$TaskMetricsSupport$class.$init$(OutputMetricsUpdater.scala:107)
at org.apache.spark.metrics.OutputMetricsUpdater$TaskMetricsUpdater.<init>(OutputMetricsUpdater.scala:151)
at org.apache.spark.metrics.OutputMetricsUpdater$.apply(OutputMetricsUpdater.scala:75)
at com.datastax.spark.connector.writer.TableWriter.writeInternal(TableWriter.scala:175)
at com.datastax.spark.connector.writer.TableWriter.insert(TableWriter.scala:163)
at com.datastax.spark.connector.writer.TableWriter.write(TableWriter.scala:150)
at com.datastax.spark.connector.RDDFunctions$$anonfun$saveToCassandra$1.apply(RDDFunctions.scala:36)
at com.datastax.spark.connector.RDDFunctions$$anonfun$saveToCassandra$1.apply(RDDFunctions.scala:36)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
17/01/18 10:15:56 ERROR Executor: Exception in task 0.0 in stage 4.0 (TID 7)
java.lang.NoClassDefFoundError: com/twitter/jsr166e/LongAdder
at org.apache.spark.metrics.OutputMetricsUpdater$TaskMetricsSupport$class.$init$(OutputMetricsUpdater.scala:107)
at org.apache.spark.metrics.OutputMetricsUpdater$TaskMetricsUpdater.<init>(OutputMetricsUpdater.scala:151)
at org.apache.spark.metrics.OutputMetricsUpdater$.apply(OutputMetricsUpdater.scala:75)
at com.datastax.spark.connector.writer.TableWriter.writeInternal(TableWriter.scala:175)
at com.datastax.spark.connector.writer.TableWriter.insert(TableWriter.scala:163)
at com.datastax.spark.connector.writer.TableWriter.write(TableWriter.scala:150)
at com.datastax.spark.connector.RDDFunctions$$anonfun$saveToCassandra$1.apply(RDDFunctions.scala:36)
at com.datastax.spark.connector.RDDFunctions$$anonfun$saveToCassandra$1.apply(RDDFunctions.scala:36)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
17/01/18 10:15:56 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker-6,5,main]
java.lang.NoClassDefFoundError: com/twitter/jsr166e/LongAdder
at org.apache.spark.metrics.OutputMetricsUpdater$TaskMetricsSupport$class.$init$(OutputMetricsUpdater.scala:107)
at org.apache.spark.metrics.OutputMetricsUpdater$TaskMetricsUpdater.<init>(OutputMetricsUpdater.scala:151)
at org.apache.spark.metrics.OutputMetricsUpdater$.apply(OutputMetricsUpdater.scala:75)
at com.datastax.spark.connector.writer.TableWriter.writeInternal(TableWriter.scala:175)
at com.datastax.spark.connector.writer.TableWriter.insert(TableWriter.scala:163)
at com.datastax.spark.connector.writer.TableWriter.write(TableWriter.scala:150)
at com.datastax.spark.connector.RDDFunctions$$anonfun$saveToCassandra$1.apply(RDDFunctions.scala:36)
at com.datastax.spark.connector.RDDFunctions$$anonfun$saveToCassandra$1.apply(RDDFunctions.scala:36)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
17/01/18 10:15:56 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker-5,5,main]
java.lang.NoClassDefFoundError: com/twitter/jsr166e/LongAdder
at org.apache.spark.metrics.OutputMetricsUpdater$TaskMetricsSupport$class.$init$(OutputMetricsUpdater.scala:107)
at org.apache.spark.metrics.OutputMetricsUpdater$TaskMetricsUpdater.<init>(OutputMetricsUpdater.scala:151)
at org.apache.spark.metrics.OutputMetricsUpdater$.apply(OutputMetricsUpdater.scala:75)
at com.datastax.spark.connector.writer.TableWriter.writeInternal(TableWriter.scala:175)
at com.datastax.spark.connector.writer.TableWriter.insert(TableWriter.scala:163)
at com.datastax.spark.connector.writer.TableWriter.write(TableWriter.scala:150)
at com.datastax.spark.connector.RDDFunctions$$anonfun$saveToCassandra$1.apply(RDDFunctions.scala:36)
at com.datastax.spark.connector.RDDFunctions$$anonfun$saveToCassandra$1.apply(RDDFunctions.scala:36)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
17/01/18 10:15:56 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker-4,5,main]
java.lang.NoClassDefFoundError: com/twitter/jsr166e/LongAdder
at org.apache.spark.metrics.OutputMetricsUpdater$TaskMetricsSupport$class.$init$(OutputMetricsUpdater.scala:107)
at org.apache.spark.metrics.OutputMetricsUpdater$TaskMetricsUpdater.<init>(OutputMetricsUpdater.scala:151)
at org.apache.spark.metrics.OutputMetricsUpdater$.apply(OutputMetricsUpdater.scala:75)
at com.datastax.spark.connector.writer.TableWriter.writeInternal(TableWriter.scala:175)
at com.datastax.spark.connector.writer.TableWriter.insert(TableWriter.scala:163)
at com.datastax.spark.connector.writer.TableWriter.write(TableWriter.scala:150)
at com.datastax.spark.connector.RDDFunctions$$anonfun$saveToCassandra$1.apply(RDDFunctions.scala:36)
at com.datastax.spark.connector.RDDFunctions$$anonfun$saveToCassandra$1.apply(RDDFunctions.scala:36)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
17/01/18 10:15:56 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker-3,5,main]
java.lang.NoClassDefFoundError: com/twitter/jsr166e/LongAdder
at org.apache.spark.metrics.OutputMetricsUpdater$TaskMetricsSupport$class.$init$(OutputMetricsUpdater.scala:107)
at org.apache.spark.metrics.OutputMetricsUpdater$TaskMetricsUpdater.<init>(OutputMetricsUpdater.scala:151)
at org.apache.spark.metrics.OutputMetricsUpdater$.apply(OutputMetricsUpdater.scala:75)
at com.datastax.spark.connector.writer.TableWriter.writeInternal(TableWriter.scala:175)
at com.datastax.spark.connector.writer.TableWriter.insert(TableWriter.scala:163)
at com.datastax.spark.connector.writer.TableWriter.write(TableWriter.scala:150)
at com.datastax.spark.connector.RDDFunctions$$anonfun$saveToCassandra$1.apply(RDDFunctions.scala:36)
at com.datastax.spark.connector.RDDFunctions$$anonfun$saveToCassandra$1.apply(RDDFunctions.scala:36)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.twitter.jsr166e.LongAdder
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 14 more
17/01/18 10:15:56 INFO SparkContext: Invoking stop() from shutdown hook
17/01/18 10:15:56 WARN TaskSetManager: Lost task 2.0 in stage 4.0 (TID 9, localhost): java.lang.NoClassDefFoundError: com/twitter/jsr166e/LongAdder
at org.apache.spark.metrics.OutputMetricsUpdater$TaskMetricsSupport$class.$init$(OutputMetricsUpdater.scala:107)
at org.apache.spark.metrics.OutputMetricsUpdater$TaskMetricsUpdater.<init>(OutputMetricsUpdater.scala:151)
at org.apache.spark.metrics.OutputMetricsUpdater$.apply(OutputMetricsUpdater.scala:75)
at com.datastax.spark.connector.writer.TableWriter.writeInternal(TableWriter.scala:175)
at com.datastax.spark.connector.writer.TableWriter.insert(TableWriter.scala:163)
at com.datastax.spark.connector.writer.TableWriter.write(TableWriter.scala:150)
at com.datastax.spark.connector.RDDFunctions$$anonfun$saveToCassandra$1.apply(RDDFunctions.scala:36)
at com.datastax.spark.connector.RDDFunctions$$anonfun$saveToCassandra$1.apply(RDDFunctions.scala:36)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
17/01/18 10:15:56 ERROR TaskSetManager: Task 2 in stage 4.0 failed 1 times; aborting job
17/01/18 10:15:56 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool
17/01/18 10:15:56 INFO TaskSetManager: Lost task 0.0 in stage 4.0 (TID 7) on executor localhost: java.lang.NoClassDefFoundError (com/twitter/jsr166e/LongAdder) [duplicate 1]
17/01/18 10:15:56 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool
17/01/18 10:15:56 INFO TaskSetManager: Lost task 3.0 in stage 4.0 (TID 10) on executor localhost: java.lang.NoClassDefFoundError (com/twitter/jsr166e/LongAdder) [duplicate 2]
17/01/18 10:15:56 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool
17/01/18 10:15:56 INFO TaskSetManager: Lost task 1.0 in stage 4.0 (TID 8) on executor localhost: java.lang.NoClassDefFoundError (com/twitter/jsr166e/LongAdder) [duplicate 3]
17/01/18 10:15:56 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool
17/01/18 10:15:56 INFO TaskSchedulerImpl: Cancelling stage 4
17/01/18 10:15:56 INFO DAGScheduler: ResultStage 4 (runJob at RDDFunctions.scala:36) failed in 0.057 s
17/01/18 10:15:56 INFO DAGScheduler: Job 4 failed: runJob at RDDFunctions.scala:36, took 0.074074 s
Traceback (most recent call last):
File "<stdin>", line 4, in <module>
File "/usr/iop/4.2.0.0/spark/python/pyspark/sql/readwriter.py", line 395, in save
self._jwrite.save()
File "/usr/iop/4.2.0.0/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 813, in __call__
File "/usr/iop/4.2.0.0/spark/python/pyspark/sql/utils.py", line 45, in deco
return f(*a, **kw)
File "/usr/iop/4.2.0.0/spark/python/lib/py4j-0.9-src.zip/py4j/protocol.py", line 308, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o96.save.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 4.0 failed 1 times, most recent failure: Lost task 2.0 in stage 4.0 (TID 9, localhost): java.lang.NoClassDefFoundError: com/twitter/jsr166e/LongAdder
at org.apache.spark.metrics.OutputMetricsUpdater$TaskMetricsSupport$class.$init$(OutputMetricsUpdater.scala:107)
at org.apache.spark.metrics.OutputMetricsUpdater$TaskMetricsUpdater.<init>(OutputMetricsUpdater.scala:151)
at org.apache.spark.metrics.OutputMetricsUpdater$.apply(OutputMetricsUpdater.scala:75)
at com.datastax.spark.connector.writer.TableWriter.writeInternal(TableWriter.scala:175)
at com.datastax.spark.connector.writer.TableWriter.insert(TableWriter.scala:163)
at com.datastax.spark.connector.writer.TableWriter.write(TableWriter.scala:150)
at com.datastax.spark.connector.RDDFunctions$$anonfun$saveToCassandra$1.apply(RDDFunctions.scala:36)
at com.datastax.spark.connector.RDDFunctions$$anonfun$saveToCassandra$1.apply(RDDFunctions.scala:36)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1922)
at com.datastax.spark.connector.RDDFunctions.saveToCassandra(RDDFunctions.scala:36)
at org.apache.spark.sql.cassandra.CassandraSourceRelation.insert(CassandraSourceRelation.scala:67)
at org.apache.spark.sql.cassandra.DefaultSource.createRelation(DefaultSource.scala:85)
at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:222)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:148)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381)
at py4j.Gateway.invoke(Gateway.java:259)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:209)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoClassDefFoundError: com/twitter/jsr166e/LongAdder
at org.apache.spark.metrics.OutputMetricsUpdater$TaskMetricsSupport$class.$init$(OutputMetricsUpdater.scala:107)
at org.apache.spark.metrics.OutputMetricsUpdater$TaskMetricsUpdater.<init>(OutputMetricsUpdater.scala:151)
at org.apache.spark.metrics.OutputMetricsUpdater$.apply(OutputMetricsUpdater.scala:75)
at com.datastax.spark.connector.writer.TableWriter.writeInternal(TableWriter.scala:175)
at com.datastax.spark.connector.writer.TableWriter.insert(TableWriter.scala:163)
at com.datastax.spark.connector.writer.TableWriter.write(TableWriter.scala:150)
at com.datastax.spark.connector.RDDFunctions$$anonfun$saveToCassandra$1.apply(RDDFunctions.scala:36)
at com.datastax.spark.connector.RDDFunctions$$anonfun$saveToCassandra$1.apply(RDDFunctions.scala:36)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more
>>> 17/01/18 10:15:56 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/static/sql,null}
17/01/18 10:15:56 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL/execution/json,null}
17/01/18 10:15:56 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL/execution,null}
17/01/18 10:15:56 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL/json,null}
17/01/18 10:15:56 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL,null}
17/01/18 10:15:56 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null}
17/01/18 10:15:56 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
17/01/18 10:15:56 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null}
17/01/18 10:15:56 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null}
17/01/18 10:15:56 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null}
17/01/18 10:15:56 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
17/01/18 10:15:56 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null}
17/01/18 10:15:56 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null}
17/01/18 10:15:56 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null}
17/01/18 10:15:56 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null}
17/01/18 10:15:56 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null}
17/01/18 10:15:56 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
17/01/18 10:15:56 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null}
17/01/18 10:15:56 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null}
17/01/18 10:15:56 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null}
17/01/18 10:15:56 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null}
17/01/18 10:15:56 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null}
17/01/18 10:15:56 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null}
17/01/18 10:15:56 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null}
17/01/18 10:15:56 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null}
17/01/18 10:15:56 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null}
17/01/18 10:15:56 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null}
17/01/18 10:15:56 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null}
17/01/18 10:15:56 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null}
17/01/18 10:15:56 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null}
17/01/18 10:15:56 INFO SparkUI: Stopped Spark web UI at http://bi4c-11709-mastermanager.bi.services.bluemix.net:4041
17/01/18 10:15:56 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
17/01/18 10:15:56 INFO MemoryStore: MemoryStore cleared
17/01/18 10:15:56 INFO BlockManager: BlockManager stopped
17/01/18 10:15:56 INFO BlockManagerMaster: BlockManagerMaster stopped
17/01/18 10:15:56 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
17/01/18 10:15:56 INFO SparkContext: Successfully stopped SparkContext
17/01/18 10:15:56 INFO ShutdownHookManager: Shutdown hook called
17/01/18 10:15:56 INFO ShutdownHookManager: Deleting directory /tmp/spark-ba89a6ec-aef7-4ec9-a464-49a410481e85/httpd-fe18195b-5741-46ac-9548-cbae2a111c79
17/01/18 10:15:56 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
17/01/18 10:15:56 INFO ShutdownHookManager: Deleting directory /tmp/spark-62fc7a83-c971-458b-81ca-72ca808b354b
17/01/18 10:15:56 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
17/01/18 10:15:56 INFO ShutdownHookManager: Deleting directory /tmp/spark-ba89a6ec-aef7-4ec9-a464-49a410481e85
17/01/18 10:15:56 INFO ShutdownHookManager: Deleting directory /tmp/spark-ba89a6ec-aef7-4ec9-a464-49a410481e85/pyspark-2ad07adc-3924-4c2c-ac8a-f96ba3693d6e
17/01/18 10:15:56 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
17/01/18 10:15:58 INFO CassandraConnector: Disconnected from Cassandra cluster: vigorous-scylladb-24
17/01/18 10:15:58 INFO SerialShutdownHooks: Successfully executed shutdown hook: Clearing session cache for C* connector
Russell Spitzer
--
You received this message because you are subscribed to the Google Groups "DataStax Spark Connector for Apache Cassandra" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spark-connector-...@lists.datastax.com.
Russell Spitzer
Software Engineer
Inderjeet Singh
Is there any way to download all the dependencies or a single assembly jar for offline access to the dependencies.?
We are behind the company Firewall.
thanks
Inderjeet
> Software Engineer