Getting timeout when running on spark

[E.T.]

I am trying to query data from Cassandra using Spark and the connector. Regular jobs run just fine on Spark and using cqlsh I am perfectly able to get the data I want in less than 5 seconds. When submitting a job however, I get a timeout exception.

My code is as follows (everything runs locally):

val conf = new SparkConf(true)
.set("spark.cassandra.connection.host", "127.0.0.1")
.setMaster("localhost")
.setAppName("Cassandra-Test")
val sc = new SparkContext(conf)

// Obtain RDD from keyspace/table
val rdd = sc.cassandraTable("test", "testtable")
println(rdd.count)
println(rdd.first)
println(rdd.map(_.getInt("col1")).count)

The error I get, is as follows:

15/01/08 08:48:36 WARN AkkaUtils: Error sending message in 1 attempts
java.util.concurrent.TimeoutException: Futures timed out after [30 seconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread$$anon$3.block(ThreadPoolBuilder.scala:169)
at scala.concurrent.forkjoin.ForkJoinPool.managedBlock(ForkJoinPool.java:3640)
at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread.blockOn(ThreadPoolBuilder.scala:167)
at scala.concurrent.Await$.result(package.scala:107)
at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:176)
at org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:218)
at org.apache.spark.storage.BlockManagerMaster.tell(BlockManagerMaster.scala:208)
at org.apache.spark.storage.BlockManagerMaster.registerBlockManager(BlockManagerMaster.scala:47)
at org.apache.spark.storage.BlockManager.initialize(BlockManager.scala:172)
at org.apache.spark.storage.BlockManager.<init>(BlockManager.scala:142)
at org.apache.spark.storage.BlockManager.<init>(BlockManager.scala:163)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:230)
at org.apache.spark.executor.Executor.<init>(Executor.scala:77)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$receiveWithLogging$1.applyOrElse(CoarseGrainedExecutorBackend.scala:60)
at scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
at org.apache.spark.util.ActorLogReceive$$anon$1.apply(ActorLogReceive.scala:53)
at org.apache.spark.util.ActorLogReceive$$anon$1.apply(ActorLogReceive.scala:42)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118)
at org.apache.spark.util.ActorLogReceive$$anon$1.applyOrElse(ActorLogReceive.scala:42)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

[Helena Edelson]

And you are using the same SparkConf settings for regular jobs w/out the connector that are passing?
This seems like a startup issue - not yet sure why. Can you add your build file w dependency versions?

[E Tromp]

Well it seems like it's not related to Cassandra or the connector... I cannot run _any_ Spark job on the cluster (shell and local work just fine) -- a simple count on lines of a file gives the same error...

2015-01-08 18:14 GMT+01:00 Helena Edelson <helena....@datastax.com>:

And you are using the same SparkConf settings for regular jobs w/out the connector that are passing?
This seems like a startup issue - not yet sure why. Can you add your build file w dependency versions?

To unsubscribe from this group and stop receiving emails from it, send an email to spark-connector-...@lists.datastax.com.

--

Met vriendelijke groet,
Erik Tromp
Linkedin