SparkEnv is not thread safe

[liu]

Hi,

The SparkEnv is not thread safe, and as Matei suggested we cannot use two or more active SparkContext instances at the same time, how to run multiple spark jobs at same time?

Thank you

[Patrick Wendell]

Hey Lui,

First of all, you can always submit multiple Spark jobs which each have their own context. They will each register with the scheduler and assume resources.

Second, can also use multiple threads which each have a reference to a single SparkContext within a single JVM.

- Patrick

--
You received this message because you are subscribed to the Google Groups "Spark Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spark-users...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[liu]

Thank you for quick reply. I tried these two methods: separate context for each thread or one SparkContext, it either throws NullPointerException or "Address in use". None of them works so far. Is there some tricks there?

Thank you

[liu]

BTW, log is here:

java.lang.NullPointerException

        at scala.collection.JavaConversions$JListWrapper.length(JavaConversions.scala:616)

        at scala.collection.SeqLike$class.size(SeqLike.scala:100)

        at scala.collection.JavaConversions$JListWrapper.size(JavaConversions.scala:615)

        at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:233)

        at scala.collection.JavaConversions$JListWrapper.toArray(JavaConversions.scala:615)

        at spark.ParallelCollection$.slice(ParallelCollection.scala:86)

        at spark.ParallelCollection.<init>(ParallelCollection.scala:35)

        at spark.SparkContext.parallelize(SparkContext.scala:195)

        at spark.api.java.JavaSparkContext.parallelize(JavaSparkContext.scala:68)

        at spark.api.java.JavaSparkContext.parallelize(JavaSparkContext.scala:73)

        at com.gdv.spark.tagcloud.TagCloudJob.runJob(TagCloudJob.java:52)

        at com.gdv.spark.tagcloud.TagCloudJob.runJob(TagCloudJob.java:34)

        at com.gdv.spark.tagcloud.TagCloudJobExecutor.execute(TagCloudJobExecutor.java:54)

        at com.gdv.spark.tagcloud.TagCloudJobExecutor.execute(TagCloudJobExecutor.java:22)

        at com.gdv.spark.tagcloud.TagCloudJobExecutor$ThreadTester.run(TagCloudJobExecutor.java:131)

        at java.lang.Thread.run(Thread.java:679)

[

Thanks

[liu]

<Moved the here>

Hi,

I tried to use a single SparkContext in multiple threads, and use set SparkEnv in run(), it works when reading input from local files, but not working when I pass in a list of String to method: JavaSparkContext.parallelize(List<String>), is this a bug?

java.lang.NullPointerException

        at scala.collection.JavaConversions$JListWrapper.length(JavaConversions.scala:616)

        at scala.collection.SeqLike$class.size(SeqLike.scala:100)

        at scala.collection.JavaConversions$JListWrapper.size(JavaConversions.scala:615)

        at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:233)

        at scala.collection.JavaConversions$JListWrapper.toArray(JavaConversions.scala:615)

        at spark.ParallelCollection$.slice(ParallelCollection.scala:86)

        at spark.ParallelCollection.<init>(ParallelCollection.scala:35)

        at spark.SparkContext.parallelize(SparkContext.scala:195)

        at spark.api.java.JavaSparkContext.parallelize(JavaSparkContext.scala:68)

        at spark.api.java.JavaSparkContext.parallelize(JavaSparkContext.scala:73)

Any suggestions?

Thanks

[Zhiguang Wen]

Hi, liu

I also met this problem before . Check this : https://groups.google.com/forum/?fromgroups=#!searchin/spark-users/Zhiguang/spark-users/PUgzg6MpTwE/fxAZh2j5dzYJ

Zhiguang

在 2013年2月13日星期三UTC+8上午2时53分05秒，liu写道：