Job aborted due to stage failure: java.net.SocketTimeoutException: Accept timed out

340 views
Skip to first unread message

Rahul Belur

unread,
Mar 25, 2015, 8:25:55 PM3/25/15
to spark...@googlegroups.com
Hello,

I'm using sparkR-submit to run a really simple job that keeps failing on a dataset consisting of 100mil rows in hdfs. The logic is to just read it in using textFile() and do a count() action on it. I also have a really small dataset consisting of 16 rows in hdfs...the exact same R code will run using this dataset. I don't really know where to go from here, hoping someone can point me in the right direction. 

Here's the submit command I'm running:
YARN_CONF_DIR=/etc/hadoop/conf SPARK_HOME=/home/raybeam/spark-1.2.1-bin-hadoop2.4 /usr/local/lib/R/site-library/SparkR/sparkR-submit --master yarn-client --num-executors 16 --executor-memory 2G /home/raybeam/bootstrap_spark.R.

And here's the error message I'm getting:
ERROR scheduler.TaskSetManager: Task 6 in stage 0.0 failed 4 times; aborting job
...
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 6 in stage 0.0 failed 4 times, most recent failure: Lost task 6.3 in stage 0.0 (TID 9, rbhadoop06): java.net.SocketTimeoutException: Accept timed out

Zongheng Yang

unread,
Mar 27, 2015, 1:58:48 PM3/27/15
to Rahul Belur, spark...@googlegroups.com
Hi Rahul,

What's your cluster size (# workers, # total memory)?  Also what's the size of the 100M-row text file? 

--
You received this message because you are subscribed to the Google Groups "SparkR Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sparkr-dev+...@googlegroups.com.
To post to this group, send email to spark...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sparkr-dev/34714582-e666-4fe8-a400-50c13b4048a6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Rahul Belur

unread,
Mar 27, 2015, 2:26:59 PM3/27/15
to spark...@googlegroups.com, rbe...@raybeam.com
17 yarn containers across 11 machines, ~80GB memory total.

File is 858.31 MB

hipho...@gmail.com

unread,
Mar 31, 2015, 4:32:35 AM3/31/15
to spark...@googlegroups.com, rbe...@raybeam.com
Hi Rahul,

Can you try to turn off SparkR daemon worker mode by creating a SparkContext with:
sparkEnvir=list(spark.sparkr.use.daemon=FALSE)
to see if this problem still exists?

在 2015年3月28日星期六 UTC+8上午2:26:59,Rahul Belur写道:
Reply all
Reply to author
Forward
0 new messages