Hello,
I'm using sparkR-submit to run a really simple job that keeps failing on a dataset consisting of 100mil rows in hdfs. The logic is to just read it in using textFile() and do a count() action on it. I also have a really small dataset consisting of 16 rows in hdfs...the exact same R code will run using this dataset. I don't really know where to go from here, hoping someone can point me in the right direction.
Here's the submit command I'm running:
YARN_CONF_DIR=/etc/hadoop/conf SPARK_HOME=/home/raybeam/spark-1.2.1-bin-hadoop2.4 /usr/local/lib/R/site-library/SparkR/sparkR-submit --master yarn-client --num-executors 16 --executor-memory 2G /home/raybeam/bootstrap_spark.R.
And here's the error message I'm getting:
ERROR scheduler.TaskSetManager: Task 6 in stage 0.0 failed 4 times; aborting job
...
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 6 in stage 0.0 failed 4 times, most recent failure: Lost task 6.3 in stage 0.0 (TID 9, rbhadoop06): java.net.SocketTimeoutException: Accept timed out