|
When submitting a job with sparkR-submit
/usr/lib64/R/library/SparkR/sparkR-submit --master yarn-client examples/pi.R yarn-client 4
notice how the master is specified twice, once for spark-submit and once for sparkR.init (which is likely a no-op)
1. Should example or sparkR.init be changed or reordered to eliminate the duplication? 2. IMO, RRDD createSparkContext should detect that even when its master param is != "", master would have already been set (via spark-submit), if so it should ignore the param (I don't think it can change the master anyway at that point)
|