[JIRA] (SPARKR-217) master specified twice when submitting R job with sparkR-submit

23 views
Skip to first unread message

Felix Cheung (JIRA)

unread,
Mar 4, 2015, 2:15:01 AM3/4/15
to spark...@googlegroups.com
Felix Cheung created an issue
 
SparkR / Improvement SPARKR-217
master specified twice when submitting R job with sparkR-submit
Issue Type: Improvement Improvement
Assignee: Unassigned
Created: 03/Mar/15 11:14 PM
Priority: Minor Minor
Reporter: Felix Cheung

When submitting a job with sparkR-submit

/usr/lib64/R/library/SparkR/sparkR-submit --master yarn-client examples/pi.R yarn-client 4

notice how the master is specified twice, once for spark-submit and once for sparkR.init (which is likely a no-op)

1. Should example or sparkR.init be changed or reordered to eliminate the duplication?
2. IMO, RRDD createSparkContext should detect that even when its master param is != "", master would have already been set (via spark-submit), if so it should ignore the param (I don't think it can change the master anyway at that point)

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4-OD-15-055#64014-sha1:797c660)
Atlassian logo

Felix Cheung (JIRA)

unread,
Mar 4, 2015, 2:16:01 AM3/4/15
to spark...@googlegroups.com

Shivaram Venkataraman (JIRA)

unread,
Mar 4, 2015, 1:41:01 PM3/4/15
to spark...@googlegroups.com

Yeah, the right fix here is to update the example – The problem is that since we have two ways of submitting jobs right now (sparkR, sparkR-submit) and sparkR requires the master to be specified while sparkR-submit picks it up from the conf / command line arguments.

Going forward I think we can deprecate the sparkR script at some point and then clean up all the examples to not require the master url.

Felix Cheung (JIRA)

unread,
Mar 5, 2015, 1:31:01 PM3/5/15
to spark...@googlegroups.com

Davies (JIRA)

unread,
Mar 19, 2015, 1:47:01 PM3/19/15
to spark...@googlegroups.com
Davies commented on Improvement SPARKR-217
 
Re: master specified twice when submitting R job with sparkR-submit

sparkR-submit is replaced with spark-submit, after we merging into Spark.

Shivaram Venkataraman (JIRA)

unread,
Mar 19, 2015, 1:59:01 PM3/19/15
to spark...@googlegroups.com

Davies One reason to keep this open is to track that we should update the examples in the Spark merge repo to remove the `master` argument. (I only updated pi.R at https://github.com/amplab-extras/spark/blob/R/examples/src/main/r/pi.R)

Davies (JIRA)

unread,
Mar 19, 2015, 3:22:01 PM3/19/15
to spark...@googlegroups.com
Davies assigned an issue to Davies
 
Change By: Davies
Assignee: Felix Cheung Davies
Status: Done In Progress
Resolution: Done

Davies (JIRA)

unread,
Mar 19, 2015, 3:22:01 PM3/19/15
to spark...@googlegroups.com

Davies (JIRA)

unread,
Mar 19, 2015, 3:35:02 PM3/19/15
to spark...@googlegroups.com

Felix Cheung (JIRA)

unread,
Mar 19, 2015, 3:45:02 PM3/19/15
to spark...@googlegroups.com

pi.R and linear_solver_mnist.R should check for length(args)

Felix Cheung (JIRA)

unread,
Mar 19, 2015, 3:45:02 PM3/19/15
to spark...@googlegroups.com

I'm making some changes, should they go to amplab-extras/spark now?

Shivaram Venkataraman (JIRA)

unread,
Mar 19, 2015, 4:20:02 PM3/19/15
to spark...@googlegroups.com

Felix Cheung I am not sure if you are subscribed to the mailing list spark...@googlegroups.com but I just sent a message yesterday about how we are posting new fixes amplab-extras/spark (Branch named R) as that will be the branch used for merging with upstream spark.

You can see the message https://groups.google.com/d/msg/sparkr-dev/TEGbqJp_2IY/APGviHUFa7sJ

Felix Cheung (JIRA)

unread,
Mar 19, 2015, 9:23:01 PM3/19/15
to spark...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages