[JIRA] (SPARKR-142) Function dependencies are captured too late, so that the value of free variables have already changed

16 views
Skip to first unread message

hao (JIRA)

unread,
Nov 26, 2014, 4:12:22 PM11/26/14
to spark...@googlegroups.com
hao created an issue
 
SparkR / New Feature SPARKR-142
Function dependencies are captured too late, so that the value of free variables have already changed
Issue Type: New Feature New Feature
Assignee: Unassigned
Created: 26/Nov/14 1:12 PM
Priority: Major Major
Reporter: hao
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4-OD-09-008#64005-sha1:e0b4c5a)
Atlassian logo

hao (JIRA)

unread,
Nov 26, 2014, 4:21:23 PM11/26/14
to spark...@googlegroups.com
hao updated an issue
Change By: hao
The function dependencies is computed when JRDD is constructed. However, this should be done when PipelinedRDD is created. Otherwise, there is possibility that the values of free variables have already changed.

E.g. 
rdd <- parallelize(...)
free.var <- 1
rdd1 <- lapply(rdd, function(x) { x + free.var } )
free.var <- 2
rdd2 <- lapply(rdd, function(x) { x + free.var} )
collect(rdd1)
collect(rdd2)

The result is wrong since both rdd1 and rdd2 will have free.var == 2, since free.var is updated before rdd1 is evaluated.

Suggested Solution: Maybe we can add a "depsBin" in the PipelinedRDD object, and compute the dependencies once the PipelinedRDD is created.

Shivaram Venkataraman (JIRA)

unread,
Nov 26, 2014, 6:02:23 PM11/26/14
to spark...@googlegroups.com
Shivaram Venkataraman commented on New Feature SPARKR-142
 
Re: Function dependencies are captured too late, so that the value of free variables have already changed

hao This is a good point and this behavior could be unintuitive to users. I think your proposal of having a depsBin per RDD is along the right lines – However I think this change is a bit tricky as in worker.R we'll need to load the right env for each RDD we evaluate (we only load one depsBin globally at the beginning right now).

FWIW I think the behavior is broken in Scala too (I just checked a similar code path to what you posted) and there have been efforts to fix this in https://issues.apache.org/jira/browse/SPARK-729 but they haven't been merged.

Antonio Piccolboni (JIRA)

unread,
Dec 4, 2014, 2:07:01 PM12/4/14
to spark...@googlegroups.com

One approach that I used in the past is to make deep copies of environments to prevent side effects. Didn't make it recursive though, the efficiency implications were bad. So in this examples you have two anon functions sharing the global.env, you can make a copy of it for each of the functions so that no side effects are possible anymore. So you wouldn't need to do anything more complicated in the worker.

This message was sent by Atlassian JIRA (v6.4-OD-11-012#64007-sha1:af469b7)
Atlassian logo

hao (JIRA)

unread,
Dec 18, 2014, 10:49:02 PM12/18/14
to spark...@googlegroups.com
hao edited a comment on New Feature SPARKR-142
[~piccolbo] & [~shivaram] Thanks for the comments. Are we considering  to  get the dependence of every function? By doing so, we can possibly solve the issue of side effect. But we may risk serializing more data associate with the function, since multiple pipelined functions may share some common free variables. I personally think the latter issue is minor, compared to the side effect, and can also be probably solved. 
This message was sent by Atlassian JIRA (v6.4-OD-12-026#64007-sha1:039f45b)
Atlassian logo

hao (JIRA)

unread,
Dec 18, 2014, 10:49:02 PM12/18/14
to spark...@googlegroups.com
hao commented on New Feature SPARKR-142

Antonio Piccolboni & Shivaram Venkataraman Thanks for the comments. Are we considering get the dependence of every function? By doing so, we can possibly solve the issue of side effect. But we may risk serializing more data associate with the function, since multiple pipelined functions may share some common free variables. I personally think the latter issue is minor, compared to the side effect, and can also be probably solved.

Shivaram Venkataraman (JIRA)

unread,
Mar 1, 2015, 2:59:02 PM3/1/15
to spark...@googlegroups.com
Shivaram Venkataraman assigned an issue to hao
 
Change By: Shivaram Venkataraman
Assignee: hao
This message was sent by Atlassian JIRA (v6.4-OD-15-055#64014-sha1:797c660)
Atlassian logo

Shivaram Venkataraman (JIRA)

unread,
Mar 1, 2015, 5:48:01 PM3/1/15
to spark...@googlegroups.com

hao (JIRA)

unread,
Mar 1, 2015, 5:54:01 PM3/1/15
to spark...@googlegroups.com
hao commented on New Feature SPARKR-142

It is fixed by https://github.com/amplab-extras/SparkR-pkg/pull/186, and no need for extra changes. I save the closure with each function and use nested functions in the pipeline as before.

However, if you could remember, an alternative way is using a accumulator to store functions. One of the benefits of that might be that we could support "..." in lapply, by storing the ... argument in each element of the accumulator. That would be a nice feature, but a minor issue.

Shivaram Venkataraman (JIRA)

unread,
Mar 1, 2015, 5:57:02 PM3/1/15
to spark...@googlegroups.com

I just verified manually that PR 186 also fixed this. (If I am not wrong this is because we call cleanClosure just before creating the RRDD in [1]).
Marking this as resolved

[1] https://github.com/amplab-extras/SparkR-pkg/blob/7afa4c9d31fc3a7e9676a75ac51e0983708ccb1a/pkg/R/RDD.R#L610

Reply all
Reply to author
Forward
0 new messages