[JIRA] (SPARKR-237) private package functions unavailable when using lapplyPartition in package

53 views
Skip to first unread message

Antonio Piccolboni (JIRA)

unread,
Mar 19, 2015, 4:49:02 PM3/19/15
to spark...@googlegroups.com
Antonio Piccolboni created an issue
 
SparkR / Bug SPARKR-237
private package functions unavailable when using lapplyPartition in package
Issue Type: Bug Bug
Assignee: Unassigned
Created: 19/Mar/15 1:48 PM
Priority: Major Major
Reporter: Antonio Piccolboni

Developing package that imports SparkR. There is a function in that package that calls lapplyPartition with a function argument that uses in its body some functions private to the package. When run, the computation fails because R can not find the private function (details below). If I fully qualify them with otherpackage:::private.function, the error moves down to the next private function. This used to work some time ago, I've been working on other stuff for a little while. This should also work by regular R scope rules. I apologize I don't have a minimal test case ready, but this was discovered developing plyrmr and the list of dependencies is long enough that it's a little bit of a burden to make you install it. I think I can put together a toy package to demonstrate the problem, if that helps.

Error in FUN(part) : could not find function "keys.spark"
Calls: source ... eval -> eval -> computeFunc -> <Anonymous> -> FUN -> FUN
Execution halted
15/03/19 12:29:16 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
org.apache.spark.SparkException: R computation failed with
Error in FUN(part) : could not find function "keys.spark"
Calls: source ... eval -> eval -> computeFunc -> <Anonymous> -> FUN -> FUN
Execution halted
at edu.berkeley.cs.amplab.sparkr.BaseRRDD.compute(RRDD.scala:80)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
at org.apache.spark.scheduler.Task.run(Task.scala:54)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4-OD-15-055#64014-sha1:797c660)
Atlassian logo

Shivaram Venkataraman (JIRA)

unread,
Mar 19, 2015, 5:04:01 PM3/19/15
to spark...@googlegroups.com
Shivaram Venkataraman commented on Bug SPARKR-237
 
Re: private package functions unavailable when using lapplyPartition in package

hao Could you take a look at this if you get a chance ?

Antonio Piccolboni We recently improved our closure serialization to only capture what is required and it looks like that might have affected the way we handle package private functions.

One clarification: In otherpackage::function_name, is the otherpackage already loaded ? If you use the `includePackage` directive it should load the package on workers before executing the closures

Antonio Piccolboni (JIRA)

unread,
Mar 19, 2015, 6:01:04 PM3/19/15
to spark...@googlegroups.com

One counter-clarification. includePackage() acts by side effect and can be executed at any time before a lapply or lapplyPartition and will affect workers for that call? If so, I just checked that it is called with the name of the package to which the private functions belong before the error occurs.

hao (JIRA)

unread,
Mar 20, 2015, 3:38:02 PM3/20/15
to spark...@googlegroups.com
hao commented on Bug SPARKR-237

Antonio Piccolboni Terribly sorry for breaking plyrmr... I think what is happening is that new closure serializer does not serialize functions in package namespaces, since packages are loaded on workers. I forgot that users might use private functions in closure. Let me make some change on that and I will submit a PR. Please try out then and let me if it works.

Shivaram Venkataraman Should I submit PR to old repo or new?

Shivaram Venkataraman (JIRA)

unread,
Mar 20, 2015, 3:47:01 PM3/20/15
to spark...@googlegroups.com

You can send it to the old repo as this fix is required for existing users as well. I can then port it to the new repo too.

hao (JIRA)

unread,
Mar 20, 2015, 3:54:03 PM3/20/15
to spark...@googlegroups.com

Antonio Piccolboni (JIRA)

unread,
Mar 25, 2015, 1:34:02 PM3/25/15
to spark...@googlegroups.com
Antonio Piccolboni commented on Bug SPARKR-237
 
Re: private package functions unavailable when using lapplyPartition in package

I would like to summarize the discussion that went on in a pull request https://github.com/RevolutionAnalytics/plyrmr/pull/72 to unify the subject under this ticket.
What it appears is that this problem depends of replacing getDependencies with cleanClosure and that it is unfixable in the short term. @hao proposed workarounds for dependent packages that include: fully qualifying all function calls; rename some variables according to unstated rules; modifying enclosing environment for promises in a way that is neither backward compatible nor consistent with R semantics (promises are not limited to one enclosing environment equal to GlobalEnv). These are extremely heavy handed, breaking the package in other ways and would be impossible to hide from the end user as the API accepts user defined functions and expressions in several if not most calls. So I am not trying to be difficult, but we can't adopt the workarounds. Considering that SparkR-238 is still open and probably related, my suggestion would be to revert to getDependencies for the moment being and make changes incrementally from there. @hao says getDependencies had correctness issues as well but I would like some detail about it, as plyrmr tests as well SparkR's own passed before cleanClosure was merged.

This message was sent by Atlassian JIRA (v6.4-OD-16-005#64014-sha1:3f74600)
Atlassian logo

Antonio Piccolboni (JIRA)

unread,
Mar 25, 2015, 1:49:01 PM3/25/15
to spark...@googlegroups.com
Antonio Piccolboni edited a comment on Bug SPARKR-237
I would like to summarize the discussion that went on in a pull request https://github.com/RevolutionAnalytics/plyrmr/pull/72 to unify the subject under this ticket.
What it appears is that this problem depends of replacing getDependencies with cleanClosure and that it is unfixable in the short term. @hao proposed workarounds for dependent packages that include: fully qualifying all function calls;  rename  renaming  some variables according to unstated rules; modifying enclosing environment for promises in a way that  is neither backward compatible nor consistent with R semantics (promises are not limited to one enclosing environment equal to GlobalEnv). These are extremely heavy handed, breaking the package in other ways and would be impossible to hide from the end user as the API accepts user defined functions and expressions in several if not most calls. So I am not trying to be difficult, but we can't adopt the workarounds. Considering that SparkR-238 is still open and probably related, my suggestion would be to revert to getDependencies for the moment being and make changes incrementally from there. @hao says getDependencies had correctness issues as well but I would like some detail about it, as plyrmr tests as well SparkR's own passed before cleanClosure was merged.  

hao (JIRA)

unread,
Mar 25, 2015, 2:02:02 PM3/25/15
to spark...@googlegroups.com
hao commented on Bug SPARKR-237

Thanks for the summary. For the old bugs in getDependencies, please refer to 142 and 196. BTW, 238 is resolved in current cleanClosure.

hao (JIRA)

unread,
Mar 25, 2015, 2:04:01 PM3/25/15
to spark...@googlegroups.com
hao edited a comment on Bug SPARKR-237
Thanks for the summary.  I am ok with the revert.  For the old bugs in getDependencies, please refer to 142 and 196. BTW, 238 is resolved in current cleanClosure.

Shivaram Venkataraman (JIRA)

unread,
Mar 25, 2015, 2:06:01 PM3/25/15
to spark...@googlegroups.com

Thanks Antonio Piccolboni for the summary and thanks [~hao] for looking at this. I'll take a closer look at the issues and get back on this thread.

Antonio Piccolboni (JIRA)

unread,
Mar 25, 2015, 2:08:03 PM3/25/15
to spark...@googlegroups.com

Thanks for the pointers, I did not appreciate that those two bugs were fixed by cleanClosure PR. Will check SparkR-238 shortly, that's great.

Antonio Piccolboni (JIRA)

unread,
Mar 25, 2015, 5:36:02 PM3/25/15
to spark...@googlegroups.com
Antonio Piccolboni edited a comment on Bug SPARKR-237
Thanks for the pointers, I did not  appreciate  realize  that those two bugs were fixed by cleanClosure PR. Will check SparkR-238 shortly, that's great. 

Shivaram Venkataraman (JIRA)

unread,
Apr 9, 2015, 9:19:02 PM4/9/15
to spark...@googlegroups.com

Now tracked by https://issues.apache.org/jira/browse/SPARK-6840 - As I mentioned in the other thread I'm marking this as duplicate just to track that the issue has been migrated to the Spark JIRA. We will backport the fix to SparkR-pkg (both branches) once its ready.

This message was sent by Atlassian JIRA (v6.4-OD-16-006#64014-sha1:e47e3fa)
Atlassian logo

Shivaram Venkataraman (JIRA)

unread,
Apr 9, 2015, 9:20:03 PM4/9/15
to spark...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages