[JIRA] (SPARKR-238) Failure in processClosure

25 views
Skip to first unread message

Antonio Piccolboni (JIRA)

unread,
Mar 19, 2015, 5:06:01 PM3/19/15
to spark...@googlegroups.com
Antonio Piccolboni created an issue
 
SparkR / Bug SPARKR-238
Failure in processClosure
Issue Type: Bug Bug
Assignee: Unassigned
Created: 19/Mar/15 2:05 PM
Priority: Major Major
Reporter: Antonio Piccolboni

Sorry another one I can't reproduce in straight SparkR.

This is a typical plyrmr example

as.data.frame(gapply(input(mtcars), identity))
Error in get(nodeChar, envir = func.env, inherits = FALSE) :
argument "..." is missing, with no default

Stack trace below. This may have appeared after the introduction of the new way of serializing closures. Using ... in a function alone doesn't reproduce the error. So I thought I'd file this to hear what you guys think while I try to isolate it better.

> traceback()
28: get(nodeChar, envir = func.env, inherits = FALSE) at utils.R#363
27: processClosure(node[[i]], oldEnv, defVars, checkedFuncs, newEnv) at utils.R#339
26: processClosure(node[[i]], oldEnv, defVars, checkedFuncs, newEnv) at utils.R#339
25: processClosure(node[[i]], oldEnv, defVars, checkedFuncs, newEnv) at utils.R#339
24: processClosure(node[[i]], oldEnv, defVars, checkedFuncs, newEnv) at utils.R#324
23: processClosure(node[[i]], oldEnv, defVars, checkedFuncs, newEnv) at utils.R#312
22: processClosure(func.body, oldEnv, defVars, checkedFuncs, newEnv) at utils.R#417
21: cleanClosure(obj, checkedFuncs) at utils.R#381
20: processClosure(node[[i]], oldEnv, defVars, checkedFuncs, newEnv) at utils.R#339
19: processClosure(node[[i]], oldEnv, defVars, checkedFuncs, newEnv) at utils.R#339
18: processClosure(node[[i]], oldEnv, defVars, checkedFuncs, newEnv) at utils.R#339
17: processClosure(node[[i]], oldEnv, defVars, checkedFuncs, newEnv) at utils.R#312
16: processClosure(node[[i]], oldEnv, defVars, checkedFuncs, newEnv) at utils.R#339
15: processClosure(node[[i]], oldEnv, defVars, checkedFuncs, newEnv) at utils.R#312
14: processClosure(func.body, oldEnv, defVars, checkedFuncs, newEnv) at utils.R#417
13: cleanClosure(obj, checkedFuncs) at utils.R#381
12: processClosure(node[[i]], oldEnv, defVars, checkedFuncs, newEnv) at utils.R#339
11: processClosure(node[[i]], oldEnv, defVars, checkedFuncs, newEnv) at utils.R#312
10: processClosure(func.body, oldEnv, defVars, checkedFuncs, newEnv) at utils.R#417
9: cleanClosure(FUN) at RDD.R#532
8: lapplyPartitionsWithIndex(X, function(s, part)

{ FUN(part) }) at generics.R#76
7: lapplyPartitionsWithIndex(X, function(s, part) { FUN(part) }

) at RDD.R#499
6: SparkR::lapplyPartition(rdd, f) at generics.R#66
5: SparkR::lapplyPartition(rdd, f)
4: as.pipespark(if (is.grouped(.data))

{ if (is.mergeable(.f)) SparkR::lapplyPartition(SparkR::reduceByKey(rdd, f.reduce, 2L), f) else SparkR::lapplyPartition(SparkR::groupByKey(rdd, 2L), f) }

else SparkR::lapplyPartition(rdd, f), grouped = is.grouped(.data)) at pipespark.R#119
3: gapply.pipespark(input(mtcars), identity) at pipe.R#107
2: gapply(input(mtcars), identity)
1: as.data.frame(gapply(input(mtcars), identity))
>

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4-OD-15-055#64014-sha1:797c660)
Atlassian logo

Shivaram Venkataraman (JIRA)

unread,
Mar 19, 2015, 6:58:01 PM3/19/15
to spark...@googlegroups.com
Shivaram Venkataraman commented on Bug SPARKR-238
 
Re: Failure in processClosure

Thanks for the report. hao Would be great if you take a look when you get time.
I am a bit busy for a couple of days and can probably take a look at it over the weekend

Antonio Piccolboni (JIRA)

unread,
Mar 19, 2015, 7:04:02 PM3/19/15
to spark...@googlegroups.com
Antonio Piccolboni commented on Bug SPARKR-238
 
Re: Failure in processClosure

Thanks, again sorry for the lack of repro and I'd like to help with testing any potential fix.

hao (JIRA)

unread,
Mar 20, 2015, 3:54:02 PM3/20/15
to spark...@googlegroups.com
hao assigned an issue to hao
Change By: hao
Assignee: hao

hao (JIRA)

unread,
Mar 20, 2015, 4:09:01 PM3/20/15
to spark...@googlegroups.com
hao commented on Bug SPARKR-238
 
Re: Failure in processClosure

I am not quite sure what is happening here. Can you print out "nodeChar" in this case by instrumenting in cleanClosure? I suspect that it is "..." in this case?

Antonio Piccolboni (JIRA)

unread,
Mar 20, 2015, 4:10:01 PM3/20/15
to spark...@googlegroups.com

Antonio Piccolboni (JIRA)

unread,
Mar 23, 2015, 2:50:02 PM3/23/15
to spark...@googlegroups.com
 
Re: Failure in processClosure

Sorry for the terse message, I wasn't at my desk. I used option(error = recover), then inspected the stack. There's three cleanClosure calls on the stack

{{
7: RDD.R#495: lapplyPartitionsWithIndex(X, function(s, part)

{ FUN(part) })
8: generics.R#76: lapplyPartitionsWithIndex(X, function(s, part) { FUN(part) }

)
9: RDD.R#528: cleanClosure(FUN)
10: utils.R#424: processClosure(func.body, oldEnv, defVars, checkedFuncs, newEnv)
11: utils.R#312: processClosure(node[[i]], oldEnv, defVars, checkedFuncs, newEnv)
12: utils.R#339: processClosure(node[[i]], oldEnv, defVars, checkedFuncs, newEnv)
13: utils.R#380: cleanClosure(obj, checkedFuncs)
14: utils.R#424: processClosure(func.body, oldEnv, defVars, checkedFuncs, newEnv)
15: utils.R#312: processClosure(node[[i]], oldEnv, defVars, checkedFuncs, newEnv)
16: utils.R#339: processClosure(node[[i]], oldEnv, defVars, checkedFuncs, newEnv)
17: utils.R#312: processClosure(node[[i]], oldEnv, defVars, checkedFuncs, newEnv)
18: utils.R#339: processClosure(node[[i]], oldEnv, defVars, checkedFuncs, newEnv)
19: utils.R#339: processClosure(node[[i]], oldEnv, defVars, checkedFuncs, newEnv)
20: utils.R#339: processClosure(node[[i]], oldEnv, defVars, checkedFuncs, newEnv)
21: utils.R#380: cleanClosure(obj, checkedFuncs)
22: utils.R#424: processClosure(func.body, oldEnv, defVars, checkedFuncs, newEnv)
23: utils.R#312: processClosure(node[[i]], oldEnv, defVars, checkedFuncs, newEnv)
24: utils.R#324: processClosure(node[[i]], oldEnv, defVars, checkedFuncs, newEnv)
25: utils.R#339: processClosure(node[[i]], oldEnv, defVars, checkedFuncs, newEnv)
26: utils.R#339: processClosure(node[[i]], oldEnv, defVars, checkedFuncs, newEnv)
27: utils.R#339: processClosure(node[[i]], oldEnv, defVars, checkedFuncs, newEnv)
28: utils.R#363: get(nodeChar, envir = func.env, inherits = FALSE)}}

I assumed you meant nodeChar that's first arg to get which then errors out. That's in processClosure and is set to "...". I don't see the equivalent in cleanClosure, but let me know if you need me to inspect anything.

This message was sent by Atlassian JIRA (v6.4-OD-16-005#64014-sha1:3f74600)
Atlassian logo

Antonio Piccolboni (JIRA)

unread,
Mar 25, 2015, 2:15:02 PM3/25/15
to spark...@googlegroups.com
 
Re: Failure in processClosure

Merged from hlin09/CleanClosureFix at a16d1c3f. I get some error messages of the same type but then the computation ends successfully.
{{
> as.data.frame(gapply(input(mtcars), identity))
<simpleError in get(nodeChar, envir = func.env, inherits = FALSE): argument "..." is missing, with no default>
<simpleError in get(nodeChar, envir = func.env, inherits = FALSE): argument "..." is missing, with no default>
mpg cyl disp hp drat wt qsec vs am gear carb
Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4
Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4
Datsun 710 22.8 4 108.0 93 3.85 2.320 18.61 1 1 4 1
Hornet 4 Drive 21.4 6 258.0 110 3.08 3.215 19.44 1 0 3 1
Hornet Sportabout 18.7 8 360.0 175 3.15 3.440 17.02 0 0 3 2
Valiant 18.1 6 225.0 105 2.76 3.460 20.22 1 0 3 1
Duster 360 14.3 8 360.0 245 3.21 3.570 15.84 0 0 3 4
Merc 240D 24.4 4 146.7 62 3.69 3.190 20.00 1 0 4 2
Merc 230 22.8 4 140.8 95 3.92 3.150 22.90 1 0 4 2}}

Is this to be expected?

hao (JIRA)

unread,
Mar 25, 2015, 7:43:01 PM3/25/15
to spark...@googlegroups.com
hao commented on Bug SPARKR-238
 
Re: Failure in processClosure

Yes, that is normal. I should remove the error message for debugging. Sorry for that. It is at least good to know that it passes at your environment.

Antonio Piccolboni (JIRA)

unread,
Mar 26, 2015, 2:38:03 PM3/26/15
to spark...@googlegroups.com
Antonio Piccolboni commented on Bug SPARKR-238
 
Re: Failure in processClosure

So you are processing "..." by catching an exception as a normal part of processing, am I guessing right? I am old school and I think this should be avoided, but Python made it acceptable, I suppose.

hao (JIRA)

unread,
Mar 26, 2015, 2:45:01 PM3/26/15
to spark...@googlegroups.com
hao commented on Bug SPARKR-238
 
Re: Failure in processClosure

I think it can be avoided. Actually, I was catching a "missing argument" as an exception, and "..." is just one of the cases that I cannot detect from "exists".

Shivaram Venkataraman (JIRA)

unread,
Apr 9, 2015, 7:41:02 PM4/9/15
to spark...@googlegroups.com
Shivaram Venkataraman commented on Bug SPARKR-238
 
Re: Failure in processClosure

Now tracked by https://issues.apache.org/jira/browse/SPARK-6837 – I'm marking this as duplicate in this JIRA just so I can track which issues have been migrated and which haven't. We will continue the discussion in the Spark JIRA and once we have a fix we will port the fix to SparkR-pkg(master, sparkr-sql)

This message was sent by Atlassian JIRA (v6.4-OD-16-006#64014-sha1:e47e3fa)
Atlassian logo

Shivaram Venkataraman (JIRA)

unread,
Apr 9, 2015, 7:41:02 PM4/9/15
to spark...@googlegroups.com
Shivaram Venkataraman resolved as Duplicate
Change By: Shivaram Venkataraman
Resolution: Duplicate
Status: Open Resolved
Reply all
Reply to author
Forward
0 new messages