[JIRA] (SPARKR-208) SQLContext.parquetFile wrong arguments

16 views
Skip to first unread message

Evert Lammerts (JIRA)

unread,
Feb 25, 2015, 8:48:03 AM2/25/15
to spark...@googlegroups.com
Evert Lammerts created an issue
 
SparkR / Bug SPARKR-208
SQLContext.parquetFile wrong arguments
Issue Type: Bug Bug
Assignee: Unassigned
Components: DataFrame
Created: 25/Feb/15 5:47 AM
Environment:

SparkR sparkr-sql branch + HDP 2.2 (Hadoop / YARN 2.6.0) + Spark 1.3.0 master + R 3.1.2 on RHEL Server 6.6

Priority: Major Major
Reporter: Evert Lammerts

calling parquetFile (in the sparkr-sql branch) fails with:

> parquetFile(sqlCtx, path)
parquetFile on 2 failed with java.util.NoSuchElementException: next on empty iterator
java.util.NoSuchElementException: next on empty iterator
        at scala.collection.Iterator$$anon$2.next(Iterator.scala:39)
        at scala.collection.Iterator$$anon$2.next(Iterator.scala:37)
        at scala.collection.IndexedSeqLike$Elements.next(IndexedSeqLike.scala:64)
        at scala.collection.IterableLike$class.head(IterableLike.scala:91)
        at scala.collection.mutable.ArrayOps$ofRef.scala$collection$IndexedSeqOptimized$$super$head(ArrayOps.scala:108)
        at scala.collection.IndexedSeqOptimized$class.head(IndexedSeqOptimized.scala:120)
        at scala.collection.mutable.ArrayOps$ofRef.head(ArrayOps.scala:108)
        at edu.berkeley.cs.amplab.sparkr.SparkRBackendHandler.handleMethodCall(SparkRBackendHandler.scala:111)
        at edu.berkeley.cs.amplab.sparkr.SparkRBackendHandler.channelRead0(SparkRBackendHandler.scala:60)
        at edu.berkeley.cs.amplab.sparkr.SparkRBackendHandler.channelRead0(SparkRBackendHandler.scala:22)

This is caused by edu.berkeley.cs.amplab.sparkr.SparkRBackendHandler#matchMethod. It tries to find a method with signature org.apache.spark.sql.SQLContext.parquetFile(java.lang.String), but only org.apache.spark.sql.SQLContext.parquetFile(java.lang.String...) exists (varargs argument).

There are three ways of fixing this:

  1. change the parquetFile signature in SQLContext.R
  2. change edu.berkeley.cs.amplab.sparkr.SparkRBackendHandler#matchMethod to handle matching of varargs arguments
  3. change org.apache.spark.sql.SQLContext.parquetFile signature to a single String

There's something to say for option 3, because jsonFiledoesn't have a varargs signature either. It's a breaking change though.

Option 2 should be done anyway, I guess. I believe this is the most simple solution, but I'm not sure how to go about it. Maybe check both for signatures of type T and type T[]. Any ideas?

For an easy fix, option 1 is most suitable. I'll put in a pull request of code that I've tested and that works for me.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4-OD-15-055#64014-sha1:797c660)
Atlassian logo

Evert Lammerts (JIRA)

unread,
Feb 25, 2015, 8:50:01 AM2/25/15
to spark...@googlegroups.com
Evert Lammerts commented on Bug SPARKR-208
 
Re: SQLContext.parquetFile wrong arguments

This:

        at edu.berkeley.cs.amplab.sparkr.SparkRBackendHandler.handleMethodCall(SparkRBackendHandler.scala:111)

is line 105 in current head. I was debugging.

Evert Lammerts (JIRA)

unread,
Feb 25, 2015, 8:53:02 AM2/25/15
to spark...@googlegroups.com

Shivaram Venkataraman (JIRA)

unread,
Mar 13, 2015, 7:06:03 PM3/13/15
to spark...@googlegroups.com

Shivaram Venkataraman (JIRA)

unread,
Mar 13, 2015, 7:06:03 PM3/13/15
to spark...@googlegroups.com

Shivaram Venkataraman (JIRA)

unread,
Mar 13, 2015, 7:07:01 PM3/13/15
to spark...@googlegroups.com
Shivaram Venkataraman resolved as Fixed
Change By: Shivaram Venkataraman
Resolution: Fixed
Status: Open Resolved
Reply all
Reply to author
Forward
0 new messages