|
calling parquetFile (in the sparkr-sql branch) fails with:
> parquetFile(sqlCtx, path)
parquetFile on 2 failed with java.util.NoSuchElementException: next on empty iterator
java.util.NoSuchElementException: next on empty iterator
at scala.collection.Iterator$$anon$2.next(Iterator.scala:39)
at scala.collection.Iterator$$anon$2.next(Iterator.scala:37)
at scala.collection.IndexedSeqLike$Elements.next(IndexedSeqLike.scala:64)
at scala.collection.IterableLike$class.head(IterableLike.scala:91)
at scala.collection.mutable.ArrayOps$ofRef.scala$collection$IndexedSeqOptimized$$super$head(ArrayOps.scala:108)
at scala.collection.IndexedSeqOptimized$class.head(IndexedSeqOptimized.scala:120)
at scala.collection.mutable.ArrayOps$ofRef.head(ArrayOps.scala:108)
at edu.berkeley.cs.amplab.sparkr.SparkRBackendHandler.handleMethodCall(SparkRBackendHandler.scala:111)
at edu.berkeley.cs.amplab.sparkr.SparkRBackendHandler.channelRead0(SparkRBackendHandler.scala:60)
at edu.berkeley.cs.amplab.sparkr.SparkRBackendHandler.channelRead0(SparkRBackendHandler.scala:22)
This is caused by edu.berkeley.cs.amplab.sparkr.SparkRBackendHandler#matchMethod. It tries to find a method with signature org.apache.spark.sql.SQLContext.parquetFile(java.lang.String), but only org.apache.spark.sql.SQLContext.parquetFile(java.lang.String...) exists (varargs argument).
There are three ways of fixing this:
-
change the parquetFile signature in SQLContext.R
-
change edu.berkeley.cs.amplab.sparkr.SparkRBackendHandler#matchMethod to handle matching of varargs arguments
-
change org.apache.spark.sql.SQLContext.parquetFile signature to a single String
There's something to say for option 3, because jsonFiledoesn't have a varargs signature either. It's a breaking change though.
Option 2 should be done anyway, I guess. I believe this is the most simple solution, but I'm not sure how to go about it. Maybe check both for signatures of type T and type T[]. Any ideas?
For an easy fix, option 1 is most suitable. I'll put in a pull request of code that I've tested and that works for me.
|