Using CrossData as Library: java.lang.NoSuchMethodError

[rvalive]

Hi,

I am trying to use CrossData as a library in Spark. Spark version is 1.6.0 and CrossData is1.2.2. I also tried 1.2.1 jar I downloaded from the link below. I am running into this problem below and appreciate your help!

Environment is RedHat 6. If you need any other details please let me know.

Thanks

I followed the instructions here:

https://stratio.atlassian.net/wiki/display/CROSSDATA1x3/Get+Started#GetStarted-AddCrossdatatoanexistingSparkdistribution.1

After starting Spark shell I can do the following without any errors:

import org.apache.spark.sql.crossdata._

import com.stratio.crossdata.connector._




But when I try to execute "val xdContext = new XDContext(sqlContext.sparkContext)" I get this error:

----

java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.analysis.FunctionRegistry$.builtin()Lorg/apache/spark/sql/catalyst/analysis/FunctionRegistry;
  at org.apache.spark.sql.crossdata.XDContext.functionRegistry$lzycompute(XDContext.scala:166)
  at org.apache.spark.sql.crossdata.XDContext.functionRegistry(XDContext.scala:165)
  at org.apache.spark.sql.UDFRegistration.<init>(UDFRegistration.scala:40)
  at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:330)
  at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:77)
  at org.apache.spark.sql.crossdata.XDContext.<init>(XDContext.scala:49)
  at org.apache.spark.sql.crossdata.XDContext.<init>(XDContext.scala:54)
  ... 52 elidedjava.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.analysis.FunctionRegistry$.builtin()Lorg/apache/spark/sql/catalyst/analysis/FunctionRegistry;
  at org.apache.spark.sql.crossdata.XDContext.functionRegistry$lzycompute(XDContext.scala:166)
  at org.apache.spark.sql.crossdata.XDContext.functionRegistry(XDContext.scala:165)
  at org.apache.spark.sql.UDFRegistration.<init>(UDFRegistration.scala:40)
  at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:330)
  at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:77)
  at org.apache.spark.sql.crossdata.XDContext.<init>(XDContext.scala:49)
  at org.apache.spark.sql.crossdata.XDContext.<init>(XDContext.scala:54)
  ... 52 elided
---


[Miguel Angel Fernandez]

Hi rvalive,

Crossdata 1.2.x versions are compatible with Spark version 1.5.X as mentioned here.

Crossdata 1.3.0 will be released this week and it will be compatible with Spark version 1.6.X.

Regards,

- Miguel