Issue in SPARK UDF Creation

12 views
Skip to first unread message

ARUN KUMAR

unread,
Aug 21, 2018, 10:35:24 AM8/21/18
to sparkjava
Hi All

             I have written a Spark Function Definitions as below

{
  "!name": " xyz-spark-functions",
  "xyz":{
     "!type": "fn(name: Column) -> DataFrame",
    "!doc": "Expose a Table",
    "!spark": "org.xyz.xyz.xyz(%c,%s,sqlContext)",
   "!sparkType": "dataframe"
  }
}

I am facing some issue like below:

not found: value df org in <console> at line number 9

                 Here, I have no Idea on what is df might be the dataframe and the org is my package name of my Spark UDF Jar.

So I edited the Spark Function Definitions as below

{
  "!name": " xyz-spark-functions",
  "xyz":{
     "!type": "fn(name: Column) -> DataFrame",
    "!doc": "Expose a Table",
    "!spark": "new org.xyz.xyz.xyz(%c,%s,sqlContext)",
   "!sparkType": "dataframe"
  }
}

And again I face a issue as below:

An error occurred error: ';' expected but '.' found. in <console> at line number 1

I am struck so please anyone can help on this to solve the issue

Thanks & Regards

Reply all
Reply to author
Forward
0 new messages