Fwd: Issues with the hoodiespark

[rakesh sharma]

Hi Hoodie Group,

I am trying to get a basic spark upset done based on the hoodie spark API. After providing all the required dependencies I am facing an issue with the fastxml module, which I am not able to resolve.

Can you please let me why am I getting this error and how to resolve it. Error provided below.

Code has been attached in the mail as well.

Environment : CDH 5.8.

Error:

-------

"18/03/22 06:26:07 ERROR ApplicationMaster: User class threw exception: java.lang.NoClassDefFoundError: com/uber/hoodie/hadoop/HoodieROTablePathFilter

java.lang.NoClassDefFoundError: com/uber/hoodie/hadoop/HoodieROTablePathFilter
	at com.uber.hoodie.DefaultSource.createRelation(DefaultSource.scala:92)
	at com.uber.hoodie.DefaultSource.createRelation(DefaultSource.scala:57)
	at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:306)
	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)
	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:156)
	at com.cybs.cdp.App$.main(App.scala:53)
	at com.cybs.cdp.App.main(App.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:645)
Caused by: java.lang.ClassNotFoundException: com.uber.hoodie.hadoop.HoodieROTablePathFilter
	at java.lang.ClassLoader.findClass(ClassLoader.java:530)
	at org.apache.spark.util.ParentClassLoader.findClass(ParentClassLoader.scala:26)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at org.apache.spark.util.ParentClassLoader.loadClass(ParentClassLoader.scala:34)
	at org.apache.spark.util.ChildFirstURLClassLoader.loadClass(MutableURLClassLoader.scala:55)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357) 

 ... 12 more"

--

Best Wishes,
Rakesh.

--

Best Wishes,
Rakesh.

[rakesh sharma]

Hi,

Pardon me the error I copied earlier is a dependency error, which I had fixed. The error I was facing is Fastxml file.

Error:

com.uber.hoodie.exception.HoodieInsertException: Failed to insert for commit time 20180322100900
	at com.uber.hoodie.HoodieWriteClient.insert(HoodieWriteClient.java:212)
	at com.uber.hoodie.DataSourceUtils.doWriteOperation(DataSourceUtils.java:139)
	at com.uber.hoodie.DefaultSource.createRelation(DefaultSource.scala:205)
	at org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:472)
	at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:48)
	at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
	at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
	at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:138)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135)
	at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116)
	at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92)
	at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92)
	at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:610)
	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:233)
	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:217)
	at com.cybs.cdp.App$.main(App.scala:64)
	at com.cybs.cdp.App.main(App.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:645)
Caused by: java.lang.NoSuchMethodError: org.codehaus.jackson.map.ObjectMapper.setVisibility(Lorg/codehaus/jackson/annotate/JsonMethod;Lorg/codehaus/jackson/annotate/JsonAutoDetect$Visibility;)Lorg/codehaus/jackson/map/ObjectMapper;
	at com.uber.hoodie.common.model.HoodieCommitMetadata.toJsonString(HoodieCommitMetadata.java:120)
	at com.uber.hoodie.HoodieWriteClient.saveWorkloadProfileMetadataToInflight(HoodieWriteClient.java:404)
	at com.uber.hoodie.HoodieWriteClient.upsertRecordsInternal(HoodieWriteClient.java:423)
	at com.uber.hoodie.HoodieWriteClient.insert(HoodieWriteClient.java:207)
	... 25 more

The environment we are running is CDH 5.8.12

Regards,

Rakesh

[mail.vinoth.chandar]

Seems like a mismatch of jars.. Some other version of jackson parser is being pulled in.. 

Do you want to take it further on the slack channel..

https://uber.github.io/hudi/community.html 

[rakesh sharma]

Thank you Vinoth,

Somehow I was able to add all the dependent jars in the spark class path and executed them. I was earlier trying to build a fat jar and even though the dependent Jars were present in fat jar, it still will error out with ClassNotFoundException.

thank you for responding at your earliest.

Regards,

Rakesh

On Thursday, March 22, 2018 at 3:22:35 PM UTC+5:30, rakesh sharma wrote:

[Vinoth Chandar]

I really suggest joining our slack channel, so we can discuss this better.. 

--
You received this message because you are subscribed to the Google Groups "hoodie-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hoodie-user...@googlegroups.com.
To post to this group, send email to hoodi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hoodie-user/a93afcaa-5fe1-4dfa-bdad-3c1f730b0e2a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[rakesh sharma]

Yes, I agree.

--

Best Wishes,
Rakesh.