Not able to run scalding job...getting error "java.lang.ClassNotFoundException: cascading.flow.hadoop.HadoopFlowConnector"

[Thiru PS]

Hi,

I'm new to cascading/ scala and more over to Java. I'm from C# background. I understand the class is not found somthing happend with my configuration.

Can some one help

[cloudera@quickstart part1]$ yarn jar build/libs/part1-fat.jar etl.Main --hdfs --input logs/NASA_access_log_Aug95.txt --output output/out124.txt
Exception in thread "main" java.lang.Throwable: If you know what exactly caused this error, please consider contributing to GitHub via following link.
https://github.com/twitter/scalding/wiki/Common-Exceptions-and-possible-reasons#javalangclassnotfoundexception
    at com.twitter.scalding.Tool$.main(Tool.scala:132)
    at com.twitter.scalding.Tool.main(Tool.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.NoClassDefFoundError: cascading/flow/hadoop/HadoopFlowConnector
    at com.twitter.scalding.Mode$.apply(Mode.scala:79)
    at com.twitter.scalding.Tool.parseModeArgs(Tool.scala:61)
    at com.twitter.scalding.Tool.run(Tool.scala:66)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at com.twitter.scalding.Tool$.main(Tool.scala:128)
    ... 7 more
Caused by: java.lang.ClassNotFoundException: cascading.flow.hadoop.HadoopFlowConnector
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    ... 12 more

[Oscar Boykin]

Does the jar build/libs/part1-fat.jar contain the class it is complaining about?

jar tf part1-fat.jar | grep HadoopFlowConnector

should find it.

If not, the question comes: how did you build this jar.

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/b6a61808-4028-41c5-98d9-0e19fc37fa53%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Oscar Boykin :: @posco :: http://twitter.com/posco

[Thiru PS]

Hello all:

Thanks,

It seems the HadoopFlowConnector is not found with in the jar

I build jar as mentioned here

http://docs.cascading.org/tutorials/scalding-data-processing/part1.html#

--
You received this message because you are subscribed to a topic in the Google Groups "cascading-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cascading-user/WOYYQkAdINw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cascading-use...@googlegroups.com.

To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/CANX%3DQ2rMx8h2Hx%2Bfc5r6cAVdyhb3tN7ktQpLKQOPe%2BJKt1-q_Q%40mail.gmail.com.

[Dan Osipov]

Are you using gradle or sbt?

You need to assemble a fat jar that contains all dependencies.

[Thiru PS]

Hello Dan,

I’m using Gradle to build

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/9be40039-dc9b-46f6-ac17-f81966b23938%40googlegroups.com.

[guo weizhan]

Have u fixed this issue? I got the same problem.

在 2015年8月6日星期四 UTC+8下午8:18:26，Thiru PS写道：

[Srikarasu]

Yes I did. It is because of scalding version. Build using lower version 2.7

Sent from my iPhone

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/843009f8-e8ce-48fc-8d35-f1b06cfd3208%40googlegroups.com.

[Guo Weizhan]

Which version you are using exactly?
I see ext.cascadingVersion = '3.0.0' in my build.gradle

Thanks

在 2015年9月17日星期四 UTC+8下午6:50:31，Thiru PS写道：

[Guo Weizhan]

Sorry for misunderstanding, I'm uisng version 2.7 now.

Thank you!
Weizhan Guo

在 2015年9月17日星期四 UTC+8下午7:01:09，Guo Weizhan写道：