Re: NoClassDefFoundError when using cascading-parquet and cascading-hive

[Andre Kelpe]

Can you share your build file with us (pom, build.gradle) and how you assemble and run the app? Best is to put it all in a gist or something similar: https://gist.github.com/

- André

On Wed, Jul 22, 2015 at 8:19 PM, <matthe...@gmail.com> wrote:

I'm new to cascading-parquet and cascading-hive so I was having some trouble getting a simple file copy dataflow running where the source tap reads data from a parquet table on hive and the sink tap writes data to a parquet table on hive as well. I don't get any compile errors but when I go to run the jar on hadoop, I get the following error:

Exception in thread "main" java.lang.NoClassDefFoundError: cascading/scheme/Scheme

        at java.lang.ClassLoader.defineClass1(Native Method)

        at java.lang.ClassLoader.defineClass(ClassLoader.java:800)

        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

        at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)

        at java.net.URLClassLoader.access$100(URLClassLoader.java:71)

        at java.net.URLClassLoader$1.run(URLClassLoader.java:361)

        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

        at java.security.AccessController.doPrivileged(Native Method)

        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:412)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Class.java:270)

        at org.apache.hadoop.util.RunJar.run(RunJar.java:214)

        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Caused by: java.lang.ClassNotFoundException: cascading.scheme.Scheme

        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

        at java.security.AccessController.doPrivileged(Native Method)

        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

        ... 17 more

I didn't experience this error when I was running a cascading job without cascading-parquet and cascading-hive though. Has anyone experienced a similar problem to this? I'm using the HiveTableDescriptor and HiveTap and then using the ParquetTupleScheme as the scheme in the HiveTap constructor. Thanks.

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/78d13345-1ddb-4dd1-84f8-2abc97461ed4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

André Kelpe
an...@concurrentinc.com
http://concurrentinc.com

[Andre Kelpe]

Yes, that is the information I was asking for. I am still a bit puzzled tbh. Can you also include the log messages before the stacktrace? I'd like to see, when that exception happens.

- André

On Wed, Jul 22, 2015 at 11:07 PM, M <matthe...@gmail.com> wrote:

This is my build.gradle file

apply plugin: 'java'

repositories {

mavenCentral()

maven {

url 'http://conjars.org/repo/'

}

maven{url 'https://oss.sonatype.org/content/repositories/snapshots'}

}

ext.cascadingVersion = '2.6.3'

ext.hadoopVersion = '2.5.0'

ext.hiveVersion = '0.13.1'

dependencies {

l  compile(group: 'cascading', name: 'cascading-core', version: cascadingVersion)

  compile(group: 'cascading', name: 'cascading-hadoop2-mr1', version: cascadingVersion)

  compile(group: 'org.apache.hadoop', name: 'hadoop-mapreduce-client-core', version: hadoopVersion)

  compile(group: 'com.twitter', name: 'parquet-cascading', version: '1.4.1') {

exclude module: 'hadoop-lzo'}

  compile(group: 'cascading', name: 'cascading-hive', version: '1.0.0')

  compile(group: 'org.apache.hive', name: 'hive-jdbc', version: hiveVersion)

  compile(group: 'org.apache.hive', name: 'hive-exec', version: hiveVersion)

}

jar {

  description = "Assembles a Hadoop ready jar file"

  doFirst {

    into( 'lib' ) {

      from configurations.compile

    }

  }

  manifest {

    attributes( "Main-Class": "hiveParquetDemo/Main" )

and sorry I'm still learning all of this and it's terminology, so what do you mean by how I assemble and run my app? I run it on hadoop with the hadoop jar command and I assemble it using the command "gradle clean jar"

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/adc352b3-1eef-4c87-b4ff-3672775fbec8%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.