ClassNotFound (but it's there)

[joe.b...@gmail.com]

What am I missing here.  I can run the jar from the command line perfectly fine but when I run it in Azkaban, it can't find the main class:

bash-4.1$ hadoop jar test-1.0-SNAPSHOT.jar foo.com.Test /tmp/t.txt /tmp/o.txt

It runs to completion and counts the words.

I then create a .zip with the following:

test.job

lib/test-1.0-SNAPSHOT.jar

The .job contains:

type=hadoopJava

job.class=foo.com.Test

classpath=./lib/*,${hadoop.home}/lib/*,${hadoop.home}/*

main.args=/tmp/TestWords.txt /tmp/output.txt

#hadoop.security.manager.class=

azkaban.should.proxy=false

hadoop.home=/usr/lib/hadoop

jobtype.class=azkaban.jobtype.HadoopJavaJob

force.output.overwrite=true

obtain.binary.token=false

The jar contains:

$ jar tf test-1.0-SNAPSHOT.jar 

META-INF/

META-INF/MANIFEST.MF

foo/

foo/com/

foo/com/Test$Reduce.class

foo/com/Test.class

foo/com/Test$Map.class

I upload it to Azkaban and try executing.  It fails as such:

27-05-2014 18:06:04 EDT wordcount INFO - Starting job wordcount at 1401228364362
27-05-2014 18:06:04 EDT wordcount INFO - Building hadoopJava job executor. 
27-05-2014 18:06:04 EDT wordcount INFO - 1 commands to execute.
27-05-2014 18:06:04 EDT wordcount INFO - Command: java  -Xms64M -Xmx256M -cp ./lib/*:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/*:/usr/lib/azkaban-solo-2.5.0/plugins/jobtypes/hadoopJava/azkaban-jobtype-2.5.0.jar:/usr/lib/azkaban-solo-2.5.0/lib/azkaban-2.5.0.jar:/usr/lib/azkaban-solo-2.5.0/plugins/jobtypes/hadoopJava/azkaban-hadoopsecuritymanager-2.5.0.jar azkaban.jobtype.HadoopJavaJobRunnerMain /tmp/TestWords.txt /tmp/output.txt
27-05-2014 18:06:04 EDT wordcount INFO - Environment variables: {JOB_NAME=wordcount, JOB_PROP_FILE=/usr/lib/azkaban-solo-2.5.0/executions/14/wordcount_props_5655190074578691998_tmp, JOB_OUTPUT_PROP_FILE=/usr/lib/azkaban-solo-2.5.0/executions/14/wordcount_output_4981125235510918604_tmp}
27-05-2014 18:06:04 EDT wordcount INFO - Working directory: /usr/lib/azkaban-solo-2.5.0/executions/14
27-05-2014 18:06:05 EDT wordcount INFO - WARN Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
27-05-2014 18:06:05 EDT wordcount INFO - INFO Running job wordcount
27-05-2014 18:06:05 EDT wordcount INFO - INFO Class name foo.com.Test
27-05-2014 18:06:05 EDT wordcount ERROR - Exception in thread "main" java.lang.ClassNotFoundException: foo.com.Test
27-05-2014 18:06:05 EDT wordcount ERROR - 	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
27-05-2014 18:06:05 EDT wordcount ERROR - 	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
27-05-2014 18:06:05 EDT wordcount ERROR - 	at java.security.AccessController.doPrivileged(Native Method)
27-05-2014 18:06:05 EDT wordcount ERROR - 	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
27-05-2014 18:06:05 EDT wordcount ERROR - 	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
27-05-2014 18:06:05 EDT wordcount ERROR - 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
27-05-2014 18:06:05 EDT wordcount ERROR - 	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
27-05-2014 18:06:05 EDT wordcount ERROR - 	at azkaban.jobtype.HadoopJavaJobRunnerMain.getObject(HadoopJavaJobRunnerMain.java:299)
27-05-2014 18:06:05 EDT wordcount ERROR - 	at azkaban.jobtype.HadoopJavaJobRunnerMain.(HadoopJavaJobRunnerMain.java:146)
27-05-2014 18:06:05 EDT wordcount ERROR - 	at azkaban.jobtype.HadoopJavaJobRunnerMain.main(HadoopJavaJobRunnerMain.java:76)
27-05-2014 18:06:05 EDT wordcount INFO - Process completed unsuccessfully in 0 seconds.
27-05-2014 18:06:05 EDT wordcount ERROR - Job run failed!
27-05-2014 18:06:05 EDT wordcount ERROR - java.lang.RuntimeException: azkaban.jobExecutor.utils.process.ProcessFailureExceptionjava.lang.RuntimeException: azkaban.jobExecutor.utils.process.ProcessFailureException
27-05-2014 18:06:05 EDT wordcount INFO - Finishing job wordcount at 1401228365172 with status FAILED

The Java file in case anyone cares:

package foo.com;

import java.io.IOException;

import java.util.*;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.*;

import org.apache.hadoop.mapred.*;

public class Test {

    public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {

        private final static IntWritable one = new IntWritable(1);

        private Text word = new Text();

        public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {

            String line = value.toString();

            StringTokenizer tokenizer = new StringTokenizer(line);

            while (tokenizer.hasMoreTokens()) {

                word.set(tokenizer.nextToken());

                output.collect(word, one);

            }

        }

    }

    public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {

        public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {

            int sum = 0;

            while (values.hasNext()) {

                sum += values.next().get();

            }

            output.collect(key, new IntWritable(sum));

        }

    }

    public static void main(String[] args) throws Exception {

        JobConf conf = new JobConf(Test.class);

        conf.setJobName("AZ foo.com.Test");

        conf.setOutputKeyClass(Text.class);

        conf.setOutputValueClass(IntWritable.class);

        conf.setMapperClass(Map.class);

        conf.setCombinerClass(Reduce.class);

        conf.setReducerClass(Reduce.class);

        conf.setInputFormat(TextInputFormat.class);

        conf.setOutputFormat(TextOutputFormat.class);

        FileInputFormat.setInputPaths(conf, new Path(args[0]));

        FileOutputFormat.setOutputPath(conf, new Path(args[1]));

        JobClient.runJob(conf);

    }

[Anthony Hsu]

Seems like it should work.  Did you double-check and verify the jar file exists in /usr/lib/azkaban-solo-2.5.0/executions/14/lib?

You can also always try manually executing the log line that says "Command: java ...".  The working directory is printed two lines underneath it.

---

From: azkab...@googlegroups.com [azkab...@googlegroups.com] on behalf of joe.b...@gmail.com [joe.b...@gmail.com]
Sent: Tuesday, May 27, 2014 3:13 PM
To: azkab...@googlegroups.com
Subject: [azkaban] ClassNotFound (but it's there)

--
You received this message because you are subscribed to the Google Groups "azkaban" group.
To unsubscribe from this group and stop receiving emails from it, send an email to azkaban-dev...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[joe.b...@gmail.com]

Thanks!  I'm an idiot.  When I was zipping up the file, I was doing a 

zip test.zip *

instead of doing 

zip -r test.zip *

It was including the lib dir, but wasn't including the jar that was in it.  Silly zip, why doesn't it do what I want it to do instead of what I told it to do?!

[renie...@gmail.com]

Hi,

I'm facing the same issue for cascading package. Actually I have the package and classes inside my jar. But its throwing the following error:

2016-06-17 04:16:44,586 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1466137081193_0012_m_000000_3: Error: java.io.IOException: Split class cascading.tap.hadoop.io.MultiInputSplit not found

Why its can't read a specific class?

Thanks,
Renien