java.lang.ClassNotFoundException: cascading.tap.hadoop.MultiInputSplit
170 views
Skip to first unread message
zero
May 10, 2010, 3:16:33 PM5/10/10
to cascading-user
Hi,
I get the following error when I submit the cascading/hadoop job on
multi-node cluster. Interestingly, the same code runs fine in pseudo
on my developer machine. I ensure that there are no class path issue
Classpath:
10/05/11 00:22:57 INFO zookeeper.ZooKeeper: Client
environment:java.class.path=/master/hadoop/hadoop-0.20.2/bin/../conf:/
usr/lib/jvm/java-6-sun/lib/tools.jar:/master/hadoop/hadoop-0.20.2/
bin/..:/master/hadoop/hadoop-0.20.2/bin/../hadoop-0.20.2-core.jar:/
master/hadoop/hadoop-0.20.2/bin/../lib/cascading-1.1.0.jar:/master/
hadoop/hadoop-0.20.2/bin/../lib/commons-cli-1.2.jar:/master/hadoop/
hadoop-0.20.2/bin/../lib/commons-codec-1.3.jar:/master/hadoop/
hadoop-0.20.2/bin/../lib/commons-collections-3.2.1.jar:/master/hadoop/
hadoop-0.20.2/bin/../lib/commons-configuration-1.6.jar:/master/hadoop/
hadoop-0.20.2/bin/../lib/commons-el-1.0.jar:/master/hadoop/
hadoop-0.20.2/bin/../lib/commons-httpclient-3.0.1.jar:/master/hadoop/
hadoop-0.20.2/bin/../lib/commons-lang-2.4.jar:/master/hadoop/
hadoop-0.20.2/bin/../lib/commons-logging-1.0.4.jar:/master/hadoop/
hadoop-0.20.2/bin/../lib/commons-logging-api-1.0.4.jar:/master/hadoop/
hadoop-0.20.2/bin/../lib/commons-net-1.4.1.jar:/master/hadoop/
hadoop-0.20.2/bin/../lib/core-3.1.1.jar:/master/hadoop/hadoop-0.20.2/
bin/../lib/hbase-0.20.4.jar:/master/hadoop/hadoop-0.20.2/bin/../lib/
hbase-0.20.4-test.jar:/master/hadoop/hadoop-0.20.2/bin/../lib/
hsqldb-1.8.0.10.jar:/master/hadoop/hadoop-0.20.2/bin/../lib/jasper-
compiler-5.5.12.jar:/master/hadoop/hadoop-0.20.2/bin/../lib/jasper-
runtime-5.5.12.jar:/master/hadoop/hadoop-0.20.2/bin/../lib/
jets3t-0.6.1.jar:/master/hadoop/hadoop-0.20.2/bin/../lib/
jetty-6.1.14.jar:/master/hadoop/hadoop-0.20.2/bin/../lib/jetty-
util-6.1.14.jar:/master/hadoop/hadoop-0.20.2/bin/../lib/jgrapht-
jdk1.6.jar:/master/hadoop/hadoop-0.20.2/bin/../lib/jsp-2.1.jar:/master/
hadoop/hadoop-0.20.2/bin/../lib/jsp-api-2.1.jar:/master/hadoop/
hadoop-0.20.2/bin/../lib/junit-3.8.1.jar:/master/hadoop/hadoop-0.20.2/
bin/../lib/kfs-0.2.2.jar:/master/hadoop/hadoop-0.20.2/bin/../lib/
log4j-1.2.15.jar:/master/hadoop/hadoop-0.20.2/bin/../lib/mockito-
all-1.8.0.jar:/master/hadoop/hadoop-0.20.2/bin/../lib/oro-2.0.8.jar:/
master/hadoop/hadoop-0.20.2/bin/../lib/servlet-api-2.5-6.1.14.jar:/
master/hadoop/hadoop-0.20.2/bin/../lib/slf4j-api-1.4.3.jar:/master/
hadoop/hadoop-0.20.2/bin/../lib/slf4j-log4j12-1.4.3.jar:/master/hadoop/
hadoop-0.20.2/bin/../lib/xmlenc-0.52.jar:/master/hadoop/hadoop-0.20.2/
bin/../lib/zookeeper-3.2.2.jar:/master/hadoop/hadoop-0.20.2/bin/../lib/
jsp-2.1/jsp-2.1.jar:/master/hadoop/hadoop-0.20.2/bin/../lib/jsp-2.1/
jsp-api-2.1.jar
Error Stacktrace from jobtracker log
-------------------------------------------------
2010-05-11 00:23:25,761 INFO org.apache.hadoop.mapred.TaskInProgress:
Error from attempt_201005102256_0013_m_000008_3: java.io.IOException:
Split class cascading.tap.hadoop.MultiInputSplit not found
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:
326)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.ClassNotFoundException:
cascading.tap.hadoop.MultiInputSplit
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:
301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:
320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:
762)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:
323)
... 2 more
More
----
10/05/11 00:22:58 INFO mapred.FileInputFormat: Total input paths to
process : 1
10/05/11 00:23:33 WARN flow.FlowStep: [id0] completion events count:
10
10/05/11 00:23:33 WARN flow.FlowStep: [id0] event = Task Id :
attempt_201005102256_0013_m_000011_0, Status : SUCCEEDED
10/05/11 00:23:33 WARN flow.FlowStep: [id0] event = Task Id :
attempt_201005102256_0013_m_000000_0, Status : FAILED
10/05/11 00:23:33 WARN flow.FlowStep: [id0] event = Task Id :
attempt_201005102256_0013_m_000001_0, Status : FAILED
10/05/11 00:23:33 WARN flow.FlowStep: [id0] event = Task Id :
attempt_201005102256_0013_m_000002_0, Status : FAILED
10/05/11 00:23:33 WARN flow.FlowStep: [id0] event = Task Id :
attempt_201005102256_0013_m_000003_0, Status : FAILED
10/05/11 00:23:33 WARN flow.FlowStep: [id0] event = Task Id :
attempt_201005102256_0013_m_000004_0, Status : FAILED
10/05/11 00:23:33 WARN flow.FlowStep: [id0] event = Task Id :
attempt_201005102256_0013_m_000005_0, Status : FAILED
10/05/11 00:23:33 WARN flow.FlowStep: [id0] event = Task Id :
attempt_201005102256_0013_m_000006_0, Status : FAILED
10/05/11 00:23:33 WARN flow.FlowStep: [id0] event = Task Id :
attempt_201005102256_0013_m_000007_0, Status : FAILED
10/05/11 00:23:33 WARN flow.FlowStep: [id0] event = Task Id :
attempt_201005102256_0013_m_000008_0, Status : FAILED
10/05/11 00:23:33 WARN flow.Flow: stopping jobs
10/05/11 00:23:33 INFO flow.FlowStep: [id0] stopping: (1/1)
cascading.hbase.HBaseTap@328049a0
10/05/11 00:23:33 WARN flow.Flow: stopped jobs
10/05/11 00:23:33 WARN flow.Flow: shutting down job executor
10/05/11 00:23:33 WARN flow.Flow: shutdown complete
Exception in thread "main"
com.onmobile.m3pu.dwbi.meghnad.dataload.DataLoaderException: step
failed: (1/1) cascading.hbase.HBaseTap@328049a0
at
com.onmobile.m3pu.dwbi.meghnad.dataload.DataLoaderTask.main(DataLoaderTask.java:
95)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ProgramDriver
$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:
139)
at com.onmobile.m3pu.dwbi.meghnad.Driver.main(Driver.java:20)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: cascading.flow.FlowException: step failed: (1/1)
cascading.hbase.HBaseTap@328049a0
at cascading.flow.FlowStepJob.blockOnJob(FlowStepJob.java:173)
at cascading.flow.FlowStepJob.start(FlowStepJob.java:138)
at cascading.flow.FlowStepJob.call(FlowStepJob.java:127)
at cascading.flow.FlowStepJob.call(FlowStepJob.java:39)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Thanks for your interest.
Regards,
zero
--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To post to this group, send email to cascadi...@googlegroups.com.
To unsubscribe from this group, send email to cascading-use...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/cascading-user?hl=en.
I get the following error when I submit the cascading/hadoop job on
multi-node cluster. Interestingly, the same code runs fine in pseudo
on my developer machine. I ensure that there are no class path issue
Classpath:
10/05/11 00:22:57 INFO zookeeper.ZooKeeper: Client
environment:java.class.path=/master/hadoop/hadoop-0.20.2/bin/../conf:/
usr/lib/jvm/java-6-sun/lib/tools.jar:/master/hadoop/hadoop-0.20.2/
bin/..:/master/hadoop/hadoop-0.20.2/bin/../hadoop-0.20.2-core.jar:/
master/hadoop/hadoop-0.20.2/bin/../lib/cascading-1.1.0.jar:/master/
hadoop/hadoop-0.20.2/bin/../lib/commons-cli-1.2.jar:/master/hadoop/
hadoop-0.20.2/bin/../lib/commons-codec-1.3.jar:/master/hadoop/
hadoop-0.20.2/bin/../lib/commons-collections-3.2.1.jar:/master/hadoop/
hadoop-0.20.2/bin/../lib/commons-configuration-1.6.jar:/master/hadoop/
hadoop-0.20.2/bin/../lib/commons-el-1.0.jar:/master/hadoop/
hadoop-0.20.2/bin/../lib/commons-httpclient-3.0.1.jar:/master/hadoop/
hadoop-0.20.2/bin/../lib/commons-lang-2.4.jar:/master/hadoop/
hadoop-0.20.2/bin/../lib/commons-logging-1.0.4.jar:/master/hadoop/
hadoop-0.20.2/bin/../lib/commons-logging-api-1.0.4.jar:/master/hadoop/
hadoop-0.20.2/bin/../lib/commons-net-1.4.1.jar:/master/hadoop/
hadoop-0.20.2/bin/../lib/core-3.1.1.jar:/master/hadoop/hadoop-0.20.2/
bin/../lib/hbase-0.20.4.jar:/master/hadoop/hadoop-0.20.2/bin/../lib/
hbase-0.20.4-test.jar:/master/hadoop/hadoop-0.20.2/bin/../lib/
hsqldb-1.8.0.10.jar:/master/hadoop/hadoop-0.20.2/bin/../lib/jasper-
compiler-5.5.12.jar:/master/hadoop/hadoop-0.20.2/bin/../lib/jasper-
runtime-5.5.12.jar:/master/hadoop/hadoop-0.20.2/bin/../lib/
jets3t-0.6.1.jar:/master/hadoop/hadoop-0.20.2/bin/../lib/
jetty-6.1.14.jar:/master/hadoop/hadoop-0.20.2/bin/../lib/jetty-
util-6.1.14.jar:/master/hadoop/hadoop-0.20.2/bin/../lib/jgrapht-
jdk1.6.jar:/master/hadoop/hadoop-0.20.2/bin/../lib/jsp-2.1.jar:/master/
hadoop/hadoop-0.20.2/bin/../lib/jsp-api-2.1.jar:/master/hadoop/
hadoop-0.20.2/bin/../lib/junit-3.8.1.jar:/master/hadoop/hadoop-0.20.2/
bin/../lib/kfs-0.2.2.jar:/master/hadoop/hadoop-0.20.2/bin/../lib/
log4j-1.2.15.jar:/master/hadoop/hadoop-0.20.2/bin/../lib/mockito-
all-1.8.0.jar:/master/hadoop/hadoop-0.20.2/bin/../lib/oro-2.0.8.jar:/
master/hadoop/hadoop-0.20.2/bin/../lib/servlet-api-2.5-6.1.14.jar:/
master/hadoop/hadoop-0.20.2/bin/../lib/slf4j-api-1.4.3.jar:/master/
hadoop/hadoop-0.20.2/bin/../lib/slf4j-log4j12-1.4.3.jar:/master/hadoop/
hadoop-0.20.2/bin/../lib/xmlenc-0.52.jar:/master/hadoop/hadoop-0.20.2/
bin/../lib/zookeeper-3.2.2.jar:/master/hadoop/hadoop-0.20.2/bin/../lib/
jsp-2.1/jsp-2.1.jar:/master/hadoop/hadoop-0.20.2/bin/../lib/jsp-2.1/
jsp-api-2.1.jar
Error Stacktrace from jobtracker log
-------------------------------------------------
2010-05-11 00:23:25,761 INFO org.apache.hadoop.mapred.TaskInProgress:
Error from attempt_201005102256_0013_m_000008_3: java.io.IOException:
Split class cascading.tap.hadoop.MultiInputSplit not found
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:
326)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.ClassNotFoundException:
cascading.tap.hadoop.MultiInputSplit
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:
301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:
320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:
762)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:
323)
... 2 more
More
----
10/05/11 00:22:58 INFO mapred.FileInputFormat: Total input paths to
process : 1
10/05/11 00:23:33 WARN flow.FlowStep: [id0] completion events count:
10
10/05/11 00:23:33 WARN flow.FlowStep: [id0] event = Task Id :
attempt_201005102256_0013_m_000011_0, Status : SUCCEEDED
10/05/11 00:23:33 WARN flow.FlowStep: [id0] event = Task Id :
attempt_201005102256_0013_m_000000_0, Status : FAILED
10/05/11 00:23:33 WARN flow.FlowStep: [id0] event = Task Id :
attempt_201005102256_0013_m_000001_0, Status : FAILED
10/05/11 00:23:33 WARN flow.FlowStep: [id0] event = Task Id :
attempt_201005102256_0013_m_000002_0, Status : FAILED
10/05/11 00:23:33 WARN flow.FlowStep: [id0] event = Task Id :
attempt_201005102256_0013_m_000003_0, Status : FAILED
10/05/11 00:23:33 WARN flow.FlowStep: [id0] event = Task Id :
attempt_201005102256_0013_m_000004_0, Status : FAILED
10/05/11 00:23:33 WARN flow.FlowStep: [id0] event = Task Id :
attempt_201005102256_0013_m_000005_0, Status : FAILED
10/05/11 00:23:33 WARN flow.FlowStep: [id0] event = Task Id :
attempt_201005102256_0013_m_000006_0, Status : FAILED
10/05/11 00:23:33 WARN flow.FlowStep: [id0] event = Task Id :
attempt_201005102256_0013_m_000007_0, Status : FAILED
10/05/11 00:23:33 WARN flow.FlowStep: [id0] event = Task Id :
attempt_201005102256_0013_m_000008_0, Status : FAILED
10/05/11 00:23:33 WARN flow.Flow: stopping jobs
10/05/11 00:23:33 INFO flow.FlowStep: [id0] stopping: (1/1)
cascading.hbase.HBaseTap@328049a0
10/05/11 00:23:33 WARN flow.Flow: stopped jobs
10/05/11 00:23:33 WARN flow.Flow: shutting down job executor
10/05/11 00:23:33 WARN flow.Flow: shutdown complete
Exception in thread "main"
com.onmobile.m3pu.dwbi.meghnad.dataload.DataLoaderException: step
failed: (1/1) cascading.hbase.HBaseTap@328049a0
at
com.onmobile.m3pu.dwbi.meghnad.dataload.DataLoaderTask.main(DataLoaderTask.java:
95)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ProgramDriver
$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:
139)
at com.onmobile.m3pu.dwbi.meghnad.Driver.main(Driver.java:20)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: cascading.flow.FlowException: step failed: (1/1)
cascading.hbase.HBaseTap@328049a0
at cascading.flow.FlowStepJob.blockOnJob(FlowStepJob.java:173)
at cascading.flow.FlowStepJob.start(FlowStepJob.java:138)
at cascading.flow.FlowStepJob.call(FlowStepJob.java:127)
at cascading.flow.FlowStepJob.call(FlowStepJob.java:39)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Thanks for your interest.
Regards,
zero
--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To post to this group, send email to cascadi...@googlegroups.com.
To unsubscribe from this group, send email to cascading-use...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/cascading-user?hl=en.
Ken Krugler
May 10, 2010, 3:23:04 PM5/10/10
to cascadi...@googlegroups.com
> I get the following error when I submit the cascading/hadoop job on
> multi-node cluster. Interestingly, the same code runs fine in pseudo
> on my developer machine. I ensure that there are no class path issue
>
> Classpath:
>
> 10/05/11 00:22:57 INFO zookeeper.ZooKeeper: Client
> environment:java.class.path=/master/hadoop/hadoop-0.20.2/bin/../conf:/
> usr/lib/jvm/java-6-sun/lib/tools.jar:/master/hadoop/hadoop-0.20.2/
> bin/..:/master/hadoop/hadoop-0.20.2/bin/../hadoop-0.20.2-core.jar:/
> master/hadoop/hadoop-0.20.2/bin/../lib/cascading-1.1.0.jar
So you've installed the cascading-1.1.0.jar file in /master/hadoop/
> multi-node cluster. Interestingly, the same code runs fine in pseudo
> on my developer machine. I ensure that there are no class path issue
>
> Classpath:
>
> 10/05/11 00:22:57 INFO zookeeper.ZooKeeper: Client
> environment:java.class.path=/master/hadoop/hadoop-0.20.2/bin/../conf:/
> usr/lib/jvm/java-6-sun/lib/tools.jar:/master/hadoop/hadoop-0.20.2/
> bin/..:/master/hadoop/hadoop-0.20.2/bin/../hadoop-0.20.2-core.jar:/
> master/hadoop/hadoop-0.20.2/bin/../lib/cascading-1.1.0.jar
hadoop-0.20.2/lib/ on every server in your cluster?
The cascading.tap.hadoop.MultiInputSplit class should be in the
Cascading 1.1 core jar.
-- Ken
Ken Krugler
+1 530-210-6378
http://bixolabs.com
e l a s t i c w e b m i n i n g
zero
May 11, 2010, 4:39:02 AM5/11/10
to cascading-user
Thanks Ken!
Yes you I had placed the jar at hadoop/lib and replicated the same on
all the nodes but for some reason it was failing on some node at all
times to which I had no understanding. However, when I bundled the
cascading jar with my application jar it works just fine.
Thanks again.
> > For more options, visit this group athttp://groups.google.com/group/cascading-user?hl=en
> > .
>
> --------------------------------------------
> Ken Krugler
> +1 530-210-6378http://bixolabs.com
Yes you I had placed the jar at hadoop/lib and replicated the same on
all the nodes but for some reason it was failing on some node at all
times to which I had no understanding. However, when I bundled the
cascading jar with my application jar it works just fine.
Thanks again.
> > .
>
> --------------------------------------------
> Ken Krugler
> +1 530-210-6378http://bixolabs.com
Reply all
Reply to author
Forward
0 new messages