ClassNotFoundException: com.nicta.scoobi.impl.exec.MscrMapper

[Jeff Zhang]

Hi all,

I use scoobi-0.6.0-cdh3-RC1 in hadoop-CDH3 cluster, but it throw ClassNotFoundException: com.nicta.scoobi.impl.exec.MscrMapper.

I check the jar file, it has the scoobi jar file. So I am not sure why it will throw ClassNotFoundException.

Here's the log details:

java.lang.RuntimeException: java.lang.ClassNotFoundException: com.nicta.scoobi.impl.exec.MscrMapper
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
	at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:212)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:602)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
	at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.lang.ClassNotFoundException: com.nicta.scoobi.impl.exec.MscrMapper
	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:247)
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:943)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:994)
	... 8 more

[Eric Torreborre]

Hi Jeff,

This might be caused by the client version of your code being different from the Scoobi version stored in your libjars directory.

If that's the case, you can use the "scoobi deletelibjars" option to remove and re-upload all the jars before your job execution.

And by the way, there is now an official 0.6.1-cdh3 version available which you should probably use instead of the RC1.

Cheers,

Eric.

[Eric Torreborre]

Jeff,

Can you please also re-run with scoobi verbose.all?

This will give us more logs to determine what's going on.

Thanks,

Eric.

[Jeff Zhang]

Hi Eric,

I have tried to deletelibjars and use scoobi-0.6.1.cdh3. Neither of them works for me.

I add parameters verbose.all, but it looks like no more logs than before, here's command I run , and also attach the logs

hadoop jar base_scala-2.0-SNAPSHOT-job.jar com.ebay.searchscience.searchmetrics.scala.base.scoobi.ScoobiExample -D mapred.job.queue.name= /sys/soj/2012/12/10/00/sessioncontainer /user/jianfezhang/dev/scoobi -- scoobi deletelibjars verbose.all

2013-01-31 14:01:01,664 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
2013-01-31 14:01:01,942 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/hadoop-zjffdu/mapred/local/taskTracker/zjffdu/jobcache/job_201301311338_0005/jars/.job.jar.crc <- /tmp/hadoop-zjffdu/mapred/local/taskTracker/zjffdu/jobcache/job_201301311338_0005/attempt_201301311338_0005_m_000000_0/work/.job.jar.crc
2013-01-31 14:01:01,947 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/hadoop-zjffdu/mapred/local/taskTracker/zjffdu/jobcache/job_201301311338_0005/jars/job.jar <- /tmp/hadoop-zjffdu/mapred/local/taskTracker/zjffdu/jobcache/job_201301311338_0005/attempt_201301311338_0005_m_000000_0/work/job.jar
2013-01-31 14:01:02,028 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId=
2013-01-31 14:01:02,616 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2013-01-31 14:01:02,619 WARN org.apache.hadoop.mapred.Child: Error running child
java.lang.RuntimeException: java.lang.ClassNotFoundException: com.nicta.scoobi.impl.exec.MscrMapper
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996)
	at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:212)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:602)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
	at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.lang.ClassNotFoundException: com.nicta.scoobi.impl.exec.MscrMapper
	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:247)
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:943)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:994)
	... 8 more
2013-01-31 14:01:02,624 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task

在 2013年1月30日星期三UTC-8下午2时57分59秒，Eric Torreborre写道：

[Jeff Zhang]

And I find the scoobi file in hdfs:  /user/zjffdu/libjars/scoobi_2.9.2-0.6.1-cdh3.jar

So not sure why the ClassNotFoundException happens here.

在 2013年1月30日星期三UTC-8下午10时06分55秒，Jeff Zhang写道：

[Eric Torreborre]

I know it's not the best way of dealing with command line options but the scoobi options are dot-separated, it should be "scoobi deletelibjars.verbose.all". 

With this you should get additional log on the client side telling us about the process of setting-up the classpath, managing the libs, and so on.

Thanks for trying again.

E.