無法做MapReduce

174 views
Skip to first unread message

蔡議霆 (Yi-Ting Tsai)

unread,
Nov 22, 2011, 5:26:25 AM11/22/11
to nthu-201...@googlegroups.com
我用助教給的範例一樣不能。出現下列錯誤訊息：
java.lang.RuntimeException: java.lang.ClassNotFoundException: mymapper
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866)
        at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java                                                                             :199)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:719)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma                                                                             tion.java:1059)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassNotFoundException: mymapper
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.jav                                                                             a:819)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:864)
        ... 8 more

11/11/22 10:21:59 INFO mapred.JobClient: Task Id : attempt_201111220853_0024_m_0                                                                             00001_0, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: mymapper
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866)
        at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java                                                                             :199)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:719)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma                                                                             tion.java:1059)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassNotFoundException: mymapper
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.jav                                                                             a:819)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:864)
        ... 8 more

11/11/22 10:22:05 INFO mapred.JobClient: Task Id : attempt_201111220853_0024_m_0                                                                             00000_1, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: mymapper
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866)
        at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java                                                                             :199)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:719)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma                                                                             tion.java:1059)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassNotFoundException: mymapper
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.jav                                                                             a:819)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:864)
        ... 8 more

11/11/22 10:22:05 INFO mapred.JobClient: Task Id : attempt_201111220853_0024_m_0                                                                             00001_1, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: mymapper
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866)
        at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java                                                                             :199)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:719)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma                                                                             tion.java:1059)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassNotFoundException: mymapper
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.jav                                                                             a:819)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:864)
        ... 8 more

11/11/22 10:22:11 INFO mapred.JobClient: Task Id : attempt_201111220853_0024_m_000000_2, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: mymapper
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866)
        at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:199)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:719)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassNotFoundException: mymapper
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:819)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:864)
        ... 8 more

11/11/22 10:22:11 INFO mapred.JobClient: Task Id : attempt_201111220853_0024_m_000001_2, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: mymapper
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866)
        at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:199)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:719)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassNotFoundException: mymapper
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:819)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:864)
        ... 8 more

周宗毅

unread,
Nov 22, 2011, 5:40:56 AM11/22/11
to nthu-201...@googlegroups.com
確認一下程式的參數 (function name)

"java.lang.ClassNotFoundException"



Yafang

unread,
Nov 22, 2011, 10:44:06 AM11/22/11
to nthu-201...@googlegroups.com
根據錯誤訊息看起來像是同學沒有將mymapper也包在同一個jar檔
也請同學確認一下jar在linux下的執行方式
謝謝

助教

jeffery

unread,
Nov 26, 2011, 11:34:16 PM11/26/11
to nthu-201...@googlegroups.com
請問一下，我將範例程式compile過後產生的wordcount.class，mymapper.class與myreducer.class都包進去jar檔，仍然會出現這個錯誤訊息，hadoop依然抓不到我的mymapper檔案，這有什麼方法解決嗎？

Yafang

unread,
Nov 28, 2011, 3:13:49 AM11/28/11
to nthu-201...@googlegroups.com
應該是你打包成jar檔 或是執行的方法有誤
我會發表一篇這部分的訊息
請同學參考一下並對照是否是因為指令錯誤
謝謝

Blue

unread,
Nov 28, 2011, 6:54:39 AM11/28/11
to nthu-201...@googlegroups.com
我也有相同問題 自己電腦建置好環境之後(single-node) 卻無法做map-reduce :(  
執行
bin/hadoop jar hadoop-example-0.20.203.0.jar wordcount input output
這一段程式碼時 系統就停住不動了:(

周宗毅

unread,
Nov 28, 2011, 7:08:04 AM11/28/11
to nthu-201...@googlegroups.com
有任何的錯誤訊息印出來嗎?

Yafang

unread,
Nov 28, 2011, 9:45:42 AM11/28/11
to nthu-201...@googlegroups.com
同學是用VM裝嗎
硬碟大小設多少呢? 
如果儲存空間不夠大的話MapReduce做到一半可能會沒有反應
要過很久才會有VM Storage space 不足之類的訊息出現
可以檢查看看
而且用VM跑的話本來反應就會比較慢 稍微等它一下吧

Blue

unread,
Nov 30, 2011, 4:12:44 AM11/30/11
to nthu-201...@googlegroups.com
我是用WUBI灌UBUNTU的＠＠是雙系統 所以硬碟跟記憶體應該沒有問題
但是我就是沒辦法MAP-REDUCE 一直卡在圖上的情況qq
請問有人有碰到相同的情況嗎?
�U* X� 2011-11-30 17:10:06.png

周宗毅

unread,
Nov 30, 2011, 5:03:12 AM11/30/11
to nthu-201...@googlegroups.com
Hi , 

你先ctrl + C 停止一下該動作

之後輸入jps確認是否所有的關於hadoop的服務都有正確啟動

如"Hadoop_Setup_Guide.pdf" 第六頁中的狀況

推測應該datanode或是namenode沒有開啟 :)

鴻昌 吳

unread,
Dec 6, 2011, 12:23:16 PM12/6/11
to nthu-201...@googlegroups.com
助教我也是有跟原PO一樣的問題發生 (找不到class :mymapper)

我也確定3個編譯過後的class檔都被包進jar檔了

執行方式:
      1. 在local端放好XXXX.jar檔
      2. hadoop jar XXXX.jar wordcount 1.txt output

為什麼他都說找不到阿
跟CLASSPATH設定有關嗎?

周宗毅

unread,
Dec 6, 2011, 12:45:57 PM12/6/11
to nthu-201...@googlegroups.com
Hi ,

這個部分請參考一下這篇討論串

Lab 3 Sample code - wordcount執行

猜測是你的class編譯過程有問題(因為沒看到你的執行過程或是log)

請試著依類似步驟重新在編譯過:)

TA


Reply all
Reply to author
Forward
0 new messages