Roman
unread,Aug 1, 2013, 8:55:26 AM8/1/13Sign in to reply to author
Sign in to forward
You do not have permission to delete messages in this group
Sign in to report message
Either email addresses are anonymous for this group or you need the view member email addresses permission to view the original message
to mr...@googlegroups.com
Hi,
I've installed the current version of mrjob (v0.4) and I was testing the first tests.
I've extended mrjob.conf with my hadoop_home and put it in /etc.
The command locally works good. But running it on hadoop I get an error that PlatformName class was not found.
I'm sure that hadoop path is ok, since I use hadoop in pseudo distributed mode for another tasks and it works.
What could be a problem with this class?
Thanks,
Roman
/etc/training@training-VirtualBox:~/mrjob-master$ python mrjob/examples/mr_word_freq_count.py README.rst -r hadoop > counts
using configs in /etc/mrjob.conf
creating tmp directory /tmp/training/mr_word_freq_count.training.20130731.211347.789825
writing wrapper script to /tmp/training/mr_word_freq_count.training.20130731.211347.789825/setup-wrapper.sh
STDERR: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/util/PlatformName
STDERR: Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.PlatformName
STDERR: at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
STDERR: at java.security.AccessController.doPrivileged(Native Method)
STDERR: at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
STDERR: at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
STDERR: at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
STDERR: at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
STDERR: Could not find the main class: org.apache.hadoop.util.PlatformName. Program will exit.
STDERR: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FsShell
STDERR: Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FsShell
STDERR: at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
STDERR: at java.security.AccessController.doPrivileged(Native Method)
STDERR: at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
STDERR: at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
STDERR: at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
STDERR: at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
STDERR: Could not find the main class: org.apache.hadoop.fs.FsShell. Program will exit.
Traceback (most recent call last):
File "mrjob/examples/mr_word_freq_count.py", line 37, in <module>
MRWordFreqCount.run()
File "/usr/local/lib/python2.7/dist-packages/mrjob-0.4-py2.7.egg/mrjob/job.py", line 500, in run
mr_job.execute()
File "/usr/local/lib/python2.7/dist-packages/mrjob-0.4-py2.7.egg/mrjob/job.py", line 518, in execute
super(MRJob, self).execute()
File "/usr/local/lib/python2.7/dist-packages/mrjob-0.4-py2.7.egg/mrjob/launch.py", line 146, in execute
self.run_job()
File "/usr/local/lib/python2.7/dist-packages/mrjob-0.4-py2.7.egg/mrjob/launch.py", line 207, in run_job
runner.run()
File "/usr/local/lib/python2.7/dist-packages/mrjob-0.4-py2.7.egg/mrjob/runner.py", line 458, in run
self._run()
File "/usr/local/lib/python2.7/dist-packages/mrjob-0.4-py2.7.egg/mrjob/hadoop.py", line 236, in _run
self._upload_local_files_to_hdfs()
File "/usr/local/lib/python2.7/dist-packages/mrjob-0.4-py2.7.egg/mrjob/hadoop.py", line 263, in _upload_local_files_to_hdfs
self._mkdir_on_hdfs(self._upload_mgr.prefix)
File "/usr/local/lib/python2.7/dist-packages/mrjob-0.4-py2.7.egg/mrjob/hadoop.py", line 271, in _mkdir_on_hdfs
self.invoke_hadoop(['fs', '-mkdir', path])
File "/usr/local/lib/python2.7/dist-packages/mrjob-0.4-py2.7.egg/mrjob/fs/hadoop.py", line 104, in invoke_hadoop
raise CalledProcessError(proc.returncode, args)
subprocess.CalledProcessError: Command '['/usr/lib/hadoop-0.20-mapreduce/bin/hadoop', 'fs', '-mkdir', 'hdfs:///user/training/tmp/mrjob/mr_word_freq_count.training.20130731.211347.789825/files/']' returned non-zero exit status 1
training@training-VirtualBox:~/mrjob-master$