mrjob installation - class PlatformName not found

44 views
Skip to first unread message

Roman

unread,
Aug 1, 2013, 8:55:26 AM8/1/13
to mr...@googlegroups.com
Hi,

I've installed the current version of mrjob (v0.4) and I was testing the first tests.
I've extended mrjob.conf with my hadoop_home and put it in /etc.
The command  locally works good. But running it on hadoop I get an error that PlatformName class was not found.
I'm sure that hadoop path is ok, since I use hadoop in pseudo distributed mode for another tasks and it works.
What could be a problem with this class?

Thanks,

Roman

/etc/training@training-VirtualBox:~/mrjob-master$ python mrjob/examples/mr_word_freq_count.py README.rst -r hadoop > counts
using configs in /etc/mrjob.conf
creating tmp directory /tmp/training/mr_word_freq_count.training.20130731.211347.789825
writing wrapper script to /tmp/training/mr_word_freq_count.training.20130731.211347.789825/setup-wrapper.sh
STDERR: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/util/PlatformName
STDERR: Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.PlatformName
STDERR:     at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
STDERR:     at java.security.AccessController.doPrivileged(Native Method)
STDERR:     at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
STDERR:     at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
STDERR:     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
STDERR:     at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
STDERR: Could not find the main class: org.apache.hadoop.util.PlatformName.  Program will exit.
STDERR: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FsShell
STDERR: Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FsShell
STDERR:     at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
STDERR:     at java.security.AccessController.doPrivileged(Native Method)
STDERR:     at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
STDERR:     at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
STDERR:     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
STDERR:     at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
STDERR: Could not find the main class: org.apache.hadoop.fs.FsShell.  Program will exit.
Traceback (most recent call last):
  File "mrjob/examples/mr_word_freq_count.py", line 37, in <module>
    MRWordFreqCount.run()
  File "/usr/local/lib/python2.7/dist-packages/mrjob-0.4-py2.7.egg/mrjob/job.py", line 500, in run
    mr_job.execute()
  File "/usr/local/lib/python2.7/dist-packages/mrjob-0.4-py2.7.egg/mrjob/job.py", line 518, in execute
    super(MRJob, self).execute()
  File "/usr/local/lib/python2.7/dist-packages/mrjob-0.4-py2.7.egg/mrjob/launch.py", line 146, in execute
    self.run_job()
  File "/usr/local/lib/python2.7/dist-packages/mrjob-0.4-py2.7.egg/mrjob/launch.py", line 207, in run_job
    runner.run()
  File "/usr/local/lib/python2.7/dist-packages/mrjob-0.4-py2.7.egg/mrjob/runner.py", line 458, in run
    self._run()
  File "/usr/local/lib/python2.7/dist-packages/mrjob-0.4-py2.7.egg/mrjob/hadoop.py", line 236, in _run
    self._upload_local_files_to_hdfs()
  File "/usr/local/lib/python2.7/dist-packages/mrjob-0.4-py2.7.egg/mrjob/hadoop.py", line 263, in _upload_local_files_to_hdfs
    self._mkdir_on_hdfs(self._upload_mgr.prefix)
  File "/usr/local/lib/python2.7/dist-packages/mrjob-0.4-py2.7.egg/mrjob/hadoop.py", line 271, in _mkdir_on_hdfs
    self.invoke_hadoop(['fs', '-mkdir', path])
  File "/usr/local/lib/python2.7/dist-packages/mrjob-0.4-py2.7.egg/mrjob/fs/hadoop.py", line 104, in invoke_hadoop
    raise CalledProcessError(proc.returncode, args)
subprocess.CalledProcessError: Command '['/usr/lib/hadoop-0.20-mapreduce/bin/hadoop', 'fs', '-mkdir', 'hdfs:///user/training/tmp/mrjob/mr_word_freq_count.training.20130731.211347.789825/files/']' returned non-zero exit status 1
training@training-VirtualBox:~/mrjob-master$
Reply all
Reply to author
Forward
0 new messages