Problems with the installation of RHadoop (java.io.IOException: Error opening job jar: loadtb)

[Evangelos Ninnis]

Hello,

I'm trying to install RHadoop on a test hadoop cluster with two nodes. 

I tried to run a simple mapreduce based on the tutorial in the wiki on GitHub:

Sys.setenv(HADOOP_HOME="/opt/ibm/biginsights")
Sys.setenv(HADOOP_CMD="/opt/ibm/biginsights/IHC/bin/hadoop")
Sys.setenv(JAVA_HOME="/opt/ibm/biginsights/jdk/jre")
Sys.setenv(HADOOP_HOME_WARN_SUPPRESS="TRUE")


small.ints = to.dfs(1:1000)
mapreduce(
  input = small.ints,
  map = function(k, v) cbind(v, v^2))

Unfortunately I get an Exception:

> Sys.setenv(HADOOP_HOME="/opt/ibm/biginsights")
> Sys.setenv(HADOOP_CMD="/opt/ibm/biginsights/IHC/bin/hadoop")
> Sys.setenv(JAVA_HOME="/opt/ibm/biginsights/jdk/jre")
> Sys.setenv(HADOOP_HOME_WARN_SUPPRESS="TRUE")
> 
> small.ints = to.dfs(1:1000)
Exception in thread "main" java.io.IOException: Error opening job jar: loadtb
        at org.apache.hadoop.util.RunJar.main(RunJar.java:90)
Caused by: java.util.zip.ZipException: error in opening zip file
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.<init>(ZipFile.java:150)
        at java.util.jar.JarFile.<init>(JarFile.java:149)
        at java.util.jar.JarFile.<init>(JarFile.java:86)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:88)
Warning message:
In to.dfs(1:1000) : Converting to.dfs argument to keyval with a NULL key
> mapreduce(
+   input = small.ints, 
+   map = function(k, v) cbind(v, v^2))
Exception in thread "main" java.io.IOException: Error opening job jar: -D
        at org.apache.hadoop.util.RunJar.main(RunJar.java:90)
Caused by: java.util.zip.ZipException: error in opening zip file
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.<init>(ZipFile.java:150)
        at java.util.jar.JarFile.<init>(JarFile.java:149)
        at java.util.jar.JarFile.<init>(JarFile.java:86)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:88)
Error in mr(map = map, reduce = reduce, combine = combine, vectorized.reduce,  : 
  hadoop streaming failed with error code 1

I'm using Hadoop 1.0.3 and R 3.0.1.

What could be the reason for this Exception?

Best regards

Evangelos

[Antonio Piccolboni]

Try to set the shell variable HADOOP_STREAMING to the streaming jar. If you don't specify it, rmr2 looks for $HADOOP_HOME/contrib/streaming/hadoop-streaming*.jar That may or may not be correct for all distributions and is considered the legacy way to configure rmr2. Setting the pair HADOOP_CMD and HADOOP_STREAMING has generalized better to a variety of distros. The error message could be more intuitive,  I agree. Please create an issue on the rmr2 github issue tracker and I will look into that.

Antonio

Evangelos

--
post: rha...@googlegroups.com ||
unsubscribe: rhadoop+u...@googlegroups.com ||
web: https://groups.google.com/d/forum/rhadoop?hl=en-US
---
You received this message because you are subscribed to the Google Groups "RHadoop" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rhadoop+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Evangelos Ninnis]

I found the same solution with setting the shell variable HADOOP_STREAMING just 5 minutes after posting.

The solution could look like this for anybody with a similar problem:

Sys.setenv(HADOOP_STREAMING="/opt/ibm/biginsights/IHC/contrib/streaming/hadoop-streaming-1.0.3.jar")

Thank you for your help.

Evangelos