Cannot run program "Rscript": java.io.IOException: error=2 in Psuedo Distributed mode Hadoop

[Vivedha Elango]

Hi All,
I am a newbie to RHadoop.when I try to run a basic maptask in the tutorial https://github.com/RevolutionAnalytics/rmr2/blob/master/docs/tutorial.md I get "Cannot run program "Rscript": java.io.IOException: error=2, No such file or directory" error.
I am running
R version is 3.1.1,
Rstudio version 0.98
Hadoop CDH4 Pseudo distributed

I have checked the Rscript path
type -a Rscript in the terminal

[root@01HW288075 Downloads]# type -a Rscript
Rscript is /usr/local/bin/Rscript
Rscript is /usr/local/lib/R/bin/Rscript

But when I execute the same command in Rstudio I get

Sys.getenv("RSCRIPT")
[1] "/usr/local/lib/R/bin"
> Rscript --help
Error: object 'Rscript' not found
> type -a Rscript
Error: unexpected symbol in "type -a Rscript"
> type Rscript
Error: unexpected symbol in "type Rscript"
> type -a --help
Error: object 'type' not found

Not sure Rscript is installed in Rstudio.From the google search on this error, I get the idea that there may be a problem with path, but not sure which path.
Here is the code I tried to run

small.ints = 1:1000
sapply(small.ints, function(x) x^2)
small.ints = to.dfs(1:1000)
mapreduce(input = small.ints,map = function(k, v) cbind(v, v^2))  



the error I am getting is 

14/08/21 19:54:48 INFO mapreduce.Job: Task Id : attempt_1408422373374_0013_m_000000_0, Status : FAILED Error: java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:424) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:160) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:416) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:155) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) ... 9 more Caused by: java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) ... 14 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) ... 17 more Caused by: java.lang.RuntimeException: configuration exception at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:222) at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66) ... 22 more Caused by: java.io.IOException: Cannot run program "Rscript": java.io.IOException: error=2, No such file or directory at java.lang.ProcessBuilder.start(ProcessBuilder.java:475) at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:209) ... 23 more Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory at java.lang.UNIXProcess.<init>(UNIXProcess.java:164) at java.lang.ProcessImpl.start(ProcessImpl.java:81) at java.lang.ProcessBuilder.start(ProcessBuilder.java:468) ... 24 more

Please help me in resolving this error.

[Antonio Piccolboni]

On Thursday, August 21, 2014 8:01:15 AM UTC-7, Vivedha Elango wrote:

Hi All,
I am a newbie to RHadoop.when I try to run a basic maptask in the tutorial https://github.com/RevolutionAnalytics/rmr2/blob/master/docs/tutorial.md I get "Cannot run program "Rscript": java.io.IOException: error=2, No such file or directory" error.
I am running
R version is 3.1.1,
Rstudio version 0.98
Hadoop CDH4 Pseudo distributed

Rstudio is not needed, so to simplify things you can test from a simple R sessions. There are differences in how the two pick up environment variables (RStudio doesn't)

 

I have checked the Rscript path
type -a Rscript in the terminal

[root@01HW288075 Downloads]# type -a Rscript
Rscript is /usr/local/bin/Rscript
Rscript is /usr/local/lib/R/bin/Rscript

The problem is that you are testing in your shell and it's not the same environment in which Rscript is called. So executing type -a in your shell doesn't prove anything. The program that's trying to call RScript doesn't run an interactive shell and executes as a different user. This is a problem that came up before, and what works is to install under /usr/bin. There are other approaches, the backend parameter cmdenv and the configuration script hadoop-env.sh. For unknown reason, they don't seem to work very often.

 

But when I execute the same command in Rstudio I get

Sys.getenv("RSCRIPT")

[1] "/usr/local/lib/R/bin"
> Rscript --help

Error: object 'Rscript' not found
> type -a Rscript
Error: unexpected symbol in "type -a Rscript" 

> type Rscript
Error: unexpected symbol in "type Rscript"
> type -a --help
Error: object 'type' not found

Not sure Rscript is installed in Rstudio.

Apparently, you don't understand the difference between unix shell, R and RStudio. Unless you are absolutely comfortable with those basics, trying to learn RHadoop will be an exercise in frustration. This forum is not the appropriate place to acquire that knowledge.

Antonio