Mac OS 10.6.8 ,rhadoop : “small.ints = to.dfs(1:10)” causing error

[king horse]

Hi All,

I am trying to run basic mapreduce example in R using rmr2 and rhdfs libraries, but I am getting an error. Single node hadoop cluster is installed properly and all five demeans are starting properly.

Following is the R code:

==============================================

Sys.setenv(HADOOP_CMD="/usr/local/hadoop/bin/hadoop")
Sys.setenv(HADOOP_HOME="/usr/local/hadoop")
Sys.setenv(HADOOP_STREAMING="/usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.2.0.jar")

Sys.setenv(HADOOP_CONF_DIR="/usr/local/hadoop/etc/hadoop")
Sys.setenv(JAVA_HOME="/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home")

library('rhdfs')

library('rmr2')

hdfs.init()

small.ints = to.dfs(1:10) // This line causing following error

====================================================

Error output:

mkdir: `/var/folders/Jy/Jy7jXOZtG-yUvVOni3n+ZU+++TI/-Tmp-//RtmpUnfVRV': No such file or directory
Error: retval == 0 is not TRUE

================================================

hdfs.init() is happening properly but to.dfs(1:10) is giving an error

It would be great if experts guide me in this matter. Thanks in advance.

[Antonio Piccolboni]

It looks like you don't have permissions to write to some directory in that path on HDFS. We are changing the default from the next version as reports like yours were starting to accumulate. Of course a new default is only an educated guess. The  best approach is to set the tmp directory with rmr.options(dfs.tempdir = "some writable path here"). It has to work for both backends, that is local file system and hdfs. That is far from ideal, so we are changing this from the next version, there's a setting from an hdfs tmpdir and for the local one it relies on R's own tmpdir.

Antonio