Debugging issues in running examples

[David]

Greetings,

First, many thanks for the RHIPE package and for the help contributed
on this forum. I've been banging my head against the wall to get
through a series of errors, and now I'm stuck on some examples from
the guide and from this forum.

In particular, I keep hitting the same errors from Hadoop. The logs
from the Hadoop jobtracker are below (for experiment 2). I've tried a
bunch of different "experiments", to test all the bases that I could
think of.

In experiment 1, the commands were:
rhwrite(list(1,2,3),"/tmp/test")
rhread("/tmp/test")

This works as expected, with the following excerpt:

Wrote 3 pairs occupying 57 bytes
[1] TRUE
> rhread("/tmp/test")
RHIPE: Read 3 pairs occupying 57 bytes, deserializing
[[1]]
[[1]][[1]]
[1] "1"
...


In experiment 2, the commands were:

In bash:
export HADOOP=/usr/lib/hadoop-0.20
export HADOOP_BIN=/usr/lib/hadoop-0.20/bin

R
rhoptions(runner = c("/general/installed/bin/R","CMD", rhoptions()
$runner))
library(Rhipe)
rhinit(TRUE,TRUE,buglevel=2000)
rhoptions(runner = c("/general/installed/bin/R","CMD", rhoptions()
$runner))
z=rhmr(map=expression(), N=3, inout=c("lapp","seq"),ifolder="/tmp/
x", mapred=list(mapred.reduce.tasks=0, mapred.reduce.max.attempts=1,
mapred.map.max.attempts=1))
rhex(z)


Result: the error logs are below. Within R, the following error
was produced:
11/03/14 15:05:33 INFO mapred.JobClient: Failed map
tasks=1
result:256
Error in readBin(f1, "raw", f2) : invalid 'n' argument

Experiment 3:
I tried the following example:

map <- expression({
lapply(seq_along(map.values),function(r){
x <- runif(map.values[[r]])

rhcollect(map.keys[[r]],c(n=map.values[[r]],mean=mean(x),sd=sd(x)))
})
})

z <- rhmr(map, ofolder="/tmp/test",
inout=c('lapply','sequence'),
N=10,mapred=list(mapred.reduce.tasks=0,
mapred.reduce.max.attempts=1, mapred.map.max.attempts=1,
mapred.job.queue.name='interactive'),jobname='test')

rhex(z)

The resulting errors were the same.

Experiment 4:
This was motivated by a comment elsewhere on the forum
Command: R CMD /general/installed/R-2.12.1-acml/lib64/R/library/
Rhipe/libs/imperious.so
Result: Segmentation fault

Experiment 5:
I looked at ldd on imperious.so, and the results were the same for
the local and compute nodes. The Hadoop installation I'm using
doesn't have local jobtrackers enabled, so I couldn't test whether
things run locally.


I think I've been over everything, but I can't figure out the issue.
I've tried versions 0.65.1 (at least the patched version cited in a
link in another thread), 0.65.2, and 0.65.3.

I've seen online that mapred.input.file == null should possibly be
replaced with map.input.file = NULL, but I see no reason why this
should be a stumbling block for me, when others have had success with
RHIPE. So, is there something I'm missing?

Thanks!

David





___________________________________________________________
stdout logs
mapred.input.file == null

The stderr log is:
stderr logs

java.lang.RuntimeException: RHMRMapRed.waitOutputThreads(): subprocess
failed with code 127
at org.godhuli.rhipe.RHMRHelper.waitOutputThreads(RHMRHelper.java:
215)
at org.godhuli.rhipe.RHMRHelper.mapRedFinished(RHMRHelper.java:196)
at org.godhuli.rhipe.RHMRMapper.cleanup(RHMRMapper.java:124)
at org.godhuli.rhipe.RHMRMapper.run(RHMRMapper.java:65)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315)
at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:
1063)
at org.apache.hadoop.mapred.Child.main(Child.java:211)

And the final part of the syslog info is:
2011-03-14 10:32:56,991 INFO org.godhuli.rhipe.RHMRHelper:
Mapper:Started external program:/general/installed/R-2.12.1-acml/lib64/
R/library/Rhipe/libs/imperious.so --slave --silent --vanilla --max-
ppsize=100000 --max-nsize=1G
2011-03-14 10:32:56,993 INFO org.godhuli.rhipe.RHMRHelper:
Mapper:Started Error Thread
2011-03-14 10:32:56,994 INFO org.godhuli.rhipe.RHMRHelper:
Mapper:Started Output Thread
2011-03-14 10:32:57,033 INFO org.godhuli.rhipe.RHMRHelper:
Mapper:MROutputThread done
2011-03-14 10:32:57,033 INFO org.godhuli.rhipe.RHMRHelper:
Mapper:MRErrorThread done
2011-03-14 10:32:57,060 WARN org.apache.hadoop.mapred.Child: Error
running child
java.lang.RuntimeException: RHMRMapRed.waitOutputThreads(): subprocess
failed with code 127
at org.godhuli.rhipe.RHMRHelper.waitOutputThreads(RHMRHelper.java:
215)
at org.godhuli.rhipe.RHMRHelper.mapRedFinished(RHMRHelper.java:196)
at org.godhuli.rhipe.RHMRMapper.cleanup(RHMRMapper.java:124)
at org.godhuli.rhipe.RHMRMapper.run(RHMRMapper.java:65)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:315)
at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:
1063)
at org.apache.hadoop.mapred.Child.main(Child.java:211)
2011-03-14 10:32:57,082 INFO org.apache.hadoop.mapred.Task: Runnning
cleanup for the task

[Saptarshi Guha]

> First, many thanks for the RHIPE package and for the help contributed
> on this forum.  I've been banging my head against the wall to get
> through a series of errors, and now I'm stuck on some examples from
> the guide and from this forum.

Sorry to hear that ...

>
> Experiment 5:
>    I looked at ldd on imperious.so, and the results were the same for
> the local and compute nodes.  The Hadoop installation I'm using
> doesn't have local jobtrackers enabled, so I couldn't test whether
> things run locally.
>

It is probably an ldd issue. So start RHIPE,

library(Rhipe)
rhinit()
rhoptions()


what is the output of rhoptions()$runner[1] ?
(i'm using v. 0.65.2)

Now with that as output (mine is /home/ba01/u132/sguha/R_LIBS/Rhipe/libs/imperious.so )
run ldd on that

ldd ...

    libprotobuf.so.6 => /apps/rhel5/protobuf-2.3.0/lib/libprotobuf.so.6 (0x00002b35eac7d000)
    libz.so.1 => /usr/lib64/libz.so.1 (0x000000327f600000)
    libR.so => /apps/rhel5/R-2.11.1/lib64/R/lib/libR.so (0x00002b35eaf6d000)
    libstdc++.so.6 => /apps/coates/gcc-4.4.0/lib64/libstdc++.so.6 (0x00002b35eb4e4000)
    libm.so.6 => /lib64/libm.so.6 (0x000000327f200000)
    libgcc_s.so.1 => /apps/coates/gcc-4.4.0/lib64/libgcc_s.so.1 (0x00002b35eb7ef000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x000000327ee00000)
    libc.so.6 => /lib64/libc.so.6 (0x000000327e600000)
    libRblas.so => /apps/rhel5/R-2.11.1/lib64/R/lib/libRblas.so (0x00002b35eba06000)
    libgfortran.so.3 => /apps/coates/gcc-4.4.0/lib64/libgfortran.so.3 (0x00002b35ebc33000)
    libreadline.so.5 => /usr/lib64/libreadline.so.5 (0x0000003281600000)
    libncurses.so.5 => /usr/lib64/libncurses.so.5 (0x0000003282200000)
    libdl.so.2 => /lib64/libdl.so.2 (0x000000327ea00000)
    /lib64/ld-linux-x86-64.so.2 (0x000000327e200000)


When Hadoop starts, RHIPE wil launch an R process with the environment hadoop starts with.
This might  not contain a library search path that includes your libraries. One thing you could try is this

mapred = list( LD_LIBRARY_PATH=" /apps/rhel5/protobuf-2.3.0/lib: /usr/lib64/:/apps/coates/gcc-4.4.0/lib64: /apps/rhel5/R-2.11.1/lib64/R/lib/")
options(width=300)
map <- expression({
  for(x in map.values)
    rhcollect(x,runif(x))
})
z <- rhmr(map=map, N=10, inout=c("lapply","sequence"),ofolder="/wsc/tmp/sg2"
                ,mapred=append(mapred,list(mapred.map.tasks=2,mapred.reduce.tasks=0)))

rhex(z)

See if that works.

Cheers
Saptarshi

[David]

This was the key, thanks!

The key was adding the LD_LIBRARY_PATH. For the benefit of others
happening by, my steps were:

1: Get the output of: rhoptions$runner[1] # /general/installed/
R-2.12.1-acml/lib64/R/library/Rhipe/libs/imperious.so

2: At the command line, run: ldd /general/installed/R-2.12.1-acml/
lib64/R/library/Rhipe/libs/imperious.so

3: I took all of the unique directories listed from LDD results and
added them to the LD_LIBRARY_PATH string.
I now create a special list/string, called str_LD_LIBRARY_PATH
(like the "mapred" list variable you suggested),
to make this easy to generalize, and included R libraries, for
good measure, e.g.:

str_LD_LIBRARY_PATH = list( LD_LIBRARY_PATH= "/usr/lib64:/lib64:/
general/installed/lib:/general/installed/R-2.12.1-acml/lib64")

This then passed via rhmr as you suggested:

z <- rhmr(map=map, N=10, inout=c("lapply","sequence"),ofolder="/

user/david/rhipe0315a",
mapred=append(str_LD_LIBRARY_PATH,list(mapred.map.tasks=2,mapred.reduce.tasks=0)))

This matched your suggestion and seems to have worked perfectly.
All of my other little tests work, too, thanks!

David

Ps. My results are inline, below.

On Mar 14, 9:59 pm, Saptarshi Guha <saptarshi.g...@gmail.com> wrote:
> what is the output of rhoptions()$runner[1] ?

/general/installed/R-2.12.1-acml/lib64/R/library/Rhipe/libs/
imperious.so

> (i'm using v. 0.65.2)
>
> Now with that as output (mine is
> /home/ba01/u132/sguha/R_LIBS/Rhipe/libs/imperious.so )
> run ldd on that

linux-vdso.so.1 => (0x00007fff2d0a2000)
libprotobuf.so.6 => /usr/lib64/libprotobuf.so.6
(0x0000003f27400000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003d37200000)
libR.so => /general/installed/lib/libR.so (0x00002af119d1b000)
libstdc++.so.6 => /general/installed/lib/libstdc++.so.6
(0x00002af11a298000)
libm.so.6 => /lib64/libm.so.6 (0x0000003d36a00000)
libgcc_s.so.1 => /general/installed/lib/libgcc_s.so.1
(0x00002af11a59e000)
libc.so.6 => /lib64/libc.so.6 (0x0000003d36600000)
/lib64/ld-linux-x86-64.so.2 (0x0000003d36200000)
libRblas.so => /general/installed/lib/libRblas.so
(0x00002af11a7b4000)
libgfortran.so.1 => /usr/lib64/libgfortran.so.1
(0x00002af11c495000)
libreadline.so.5 => /usr/lib64/libreadline.so.5
(0x0000003d3ea00000)
libtermcap.so.2 => /lib64/libtermcap.so.2 (0x0000003d39200000)
libdl.so.2 => /lib64/libdl.so.2 (0x0000003d36e00000)
librt.so.1 => /lib64/librt.so.1 (0x0000003d37a00000)
libacml_mv.so => /general/installed/lib/libacml_mv.so
(0x00002af11c72e000)

[Saptarshi Guha]

If you can adjust $HADOOP_CONF/hadoop-env.sh
you can place the LD_LIBRARY_PATH environment variable inside that and restart your cluster.

This approach makes the approach described below redundant.