One question about running RHIPE code

[Zihang Yin]

When I try to run the following code
m <- expression({
  y <- strsplit(unlist(map.values)," ")
  lapply(y,function(r) rhcollect(r,T))
})
r <- expression(
    pre={
      count=0
    },
    reduce={
      count <- sum(as.numeric(unlist(reduce.

values)),count)
    },post={
      rhcollect(reduce.key,count)
    })
z=rhmr(map=m,reduce=r,comb=T,inout=c("text","sequence"),ifolder="/root/test1",ofolder='/root/out')
rhex(z)


I get this error message:
10/12/09 16:30:56 INFO mapred.JobClient: Task Id : attempt_201012091620_0001_m_000000_0, Status : FAILED
java.lang.RuntimeException: RHMRMapRed.waitOutputThreads(): subprocess failed with code 127
    at org.godhuli.rhipe.RHMRHelper.waitOutputThreads(RHMRHelper.java:215)
    at org.godhuli.rhipe.RHMRHelper.mapRedFinished(RHMRHelper.java:196)
    at org.godhuli.rhipe.RHMRMapper.cleanup(RHMRMapper.java:124)
    at org.godhuli.rhipe.RHMRMapper.run(RHMRMapper.java:65)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)

attempt_201012091620_0001_m_000000_0: java.lang.RuntimeException: RHMRMapRed.waitOutputThreads(): subprocess failed with code 127
attempt_201012091620_0001_m_000000_0:     at org.godhuli.rhipe.RHMRHelper.waitOutputThreads(RHMRHelper.java:215)
attempt_201012091620_0001_m_000000_0:     at org.godhuli.rhipe.RHMRHelper.mapRedFinished(RHMRHelper.java:196)
attempt_201012091620_0001_m_000000_0:     at org.godhuli.rhipe.RHMRMapper.cleanup(RHMRMapper.java:124)
attempt_201012091620_0001_m_000000_0:     at org.godhuli.rhipe.RHMRMapper.run(RHMRMapper.java:65)
attempt_201012091620_0001_m_000000_0:     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
attempt_201012091620_0001_m_000000_0:     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
attempt_201012091620_0001_m_000000_0:     at org.apache.hadoop.mapred.Child.main(Child.java:170)
10/12/09 16:31:02 INFO mapred.JobClient: Task Id : attempt_201012091620_0001_m_000000_1, Status : FAILED
java.lang.RuntimeException: RHMRMapRed.waitOutputThreads(): subprocess failed with code 127
    at org.godhuli.rhipe.RHMRHelper.waitOutputThreads(RHMRHelper.java:215)
    at org.godhuli.rhipe.RHMRHelper.mapRedFinished(RHMRHelper.java:196)
    at org.godhuli.rhipe.RHMRMapper.cleanup(RHMRMapper.java:124)
    at org.godhuli.rhipe.RHMRMapper.run(RHMRMapper.java:65)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)

attempt_201012091620_0001_m_000000_1: java.lang.RuntimeException: RHMRMapRed.waitOutputThreads(): subprocess failed with code 127
attempt_201012091620_0001_m_000000_1:     at org.godhuli.rhipe.RHMRHelper.waitOutputThreads(RHMRHelper.java:215)
attempt_201012091620_0001_m_000000_1:     at org.godhuli.rhipe.RHMRHelper.mapRedFinished(RHMRHelper.java:196)
attempt_201012091620_0001_m_000000_1:     at org.godhuli.rhipe.RHMRMapper.cleanup(RHMRMapper.java:124)
attempt_201012091620_0001_m_000000_1:     at org.godhuli.rhipe.RHMRMapper.run(RHMRMapper.java:65)
attempt_201012091620_0001_m_000000_1:     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
attempt_201012091620_0001_m_000000_1:     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
attempt_201012091620_0001_m_000000_1:     at org.apache.hadoop.mapred.Child.main(Child.java:170)
10/12/09 16:31:08 INFO mapred.JobClient: Task Id : attempt_201012091620_0001_m_000000_2, Status : FAILED
java.lang.RuntimeException: RHMRMapRed.waitOutputThreads(): subprocess failed with code 127
    at org.godhuli.rhipe.RHMRHelper.waitOutputThreads(RHMRHelper.java:215)
    at org.godhuli.rhipe.RHMRHelper.mapRedFinished(RHMRHelper.java:196)
    at org.godhuli.rhipe.RHMRMapper.cleanup(RHMRMapper.java:124)
    at org.godhuli.rhipe.RHMRMapper.run(RHMRMapper.java:65)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)

attempt_201012091620_0001_m_000000_2: java.lang.RuntimeException: RHMRMapRed.waitOutputThreads(): subprocess failed with code 127
attempt_201012091620_0001_m_000000_2:     at org.godhuli.rhipe.RHMRHelper.waitOutputThreads(RHMRHelper.java:215)
attempt_201012091620_0001_m_000000_2:     at org.godhuli.rhipe.RHMRHelper.mapRedFinished(RHMRHelper.java:196)
attempt_201012091620_0001_m_000000_2:     at org.godhuli.rhipe.RHMRMapper.cleanup(RHMRMapper.java:124)
attempt_201012091620_0001_m_000000_2:     at org.godhuli.rhipe.RHMRMapper.run(RHMRMapper.java:65)
attempt_201012091620_0001_m_000000_2:     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
attempt_201012091620_0001_m_000000_2:     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
attempt_201012091620_0001_m_000000_2:     at org.apache.hadoop.mapred.Child.main(Child.java:170)
10/12/09 16:31:17 INFO mapred.JobClient: Job complete: job_201012091620_0001
10/12/09 16:31:17 INFO mapred.JobClient: Counters: 3
10/12/09 16:31:17 INFO mapred.JobClient:   Job Counters
10/12/09 16:31:17 INFO mapred.JobClient:     Launched map tasks=4
10/12/09 16:31:17 INFO mapred.JobClient:     Data-local map tasks=4
10/12/09 16:31:17 INFO mapred.JobClient:     Failed map tasks=1
result:256
Error in readBin(f1, "raw", f2) : invalid 'n' argument

Also, I think I get the same error when run other rhipe script

[Saptarshi Guha]

Hi,

Probably an installation issue.

a) Do you have permissions to run local jobs? Then try

z=rhmr(map=m,reduce=r,comb=T,inout=c("text","sequence"),ifolder="/root/test1",ofolder='/root/out')

rhex(z,mapred=list(mapred.job.tracker='local'))

b) Also, on the worker nodes try this

R_HOME=/ln/meraki/custom/lib64/R/ RHIPEWHAT=0 RHIPELOGFILE="/tmp/x"
rhipe_outputformat_class=foo rhipe_combiner=0
/ln/meraki/custom/lib64/R/library/Rhipe/libs/imperious.so

Does (b) work (i.e does it )?

Change R_HOME (you can get it by typing R.home() in R) and change the
path of imperious to where Rhipe is located
R
library(Rhipe)
rhoptions()$runner[1]

Saptarshi

[Yin Zihang]

for a) I get the following error.

10/12/09 17:20:54 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=
10/12/09 17:20:54 INFO input.FileInputFormat: Total input paths to
process : 1
10/12/09 17:20:55 INFO rhipe.RHMR: Tracking URL ----> http://localhost:8080/
10/12/09 17:20:55 INFO mapred.JobClient: Running job: job_local_0001
10/12/09 17:20:55 INFO input.FileInputFormat: Total input paths to
process : 1
10/12/09 17:20:55 INFO mapred.MapTask: io.sort.mb = 100
10/12/09 17:20:56 INFO mapred.MapTask: data buffer = 79691776/99614720
10/12/09 17:20:56 INFO mapred.MapTask: record buffer = 262144/327680
10/12/09 17:20:56 INFO rhipe.RHMRHelper: Mapper:Started external
program:/usr/local/lib64/R/library/Rhipe/libs/imperious.so --slave
--silent --vanilla --max-ppsize=100000 --max-nsize=1G
10/12/09 17:20:56 INFO rhipe.RHMRHelper: Mapper:Started Error Thread
*** buffer overflow detected ***:
/usr/local/lib64/R/library/Rhipe/libs/imperious.so terminated
10/12/09 17:20:56 INFO rhipe.RHMRHelper: Mapper:Started Output Thread
======= Backtrace: =========
/lib/libc.so.6(__fortify_fail+0x37)[0x7f38160f3217]
/lib/libc.so.6(+0xfe0d0)[0x7f38160f20d0]
/lib/libc.so.6(+0xfd7cb)[0x7f38160f17cb]
/lib/libc.so.6(__snprintf_chk+0x7a)[0x7f38160f169a]
/usr/local/lib64/R/library/Rhipe/libs/imperious.so[0x411d95]
/lib/libc.so.6(__libc_start_main+0xfd)[0x7f3816012c4d]
/usr/local/lib64/R/library/Rhipe/libs/imperious.so[0x4061f9]
======= Memory map: ========
00400000-00416000 r-xp 00000000 08:01 394780
/usr/local/lib64/R/library/Rhipe/libs/imperious.so
00615000-00616000 r--p 00015000 08:01 394780
/usr/local/lib64/R/library/Rhipe/libs/imperious.so
00616000-00617000 rw-p 00016000 08:01 394780
/usr/local/lib64/R/library/Rhipe/libs/imperious.so
0143d000-0145e000 rw-p 0143d000 00:00 0
[heap]
7f3815456000-7f3815494000 r-xp 00000000 08:01 213121
/lib/libncurses.so.5.7
7f3815494000-7f3815694000 ---p 0003e000 08:01 213121
/lib/libncurses.so.5.7
7f3815694000-7f3815698000 r--p 0003e000 08:01 213121
/lib/libncurses.so.5.7
7f3815698000-7f3815699000 rw-p 00042000 08:01 213121
/lib/libncurses.so.5.7
7f3815699000-7f381569b000 r-xp 00000000 08:01 213101
/lib/libdl-2.11.1.so
7f381569b000-7f381589b000 ---p 00002000 08:01 213101
/lib/libdl-2.11.1.so
7f381589b000-7f381589c000 r--p 00002000 08:01 213101
/lib/libdl-2.11.1.so
7f381589c000-7f381589d000 rw-p 00003000 08:01 213101
/lib/libdl-2.11.1.so
7f381589d000-7f38158d6000 r-xp 00000000 08:01 213154
/lib/libreadline.so.6.1
7f38158d6000-7f3815ad5000 ---p 00039000 08:01 213154
/lib/libreadline.so.6.1
7f3815ad5000-7f3815ad7000 r--p 00038000 08:01 213154
/lib/libreadline.so.6.1
7f3815ad7000-7f3815add000 rw-p 0003a000 08:01 213154
/lib/libreadline.so.6.1
7f3815add000-7f3815ade000 rw-p 7f3815add000 00:00 0
7f3815ade000-7f3815bc9000 r-xp 00000000 08:01 32769
/usr/lib/libgfortran.so.3.0.0
7f3815bc9000-7f3815dc8000 ---p 000eb000 08:01 32769
/usr/lib/libgfortran.so.3.0.0
7f3815dc8000-7f3815dc9000 r--p 000ea000 08:01 32769
/usr/lib/libgfortran.so.3.0.0
7f3815dc9000-7f3815dca000 rw-p 000eb000 08:01 32769
/usr/lib/libgfortran.so.3.0.0
7f3815dca000-7f3815dcb000 rw-p 7f3815dca000 00:00 0
7f3815dcb000-7f3815df3000 r-xp 00000000 08:01 386725
/usr/local/lib64/R/lib/libRblas.so
7f3815df3000-7f3815ff2000 ---p 00028000 08:01 386725
/usr/local/lib64/R/lib/libRblas.so
7f3815ff2000-7f3815ff3000 r--p 00027000 08:01 386725
/usr/local/lib64/R/lib/libRblas.so
7f3815ff3000-7f3815ff4000 rw-p 00028000 08:01 386725
/usr/local/lib64/R/lib/libRblas.so
7f3815ff4000-7f381616e000 r-xp 00000000 08:01 213093
/lib/libc-2.11.1.so
7f381616e000-7f381636d000 ---p 0017a000 08:01 213093
/lib/libc-2.11.1.so
7f381636d000-7f3816371000 r--p 00179000 08:01 213093
/lib/libc-2.11.1.so
7f3816371000-7f3816372000 rw-p 0017d000 08:01 213093
/lib/libc-2.11.1.so
7f3816372000-7f3816377000 rw-p 7f3816372000 00:00 0
7f3816377000-7f381638f000 r-xp 00000000 08:01 213153
/lib/libpthread-2.11.1.so
7f381638f000-7f381658e000 ---p 00018000 08:01 213153
/lib/libpthread-2.11.1.so
7f381658e000-7f381658f000 r--p 00017000 08:01 213153
/lib/libpthread-2.11.1.so
7f381658f000-7f3816590000 rw-p 00018000 08:01 213153
/lib/libpthread-2.11.1.so
7f3816590000-7f3816594000 rw-p 7f3816590000 00:00 0
7f3816594000-7f38165aa000 r-xp 00000000 08:01 213111
/lib/libgcc_s.so.1
7f38165aa000-7f38167a9000 ---p 00016000 08:01 213111
/lib/libgcc_s.so.1
7f38167a9000-7f381610/12/09 17:20:56 INFO rhipe.RHMRHelper:
Mapper:MRErrorThread done
10/12/09 17:20:56 INFO rhipe.RHMRHelper: Mapper:MROutputThread done
java.lang.RuntimeException: RHMRMapRed.waitOutputThreads(): subprocess
failed with code 134

at org.godhuli.rhipe.RHMRHelper.waitOutputThreads(RHMRHelper.java:215)
at org.godhuli.rhipe.RHMRHelper.mapRedFinished(RHMRHelper.java:196)
at org.godhuli.rhipe.RHMRMapper.cleanup(RHMRMapper.java:124)
at org.godhuli.rhipe.RHMRMapper.run(RHMRMapper.java:65)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at

org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
10/12/09 17:20:56 WARN mapred.LocalJobRunner: job_local_0001
java.lang.RuntimeException: RHMRMapRed.waitOutputThreads(): subprocess
failed with code 134

at org.godhuli.rhipe.RHMRHelper.waitOutputThreads(RHMRHelper.java:215)
at org.godhuli.rhipe.RHMRHelper.mapRedFinished(RHMRHelper.java:196)
at org.godhuli.rhipe.RHMRMapper.cleanup(RHMRMapper.java:124)
at org.godhuli.rhipe.RHMRMapper.run(RHMRMapper.java:65)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at

org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
10/12/09 17:20:56 INFO mapred.JobClient: map 0% reduce 0%
10/12/09 17:20:56 INFO mapred.JobClient: Job complete: job_local_0001
10/12/09 17:20:56 INFO mapred.JobClient: Counters: 0

result:256
Error in readBin(f1, "raw", f2) : invalid 'n' argument

On 12/09/2010 12:17 PM, Saptarshi Guha wrote:
> z=rhmr(map=m,reduce=r,comb=T,inout=c("text","sequence"),ifolder="/root/test1",ofolder='/root/out')
> rhex(z,mapred=list(mapred.job.tracker='local'))
>

--
Best Regards, Zihang Yin 919-225-1717 Computer Science Duke University

[Yin Zihang]

for b) I don't understand what you want me to do/

R_HOME=/ln/meraki/custom/lib64/R/  (I set this as my environment varaible? right now I think this is /usr/local/lib64/R/) 


what's the following command?

RHIPEWHAT=0 RHIPELOGFILE="/tmp/x"

rhipe_outputformat_class=foo rhipe_combiner=0

/ln/meraki/custom/lib64/R/library/Rhipe/libs/imperious.so

I just run this in the worknode's terminal. Should I run this in R?

On 12/09/2010 12:17 PM, Saptarshi Guha wrote:

R_HOME=/ln/meraki/custom/lib64/R/ RHIPEWHAT=0 RHIPELOGFILE="/tmp/x"
rhipe_outputformat_class=foo rhipe_combiner=0
/ln/meraki/custom/lib64/R/library/Rhipe/libs/imperious.so
  

[Yin Zihang]

After I input

R
library(Rhipe)
rhoptions()$runner[1]

I get this
[1] "/usr/local/lib64/R/library/Rhipe/libs/imperious.so"
Then I rerun the job

I get the following error

ava.io.IOException: Cannot run program
"/usr/local/lib64/R/library/Rhipe/libs/imperious.so":
java.io.IOException: error=12, Cannot allocate memory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.godhuli.rhipe.RHMRHelper.setup(RHMRHelper.java:152)
at org.godhuli.rhipe.RHMRMapper.setup(RHMRMapper.java:78)
at org.godhuli.rhipe.RHMRMapper.run(RHMRMapper.java:50)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at

org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
Caused by: java.io.IOException: java.io.IOException: error=12, Cannot
allocate memory
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 6 more
10/12/09 17:28:41 WARN mapred.LocalJobRunner: job_local_0002
java.lang.RuntimeException: configuration exception
at org.godhuli.rhipe.RHMRHelper.setup(RHMRHelper.java:167)
at org.godhuli.rhipe.RHMRMapper.setup(RHMRMapper.java:78)
at org.godhuli.rhipe.RHMRMapper.run(RHMRMapper.java:50)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at

org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
Caused by: java.io.IOException: Cannot run program
"/usr/local/lib64/R/library/Rhipe/libs/imperious.so":
java.io.IOException: error=12, Cannot allocate memory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.godhuli.rhipe.RHMRHelper.setup(RHMRHelper.java:152)
... 5 more
Caused by: java.io.IOException: java.io.IOException: error=12, Cannot
allocate memory
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 6 more

On 12/09/2010 12:17 PM, Saptarshi Guha wrote:

[Saptarshi Guha]

No, not in R, run it on your current machine and on the work nodes.
paste it into the terminal - as one line without newlines

R_HOME=/ln/meraki/custom/lib64/R/ RHIPEWHAT=0 RHIPELOGFILE="/tmp/x"
rhipe_outputformat_class=foo rhipe_combiner=0
/ln/meraki/custom/lib64/R/library/Rhipe/libs/imperious.so

[Saptarshi Guha]

possibly a loaded system? no mem/no scratch?

aused by: java.io.IOException: java.io.IOException: error=12, Cannot
allocate memory

[Yin Zihang]

I checked cat /proc/meminfo

MemTotal: 499248 kB
MemFree: 302936 kB
Buffers: 1976 kB
Cached: 35404 kB
SwapCached: 8496 kB
Active: 35800 kB

I don't think that's the problem

[Yin Zihang]

The error now looks like this

======= Memory map: ========
00400000-00416000 r-xp 00000000 08:01 394780
/usr/local/lib64/R/library/Rhipe/libs/imperious.so
00615000-00616000 r--p 00015000 08:01 394780
/usr/local/lib64/R/library/Rhipe/libs/imperious.so
00616000-00617000 rw-p 00016000 08:01 394780
/usr/local/lib64/R/library/Rhipe/libs/imperious.so

023b4000-023d5000 rw-p 023b4000 00:00 0
[heap]
7fd19b39b000-7fd19b3d9000 r-xp 00000000 08:01 213121
/lib/libncurses.so.5.7
7fd19b3d9000-7fd19b5d9000 ---p 0003e000 08:01 213121
/lib/libncurses.so.5.7
7fd19b5d9000-7fd19b5dd000 r--p 0003e000 08:01 213121
/lib/libncurses.so.5.7
7fd19b5dd000-7fd19b5de000 rw-p 00042000 08:01 213121
/lib/libncurses.so.5.7
7fd19b5de000-7fd19b5e0000 r-xp 00000000 08:01 213101
/lib/libdl-2.11.1.so
7fd19b5e0000-7fd19b7e0000 ---p 00002000 08:01 213101
/lib/libdl-2.11.1.so
7fd19b7e0000-7fd19b7e1000 r--p 00002000 08:01 213101
/lib/libdl-2.11.1.so
7fd19b7e1000-7fd19b7e2000 rw-p 00003000 08:01 213101
/lib/libdl-2.11.1.so
7fd19b7e2000-7fd19b81b000 r-xp 00000000 08:01 213154
/lib/libreadline.so.6.1
7fd19b81b000-7fd19ba1a000 ---p 00039000 08:01 213154
/lib/libreadline.so.6.1
7fd19ba1a000-7fd19ba1c000 r--p 00038000 08:01 213154
/lib/libreadline.so.6.1
7fd19ba1c000-7fd19ba22000 rw-p 0003a000 08:01 213154
/lib/libreadline.so.6.1
7fd19ba22000-7fd19ba23000 rw-p 7fd19ba22000 00:00 0
7fd19ba23000-7fd19bb0e000 r-xp 00000000 08:01 32769
/usr/lib/libgfortran.so.3.0.0
7fd19bb0e000-7fd19bd0d000 ---p 000eb000 08:01 32769
/usr/lib/libgfortran.so.3.0.0
7fd19bd0d000-7fd19bd0e000 r--p 000ea000 08:01 32769
/usr/lib/libgfortran.so.3.0.0
7fd19bd0e000-7fd19bd0f000 rw-p 000eb000 08:01 32769
/usr/lib/libgfortran.so.3.0.0
7fd19bd0f000-7fd19bd10000 rw-p 7fd19bd0f000 00:00 0
7fd19bd10000-7fd19bd38000 r-xp 00000000 08:01 386725
/usr/local/lib64/R/lib/libRblas.so
7fd19bd38000-7fd19bf37000 ---p 00028000 08:01 386725
/usr/local/lib64/R/lib/libRblas.so
7fd19bf37000-7fd19bf38000 r--p 00027000 08:01 386725
/usr/local/lib64/R/lib/libRblas.so
7fd19bf38000-7fd19bf39000 rw-p 00028000 08:01 386725
/usr/local/lib64/R/lib/libRblas.so
7fd19bf39000-7fd19c0b3000 r-xp 00000000 08:01 213093
/lib/libc-2.11.1.so
7fd19c0b3000-7fd19c2b2000 ---p 0017a000 08:01 213093
/lib/libc-2.11.1.so
7fd19c2b2000-7fd19c2b6000 r--p 00179000 08:01 213093
/lib/libc-2.11.1.so
7fd19c2b6000-7fd19c2b7000 rw-p 0017d000 08:01 213093
/lib/libc-2.11.1.so
7fd19c2b7000-7fd19c2bc000 rw-p 7fd19c2b7000 00:00 0
7fd19c2bc000-7fd19c2d4000 r-xp 00000000 08:01 213153
/lib/libpthread-2.11.1.so
7fd19c2d4000-7fd19c4d3000 ---p 00018000 08:01 213153
/lib/libpthread-2.11.1.so
7fd19c4d3000-7fd19c4d4000 r--p 00017000 08:01 213153
/lib/libpthread-2.11.1.so
7fd19c4d4000-7fd19c4d5000 rw-p 00018000 08:01 213153
/lib/libpthread-2.11.1.so
7fd19c4d5000-7fd19c4d9000 rw-p 7fd19c4d5000 00:00 0
7fd19c4d9000-7fd19c4ef000 r-xp 00000000 08:01 213111
/lib/libgcc_s.so.1
7fd19c4ef000-7fd19c6ee000 ---p 00016000 08:01 213111
/lib/libgcc_s.so.1
7fd19c6ee000-7fd19c10/12/09 17:39:48 INFO rhipe.RHMRHelper:

Mapper:Started external
program:/usr/local/lib64/R/library/Rhipe/libs/imperious.so --slave
--silent --vanilla --max-ppsize=100000 --max-nsize=1G

10/12/09 17:39:48 INFO rhipe.RHMRHelper: Mapper:Started Error Thread
10/12/09 17:39:48 INFO rhipe.RHMRHelper: Mapper:MRErrorThread done
10/12/09 17:39:48 INFO rhipe.RHMRHelper: Mapper:MROutputThread done
10/12/09 17:39:48 INFO rhipe.RHMRHelper: Mapper:Started Output Thread
java.lang.RuntimeException: RHMRMapRed.waitOutputThreads(): subprocess
failed with code 134

at org.godhuli.rhipe.RHMRHelper.waitOutputThreads(RHMRHelper.java:215)
at org.godhuli.rhipe.RHMRHelper.mapRedFinished(RHMRHelper.java:196)
at org.godhuli.rhipe.RHMRMapper.cleanup(RHMRMapper.java:124)
at org.godhuli.rhipe.RHMRMapper.run(RHMRMapper.java:65)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at

org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
10/12/09 17:39:48 WARN mapred.LocalJobRunner: job_local_0001
java.lang.RuntimeException: RHMRMapRed.waitOutputThreads(): subprocess
failed with code 134

at org.godhuli.rhipe.RHMRHelper.waitOutputThreads(RHMRHelper.java:215)
at org.godhuli.rhipe.RHMRHelper.mapRedFinished(RHMRHelper.java:196)
at org.godhuli.rhipe.RHMRMapper.cleanup(RHMRMapper.java:124)
at org.godhuli.rhipe.RHMRMapper.run(RHMRMapper.java:65)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at

org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
10/12/09 17:39:49 INFO mapred.JobClient: map 0% reduce 0%
10/12/09 17:39:49 INFO mapred.JobClient: Job complete: job_local_0001
10/12/09 17:39:49 INFO mapred.JobClient: Counters: 0

result:256
Error in readBin(f1, "raw", f2) : invalid 'n' argument

On 12/09/2010 12:31 PM, Saptarshi Guha wrote:

[Yin Zihang]

-bash: /ln/meraki/custom/lib64/R/library/Rhipe/libs/imperious.so: No such file or directory

[Yin Zihang]

Basically
If I run rhex(z,mapred=list(mapred.job.tracker='local')), I got the
following info

java.lang.RuntimeException: RHMRMapRed.waitOutputThreads(): subprocess
failed with code 134

If I run rhex(z)
I get
0/12/09 17:53:03 INFO mapred.JobClient: Task Id :
attempt_201012091620_0006_m_000001_2, Status : FAILED

[Saptarshi Guha]

That is my path. You need to change it to your path.

R
library(Rhipe)
rhoptions()$runner[1]

[Yin Zihang]

R_HOME=/usr/local/lib64/R/ RHIPEWHAT=0 RHIPELOGFILE="/tmp/x"
rhipe_outputformat_class=foo rhipe_combiner=0
/usr/local/lib64/R/library/Rhipe/libs/imperious.so

I get this
/usr/local/lib64/R/library/Rhipe/libs/imperious.so: error while loading shared libraries: libR.so: cannot open shared object file: No such file or directory

[Saptarshi Guha]

That wont help. Segfaults can be nasty. Also, i fixed a glaring buffer
overun (sprintf).
New version is http://www.stat.purdue.edu/~sguha/rhipe/dn/Rhipe_0.65.1.tar.gz

[Saptarshi Guha]

Your R libraries are not exposed. Place the R library using LD_LIBRARY_PATH

[Yin Zihang]

done that. but same error message.

[Yin Zihang]

I am now using the newest version

[Saptarshi Guha]

If you get this error

I get this
/usr/local/lib64/R/library/Rhipe/libs/imperious.so: error while
loading shared libraries: libR.so: cannot open shared object file: No
such file or directory

it means its not in the library path.
what does ldd /usr/local/lib64/R/library/Rhipe/libs/imperious.so

say?

[Yin Zihang]

I think I already fix this problem.

It says:
linux-vdso.so.1 => (0x00007fff30dfe000)
libprotobuf.so.6 => /usr/local/lib/libprotobuf.so.6
(0x00007f912897d000)
libR.so => /usr/local/lib64/R/lib/libR.so (0x00007f91283f5000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007f91280e0000)
libm.so.6 => /lib/libm.so.6 (0x00007f9127e5d000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007f9127c46000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00007f9127a28000)
libc.so.6 => /lib/libc.so.6 (0x00007f91276a5000)
libRblas.so => /usr/local/lib64/R/lib/libRblas.so (0x00007f912747c000)
libgfortran.so.3 => /usr/lib/libgfortran.so.3 (0x00007f912718e000)
libreadline.so.6 => /lib/libreadline.so.6 (0x00007f9126f4d000)
libdl.so.2 => /lib/libdl.so.2 (0x00007f9126d49000)
/lib64/ld-linux-x86-64.so.2 (0x00007f9128c63000)
libncurses.so.5 => /lib/libncurses.so.5 (0x00007f9126b05000)

[Yin Zihang]

This is the job log file

<?xml version="1.0" encoding="UTF-8" standalone="no"?><configuration>
<property><name>fs.s3n.impl</name><value>org.apache.hadoop.fs.s3native.NativeS3FileSystem</value></property>
<property><name>mapred.task.cache.levels</name><value>2</value></property>
<property><name>hadoop.tmp.dir</name><value>/usr/local/TMP_DIR</value></property>
<property><name>hadoop.native.lib</name><value>true</value></property>
<property><name>map.sort.class</name><value>org.apache.hadoop.util.QuickSort</value></property>
<property><name>dfs.namenode.decommission.nodes.per.interval</name><value>5</value></property>
<property><name>dfs.https.need.client.auth</name><value>false</value></property>
<property><name>rhipe_outputformat_valueclass</name><value>org.godhuli.rhipe.RHBytesWritable</value></property>
<property><name>rhipe_lapply_lengthofinput</name><value>10</value></property>
<property><name>ipc.client.idlethreshold</name><value>4000</value></property>
<property><name>mapred.system.dir</name><value>${hadoop.tmp.dir}/mapred/system</value></property>
<property><name>mapred.job.tracker.persist.jobstatus.hours</name><value>0</value></property>
<property><name>dfs.namenode.logging.level</name><value>info</value></property>
<property><name>dfs.datanode.address</name><value>0.0.0.0:50010</value></property>
<property><name>io.skip.checksum.errors</name><value>false</value></property>
<property><name>fs.default.name</name><value>hdfs://192.168.1.66:9000</value></property>
<property><name>rhipe_inputformat_class</name><value>org.godhuli.rhipe.LApplyInputFormat</value></property>
<property><name>mapred.textoutputformat.usekey</name><value>TRUE</value></property>
<property><name>mapred.child.tmp</name><value>./tmp</value></property>
<property><name>fs.har.impl.disable.cache</name><value>true</value></property>
<property><name>dfs.safemode.threshold.pct</name><value>0.999f</value></property>
<property><name>mapred.skip.reduce.max.skip.groups</name><value>0</value></property>
<property><name>dfs.namenode.handler.count</name><value>10</value></property>
<property><name>dfs.blockreport.initialDelay</name><value>0</value></property>
<property><name>rhipe_setup_reduce</name><value>A
2
134144
131840
20
0
</value></property>
<property><name>mapred.jobtracker.instrumentation</name><value>org.apache.hadoop.mapred.JobTrackerMetricsInst</value></property>
<property><name>mapred.tasktracker.dns.nameserver</name><value>default</value></property>
<property><name>io.sort.factor</name><value>10</value></property>
<property><name>mapred.task.timeout</name><value>600000</value></property>
<property><name>mapred.max.tracker.failures</name><value>4</value></property>
<property><name>hadoop.rpc.socket.factory.class.default</name><value>org.apache.hadoop.net.StandardSocketFactory</value></property>
<property><name>mapred.mapoutput.key.class</name><value>org.godhuli.rhipe.RHBytesWritable</value></property>
<property><name>fs.hdfs.impl</name><value>org.apache.hadoop.hdfs.DistributedFileSystem</value></property>
<property><name>mapred.queue.default.acl-administer-jobs</name><value>*</value></property>
<property><name>mapred.queue.default.acl-submit-job</name><value>*</value></property>
<property><name>mapred.output.key.class</name><value>org.godhuli.rhipe.RHBytesWritable</value></property>
<property><name>mapred.skip.map.auto.incr.proc.count</name><value>true</value></property>
<property><name>io.mapfile.bloom.size</name><value>1048576</value></property>
<property><name>dfs.safemode.extension</name><value>30000</value></property>
<property><name>tasktracker.http.threads</name><value>40</value></property>
<property><name>mapred.job.shuffle.merge.percent</name><value>0.66</value></property>
<property><name>mapreduce.inputformat.class</name><value>org.godhuli.rhipe.LApplyInputFormat</value></property>
<property><name>fs.ftp.impl</name><value>org.apache.hadoop.fs.ftp.FTPFileSystem</value></property>
<property><name>user.name</name><value>root</value></property>
<property><name>mapred.output.compress</name><value>false</value></property>
<property><name>io.bytes.per.checksum</name><value>512</value></property>
<property><name>rhipe_reduce</name><value>A
2
134144
131840
254
</value></property>
<property><name>topology.node.switch.mapping.impl</name><value>org.apache.hadoop.net.ScriptBasedMapping</value></property>
<property><name>dfs.https.server.keystore.resource</name><value>ssl-server.xml</value></property>
<property><name>mapred.reduce.slowstart.completed.maps</name><value>0.05</value></property>
<property><name>mapred.reduce.max.attempts</name><value>4</value></property>
<property><name>fs.ramfs.impl</name><value>org.apache.hadoop.fs.InMemoryFileSystem</value></property>
<property><name>dfs.name.edits.dir</name><value>${dfs.name.dir}</value></property>
<property><name>mapred.skip.map.max.skip.records</name><value>0</value></property>
<property><name>hadoop.job.ugi</name><value>root,root</value></property>
<property><name>mapred.job.tracker.persist.jobstatus.dir</name><value>/jobtracker/jobsInfo</value></property>
<property><name>mapred.jar</name><value>/usr/local/TMP_DIR/mapred/local/jobTracker/job_201012091620_0009.jar</value></property>
<property><name>dfs.block.size</name><value>67108864</value></property>
<property><name>fs.s3.buffer.dir</name><value>${hadoop.tmp.dir}/s3</value></property>
<property><name>job.end.retry.attempts</name><value>0</value></property>
<property><name>fs.file.impl</name><value>org.apache.hadoop.fs.LocalFileSystem</value></property>
<property><name>mapred.local.dir.minspacestart</name><value>0</value></property>
<property><name>mapred.output.compression.type</name><value>RECORD</value></property>
<property><name>rhipe_stream_buffer</name><value>10240</value></property>
<property><name>dfs.datanode.ipc.address</name><value>0.0.0.0:50020</value></property>
<property><name>dfs.permissions</name><value>true</value></property>
<property><name>topology.script.number.args</name><value>100</value></property>
<property><name>R_HOME</name><value>/usr/local/lib64/R</value></property>
<property><name>io.mapfile.bloom.error.rate</name><value>0.005</value></property>
<property><name>mapred.max.tracker.blacklists</name><value>4</value></property>
<property><name>rhipe.use.hadoop.combiner</name><value>FALSE</value></property>
<property><name>mapred.task.profile.maps</name><value>0-2</value></property>
<property><name>dfs.datanode.https.address</name><value>0.0.0.0:50475</value></property>
<property><name>mapred.userlog.retain.hours</name><value>24</value></property>
<property><name>dfs.secondary.http.address</name><value>0.0.0.0:50090</value></property>
<property><name>dfs.replication.max</name><value>512</value></property>
<property><name>mapred.job.tracker.persist.jobstatus.active</name><value>false</value></property>
<property><name>rhipe_job_verbose</name><value>TRUE</value></property>
<property><name>hadoop.security.authorization</name><value>false</value></property>
<property><name>local.cache.size</name><value>10737418240</value></property>
<property><name>mapred.min.split.size</name><value>0</value></property>
<property><name>mapred.map.tasks</name><value>2</value></property>
<property><name>RHIPE_DEBUG</name><value>0</value></property>
<property><name>mapred.child.java.opts</name><value>-Xmx300m</value></property>
<property><name>rhipe_command</name><value>/usr/local/lib64/R/library/Rhipe/libs/imperious.so
--slave --silent --vanilla --max-ppsize=100000
--max-nsize=1G</value></property>
<property><name>mapred.output.value.class</name><value>org.godhuli.rhipe.RHBytesWritable</value></property>
<property><name>dfs.https.client.keystore.resource</name><value>ssl-client.xml</value></property>
<property><name>mapred.job.queue.name</name><value>default</value></property>
<property><name>dfs.https.address</name><value>0.0.0.0:50470</value></property>
<property><name>dfs.balance.bandwidthPerSec</name><value>1048576</value></property>
<property><name>ipc.server.listen.queue.size</name><value>128</value></property>
<property><name>rhipe_map</name><value>A
2
134144
131840
20
1
6
1
9
1
{
2
6
1
9
6
lapply
2
6
1
9
9
seq_along
2
1
9
10
map.values
254
2
6
1
9
8
function
2
1026
1
9
1
r
251
254
2
6
511
2
6
1
9
2
&lt;-
2
1
9
1
x
2
6
1
9
5
runif
2
6
1
9
2
[[
2
1279
2
1791
254
254
254
2
6
1
9
9
rhcollect
2
6
2815
2
1
9
8
map.keys
2
1791
254
2
6
1
9
1
c
1026
1
9
1
n
6
2815
2
1279
2
1791
254
1026
1
9
4
mean
6
4095
2
2303
254
1026
1
9
2
sd
6
4351
2
2303
254
254
254
254
2
16
4
9
12
function(r){
9
27
x\040&lt;-\040runif(map.values[[r]])
9
67
rhcollect(map.keys[[r]],c(n=map.values[[r]],mean=mean(x),sd=sd(x)))
9
1
}
254
254
254
</value></property>
<property><name>group.name</name><value>root</value></property>
<property><name>mapred.inmem.merge.threshold</name><value>1000</value></property>
<property><name>job.end.retry.interval</name><value>30000</value></property>
<property><name>mapred.skip.attempts.to.start.skipping</name><value>2</value></property>
<property><name>fs.checkpoint.dir</name><value>${hadoop.tmp.dir}/dfs/namesecondary</value></property>
<property><name>mapred.reduce.tasks</name><value>0</value></property>
<property><name>mapred.merge.recordsBeforeProgress</name><value>10000</value></property>
<property><name>mapred.userlog.limit.kb</name><value>0</value></property>
<property><name>dfs.max.objects</name><value>0</value></property>
<property><name>webinterface.private.actions</name><value>false</value></property>
<property><name>rhipe_output_folder</name><value>/root/test/</value></property>
<property><name>io.sort.spill.percent</name><value>0.80</value></property>
<property><name>mapred.job.shuffle.input.buffer.percent</name><value>0.70</value></property>
<property><name>mapred.job.split.file</name><value>hdfs://ip-192-168-1-66.local:9000/usr/local/TMP_DIR/mapred/system/job_201012091620_0009/job.split</value></property>
<property><name>mapred.job.name</name><value>test</value></property>
<property><name>dfs.datanode.dns.nameserver</name><value>default</value></property>
<property><name>mapred.map.tasks.speculative.execution</name><value>true</value></property>
<property><name>hadoop.util.hash.type</name><value>murmur</value></property>
<property><name>dfs.blockreport.intervalMsec</name><value>3600000</value></property>
<property><name>mapred.map.max.attempts</name><value>4</value></property>
<property><name>rhipe_setup_map</name><value>A
2
134144
131840
20
0
</value></property>
<property><name>rhipe_reduce_prekey</name><value>A
2
134144
131840
254
</value></property>
<property><name>dfs.client.block.write.retries</name><value>3</value></property>
<property><name>mapred.job.tracker.handler.count</name><value>10</value></property>
<property><name>mapred.tasktracker.expiry.interval</name><value>600000</value></property>
<property><name>dfs.https.enable</name><value>false</value></property>
<property><name>mapred.jobtracker.maxtasks.per.job</name><value>-1</value></property>
<property><name>mapred.jobtracker.job.history.block.size</name><value>3145728</value></property>
<property><name>keep.failed.task.files</name><value>false</value></property>
<property><name>rhipe_cleanup_reduce</name><value>A
2
134144
131840
20
0
</value></property>
<property><name>mapreduce.outputformat.class</name><value>org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat</value></property>
<property><name>ipc.client.tcpnodelay</name><value>false</value></property>
<property><name>mapred.task.profile.reduces</name><value>0-2</value></property>
<property><name>mapred.output.compression.codec</name><value>org.apache.hadoop.io.compress.DefaultCodec</value></property>
<property><name>io.map.index.skip</name><value>0</value></property>
<property><name>mapred.working.dir</name><value>hdfs://ip-192-168-1-66.local:9000/user/root</value></property>
<property><name>ipc.server.tcpnodelay</name><value>false</value></property>
<property><name>mapred.used.genericoptionsparser</name><value>true</value></property>
<property><name>mapred.mapper.new-api</name><value>true</value></property>
<property><name>rhipe_partitioner_class</name><value>none</value></property>
<property><name>dfs.default.chunk.view.size</name><value>32768</value></property>
<property><name>hadoop.logfile.size</name><value>10000000</value></property>
<property><name>mapred.reduce.tasks.speculative.execution</name><value>true</value></property>
<property><name>dfs.datanode.du.reserved</name><value>0</value></property>
<property><name>rhipe_inputformat_valueclass</name><value>org.godhuli.rhipe.RHNumeric</value></property>
<property><name>fs.checkpoint.period</name><value>3600</value></property>
<property><name>rhipe_map_buff_size</name><value>3000</value></property>
<property><name>dfs.web.ugi</name><value>webuser,webgroup</value></property>
<property><name>mapred.job.reuse.jvm.num.tasks</name><value>-1</value></property>
<property><name>mapred.jobtracker.completeuserjobs.maximum</name><value>100</value></property>
<property><name>dfs.df.interval</name><value>60000</value></property>
<property><name>dfs.data.dir</name><value>/usr/local/DATA_DIR</value></property>
<property><name>rhipe_reduce_justcollect</name><value>TRUE</value></property>
<property><name>fs.s3.maxRetries</name><value>4</value></property>
<property><name>dfs.datanode.dns.interface</name><value>default</value></property>
<property><name>rhipe_jobname</name><value>test</value></property>
<property><name>mapred.mapoutput.value.class</name><value>org.godhuli.rhipe.RHBytesWritable</value></property>
<property><name>dfs.support.append</name><value>false</value></property>
<property><name>rhipe_inputformat_keyclass</name><value>org.godhuli.rhipe.RHNumeric</value></property>
<property><name>dfs.permissions.supergroup</name><value>supergroup</value></property>
<property><name>mapred.local.dir</name><value>${hadoop.tmp.dir}/mapred/local</value></property>
<property><name>fs.hftp.impl</name><value>org.apache.hadoop.hdfs.HftpFileSystem</value></property>
<property><name>rhipe_map_output_keyclass</name><value>org.godhuli.rhipe.RHBytesWritable</value></property>
<property><name>fs.trash.interval</name><value>0</value></property>
<property><name>fs.s3.sleepTimeSeconds</name><value>10</value></property>
<property><name>dfs.replication.min</name><value>1</value></property>
<property><name>mapred.submit.replication</name><value>10</value></property>
<property><name>fs.har.impl</name><value>org.apache.hadoop.fs.HarFileSystem</value></property>
<property><name>mapred.map.output.compression.codec</name><value>org.apache.hadoop.io.compress.DefaultCodec</value></property>
<property><name>mapred.tasktracker.dns.interface</name><value>default</value></property>
<property><name>dfs.namenode.decommission.interval</name><value>30</value></property>
<property><name>dfs.http.address</name><value>0.0.0.0:50070</value></property>
<property><name>rhipe_job_async</name><value>FALSE</value></property>
<property><name>dfs.heartbeat.interval</name><value>3</value></property>
<property><name>mapred.job.tracker</name><value>192.168.1.66:9100</value></property>
<property><name>io.seqfile.sorter.recordlimit</name><value>1000000</value></property>
<property><name>dfs.name.dir</name><value>/usr/local/NAME_DIR</value></property>
<property><name>mapred.line.input.format.linespermap</name><value>1</value></property>
<property><name>mapred.jobtracker.taskScheduler</name><value>org.apache.hadoop.mapred.JobQueueTaskScheduler</value></property>
<property><name>mapred.create.symlink</name><value>yes</value></property>
<property><name>rhipe_cleanup_map</name><value>A
2
134144
131840
20
0
</value></property>
<property><name>mapred.tasktracker.instrumentation</name><value>org.apache.hadoop.mapred.TaskTrackerMetricsInst</value></property>
<property><name>dfs.datanode.http.address</name><value>0.0.0.0:50075</value></property>
<property><name>rhipe_reduce_postkey</name><value>A
2
134144
131840
254
</value></property>
<property><name>rhipe_send_keys_to_map</name><value>1</value></property>
<property><name>mapred.tasktracker.procfsbasedprocesstree.sleeptime-before-sigkill</name><value>5000</value></property>
<property><name>mapred.local.dir.minspacekill</name><value>0</value></property>
<property><name>dfs.replication.interval</name><value>3</value></property>
<property><name>io.sort.record.percent</name><value>0.05</value></property>
<property><name>mapreduce.reduce.class</name><value>org.godhuli.rhipe.RHMRReducer</value></property>
<property><name>fs.kfs.impl</name><value>org.apache.hadoop.fs.kfs.KosmosFileSystem</value></property>
<property><name>mapred.temp.dir</name><value>${hadoop.tmp.dir}/mapred/temp</value></property>
<property><name>mapred.tasktracker.reduce.tasks.maximum</name><value>2</value></property>
<property><name>dfs.replication</name><value>2</value></property>
<property><name>fs.checkpoint.edits.dir</name><value>${fs.checkpoint.dir}</value></property>
<property><name>mapred.job.reduce.input.buffer.percent</name><value>0.0</value></property>
<property><name>mapred.tasktracker.indexcache.mb</name><value>10</value></property>
<property><name>hadoop.logfile.count</name><value>10</value></property>
<property><name>mapred.skip.reduce.auto.incr.proc.count</name><value>true</value></property>
<property><name>rhipe_map_output_valueclass</name><value>org.godhuli.rhipe.RHBytesWritable</value></property>
<property><name>io.seqfile.compress.blocksize</name><value>1000000</value></property>
<property><name>fs.s3.block.size</name><value>67108864</value></property>
<property><name>mapred.tasktracker.taskmemorymanager.monitoring-interval</name><value>5000</value></property>
<property><name>rhipe_copy_file</name><value>FALSE</value></property>
<property><name>mapred.acls.enabled</name><value>false</value></property>
<property><name>rhipe_outputformat_keyclass</name><value>org.godhuli.rhipe.RHBytesWritable</value></property>
<property><name>mapred.queue.names</name><value>default</value></property>
<property><name>dfs.access.time.precision</name><value>3600000</value></property>
<property><name>fs.hsftp.impl</name><value>org.apache.hadoop.hdfs.HsftpFileSystem</value></property>
<property><name>mapred.task.tracker.http.address</name><value>0.0.0.0:50060</value></property>
<property><name>mapred.reduce.parallel.copies</name><value>5</value></property>
<property><name>io.seqfile.lazydecompress</name><value>true</value></property>
<property><name>mapred.output.dir</name><value>/root/test</value></property>
<property><name>io.sort.mb</name><value>100</value></property>
<property><name>ipc.client.connection.maxidletime</name><value>10000</value></property>
<property><name>mapred.compress.map.output</name><value>true</value></property>
<property><name>mapred.task.tracker.report.address</name><value>127.0.0.1:0</value></property>
<property><name>ipc.client.kill.max</name><value>10</value></property>
<property><name>ipc.client.connect.max.retries</name><value>10</value></property>
<property><name>mapreduce.map.class</name><value>org.godhuli.rhipe.RHMRMapper</value></property>
<property><name>fs.s3.impl</name><value>org.apache.hadoop.fs.s3.S3FileSystem</value></property>
<property><name>rhipe_reduce_buff_size</name><value>6000</value></property>
<property><name>rhipe_outputformat_class</name><value>org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat</value></property>
<property><name>mapred.job.tracker.http.address</name><value>0.0.0.0:50030</value></property>
<property><name>io.file.buffer.size</name><value>4096</value></property>
<property><name>mapred.jobtracker.restart.recover</name><value>false</value></property>
<property><name>io.serializations</name><value>org.apache.hadoop.io.serializer.WritableSerialization</value></property>
<property><name>rhipe_combiner</name><value>0</value></property>
<property><name>dfs.datanode.handler.count</name><value>3</value></property>
<property><name>mapred.reduce.copy.backoff</name><value>300</value></property>
<property><name>mapred.task.profile</name><value>false</value></property>
<property><name>dfs.replication.considerLoad</name><value>true</value></property>
<property><name>jobclient.output.filter</name><value>FAILED</value></property>
<property><name>rhipe_map_input_type</name><value>default</value></property>
<property><name>mapred.tasktracker.map.tasks.maximum</name><value>2</value></property>
<property><name>io.compression.codecs</name><value>org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec</value></property>
<property><name>fs.checkpoint.size</name><value>67108864</value></property>
</configuration>

On 12/09/2010 01:21 PM, Saptarshi Guha wrote:

[Saptarshi Guha]

The job log file is not important (right now). It has to be able to
run at the command line.
Once that is ready, we can try running it in hadoop.

So, does this work

RHIPEWHAT=0 RHIPELOGFILE="/tmp/x" rhipe_outputformat_class=foo
rhipe_combiner=0 R CMD
/ln/meraki/custom/lib64/R/library/Rhipe/libs/imperious.so

(one line, change the last path)

[Yin Zihang]

Another log

eta VERSION="1" .
Job JOBID="job_201012091620_0009" JOBNAME="test" USER="root"
SUBMIT_TIME="1291919194265"
JOBCONF="hdfs://ip-192-168-1-66\.local:9000/usr/local/TMP_DIR/mapred/system/job_201012091620_0009/job\.xml"
.
Job JOBID="job_201012091620_0009" JOB_PRIORITY="NORMAL" .
Job JOBID="job_201012091620_0009" LAUNCH_TIME="1291919194539"
TOTAL_MAPS="2" TOTAL_REDUCES="0" JOB_STATUS="PREP" .
Task TASKID="task_201012091620_0009_m_000003" TASK_TYPE="SETUP"
START_TIME="1291919196162" SPLITS="" .
MapAttempt TASK_TYPE="SETUP" TASKID="task_201012091620_0009_m_000003"
TASK_ATTEMPT_ID="attempt_201012091620_0009_m_000003_0"
START_TIME="1291919196420"
TRACKER_NAME="tracker_ip-192-168-1-67\.local:localhost/127\.0\.0\.1:56007"
HTTP_PORT="50060" .
MapAttempt TASK_TYPE="SETUP" TASKID="task_201012091620_0009_m_000003"
TASK_ATTEMPT_ID="attempt_201012091620_0009_m_000003_0"
TASK_STATUS="SUCCESS" FINISH_TIME="1291919197659"
HOSTNAME="/default-rack/ip-192-168-1-67\.local" STATE_STRING="setup"
COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
Task TASKID="task_201012091620_0009_m_000003" TASK_TYPE="SETUP"
TASK_STATUS="SUCCESS" FINISH_TIME="1291919199172"
COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
Job JOBID="job_201012091620_0009" JOB_STATUS="RUNNING" .
Task TASKID="task_201012091620_0009_m_000000" TASK_TYPE="MAP"
START_TIME="1291919199173" SPLITS="" .
Task TASKID="task_201012091620_0009_m_000001" TASK_TYPE="MAP"
START_TIME="1291919202181" SPLITS="" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201012091620_0009_m_000000"
TASK_ATTEMPT_ID="attempt_201012091620_0009_m_000000_0"
START_TIME="1291919199220"
TRACKER_NAME="tracker_ip-192-168-1-67\.local:localhost/127\.0\.0\.1:56007"
HTTP_PORT="50060" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201012091620_0009_m_000000"
TASK_ATTEMPT_ID="attempt_201012091620_0009_m_000000_0"
TASK_STATUS="FAILED" FINISH_TIME="1291919204292"
HOSTNAME="ip-192-168-1-67\.local" ERROR="java\.lang\.RuntimeException:
RHMRMapRed\.waitOutputThreads(): subprocess failed with code 127
at
org\.godhuli\.rhipe\.RHMRHelper\.waitOutputThreads(RHMRHelper\.java:215)
at
org\.godhuli\.rhipe\.RHMRHelper\.mapRedFinished(RHMRHelper\.java:196)
at org\.godhuli\.rhipe\.RHMRMapper\.cleanup(RHMRMapper\.java:124)
at org\.godhuli\.rhipe\.RHMRMapper\.run(RHMRMapper\.java:65)
at
org\.apache\.hadoop\.mapred\.MapTask\.runNewMapper(MapTask\.java:621)
at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:305)
at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201012091620_0009_m_000001"
TASK_ATTEMPT_ID="attempt_201012091620_0009_m_000001_0"
START_TIME="1291919202292"
TRACKER_NAME="tracker_ip-192-168-1-67\.local:localhost/127\.0\.0\.1:56007"
HTTP_PORT="50060" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201012091620_0009_m_000001"
TASK_ATTEMPT_ID="attempt_201012091620_0009_m_000001_0"
TASK_STATUS="FAILED" FINISH_TIME="1291919207297"
HOSTNAME="ip-192-168-1-67\.local" ERROR="java\.lang\.RuntimeException:
RHMRMapRed\.waitOutputThreads(): subprocess failed with code 127
at
org\.godhuli\.rhipe\.RHMRHelper\.waitOutputThreads(RHMRHelper\.java:215)
at
org\.godhuli\.rhipe\.RHMRHelper\.mapRedFinished(RHMRHelper\.java:196)
at org\.godhuli\.rhipe\.RHMRMapper\.cleanup(RHMRMapper\.java:124)
at org\.godhuli\.rhipe\.RHMRMapper\.run(RHMRMapper\.java:65)
at
org\.apache\.hadoop\.mapred\.MapTask\.runNewMapper(MapTask\.java:621)
at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:305)
at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201012091620_0009_m_000000"
TASK_ATTEMPT_ID="attempt_201012091620_0009_m_000000_1"
START_TIME="1291919205225"
TRACKER_NAME="tracker_ip-192-168-1-67\.local:localhost/127\.0\.0\.1:56007"
HTTP_PORT="50060" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201012091620_0009_m_000000"
TASK_ATTEMPT_ID="attempt_201012091620_0009_m_000000_1"
TASK_STATUS="FAILED" FINISH_TIME="1291919210101"
HOSTNAME="ip-192-168-1-67\.local" ERROR="java\.lang\.RuntimeException:
RHMRMapRed\.waitOutputThreads(): subprocess failed with code 127
at
org\.godhuli\.rhipe\.RHMRHelper\.waitOutputThreads(RHMRHelper\.java:215)
at
org\.godhuli\.rhipe\.RHMRHelper\.mapRedFinished(RHMRHelper\.java:196)
at org\.godhuli\.rhipe\.RHMRMapper\.cleanup(RHMRMapper\.java:124)
at org\.godhuli\.rhipe\.RHMRMapper\.run(RHMRMapper\.java:65)
at
org\.apache\.hadoop\.mapred\.MapTask\.runNewMapper(MapTask\.java:621)
at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:305)
at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201012091620_0009_m_000001"
TASK_ATTEMPT_ID="attempt_201012091620_0009_m_000001_1"
START_TIME="1291919208248"
TRACKER_NAME="tracker_ip-192-168-1-67\.local:localhost/127\.0\.0\.1:56007"
HTTP_PORT="50060" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201012091620_0009_m_000001"
TASK_ATTEMPT_ID="attempt_201012091620_0009_m_000001_1"
TASK_STATUS="FAILED" FINISH_TIME="1291919213291"
HOSTNAME="ip-192-168-1-67\.local" ERROR="java\.lang\.RuntimeException:
RHMRMapRed\.waitOutputThreads(): subprocess failed with code 127
at
org\.godhuli\.rhipe\.RHMRHelper\.waitOutputThreads(RHMRHelper\.java:215)
at
org\.godhuli\.rhipe\.RHMRHelper\.mapRedFinished(RHMRHelper\.java:196)
at org\.godhuli\.rhipe\.RHMRMapper\.cleanup(RHMRMapper\.java:124)
at org\.godhuli\.rhipe\.RHMRMapper\.run(RHMRMapper\.java:65)
at
org\.apache\.hadoop\.mapred\.MapTask\.runNewMapper(MapTask\.java:621)
at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:305)
at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201012091620_0009_m_000000"
TASK_ATTEMPT_ID="attempt_201012091620_0009_m_000000_2"
START_TIME="1291919211252"
TRACKER_NAME="tracker_ip-192-168-1-67\.local:localhost/127\.0\.0\.1:56007"
HTTP_PORT="50060" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201012091620_0009_m_000000"
TASK_ATTEMPT_ID="attempt_201012091620_0009_m_000000_2"
TASK_STATUS="FAILED" FINISH_TIME="1291919216328"
HOSTNAME="ip-192-168-1-67\.local" ERROR="java\.lang\.RuntimeException:
RHMRMapRed\.waitOutputThreads(): subprocess failed with code 127
at
org\.godhuli\.rhipe\.RHMRHelper\.waitOutputThreads(RHMRHelper\.java:215)
at
org\.godhuli\.rhipe\.RHMRHelper\.mapRedFinished(RHMRHelper\.java:196)
at org\.godhuli\.rhipe\.RHMRMapper\.cleanup(RHMRMapper\.java:124)
at org\.godhuli\.rhipe\.RHMRMapper\.run(RHMRMapper\.java:65)
at
org\.apache\.hadoop\.mapred\.MapTask\.runNewMapper(MapTask\.java:621)
at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:305)
at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201012091620_0009_m_000001"
TASK_ATTEMPT_ID="attempt_201012091620_0009_m_000001_2"
START_TIME="1291919214262"
TRACKER_NAME="tracker_ip-192-168-1-67\.local:localhost/127\.0\.0\.1:56007"
HTTP_PORT="50060" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201012091620_0009_m_000001"
TASK_ATTEMPT_ID="attempt_201012091620_0009_m_000001_2"
TASK_STATUS="FAILED" FINISH_TIME="1291919219292"
HOSTNAME="ip-192-168-1-67\.local" ERROR="java\.lang\.RuntimeException:
RHMRMapRed\.waitOutputThreads(): subprocess failed with code 127
at
org\.godhuli\.rhipe\.RHMRHelper\.waitOutputThreads(RHMRHelper\.java:215)
at
org\.godhuli\.rhipe\.RHMRHelper\.mapRedFinished(RHMRHelper\.java:196)
at org\.godhuli\.rhipe\.RHMRMapper\.cleanup(RHMRMapper\.java:124)
at org\.godhuli\.rhipe\.RHMRMapper\.run(RHMRMapper\.java:65)
at
org\.apache\.hadoop\.mapred\.MapTask\.runNewMapper(MapTask\.java:621)
at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:305)
at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201012091620_0009_m_000000"
TASK_ATTEMPT_ID="attempt_201012091620_0009_m_000000_3"
START_TIME="1291919217269"
TRACKER_NAME="tracker_ip-192-168-1-67\.local:localhost/127\.0\.0\.1:56007"
HTTP_PORT="50060" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201012091620_0009_m_000000"
TASK_ATTEMPT_ID="attempt_201012091620_0009_m_000000_3"
TASK_STATUS="FAILED" FINISH_TIME="1291919222276"
HOSTNAME="ip-192-168-1-67\.local" ERROR="java\.lang\.RuntimeException:
RHMRMapRed\.waitOutputThreads(): subprocess failed with code 127
at
org\.godhuli\.rhipe\.RHMRHelper\.waitOutputThreads(RHMRHelper\.java:215)
at
org\.godhuli\.rhipe\.RHMRHelper\.mapRedFinished(RHMRHelper\.java:196)
at org\.godhuli\.rhipe\.RHMRMapper\.cleanup(RHMRMapper\.java:124)
at org\.godhuli\.rhipe\.RHMRMapper\.run(RHMRMapper\.java:65)
at
org\.apache\.hadoop\.mapred\.MapTask\.runNewMapper(MapTask\.java:621)
at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:305)
at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
" .
Task TASKID="task_201012091620_0009_m_000000" TASK_TYPE="MAP"
TASK_STATUS="FAILED" FINISH_TIME="1291919222276"
ERROR="java\.lang\.RuntimeException: RHMRMapRed\.waitOutputThreads():

subprocess failed with code 127
at

org\.godhuli\.rhipe\.RHMRHelper\.waitOutputThreads(RHMRHelper\.java:215)
at
org\.godhuli\.rhipe\.RHMRHelper\.mapRedFinished(RHMRHelper\.java:196)
at org\.godhuli\.rhipe\.RHMRMapper\.cleanup(RHMRMapper\.java:124)
at org\.godhuli\.rhipe\.RHMRMapper\.run(RHMRMapper\.java:65)
at
org\.apache\.hadoop\.mapred\.MapTask\.runNewMapper(MapTask\.java:621)
at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:305)
at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
" TASK_ATTEMPT_ID="" .
Task TASKID="task_201012091620_0009_m_000002" TASK_TYPE="CLEANUP"
START_TIME="1291919223249" SPLITS="" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201012091620_0009_m_000001"
TASK_ATTEMPT_ID="attempt_201012091620_0009_m_000001_3"
START_TIME="1291919220280"
TRACKER_NAME="tracker_ip-192-168-1-67\.local:localhost/127\.0\.0\.1:56007"
HTTP_PORT="50060" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201012091620_0009_m_000001"
TASK_ATTEMPT_ID="attempt_201012091620_0009_m_000001_3"
TASK_STATUS="KILLED" FINISH_TIME="1291919225211"
HOSTNAME="ip-192-168-1-67\.local" ERROR="java\.lang\.RuntimeException:
RHMRMapRed\.waitOutputThreads(): subprocess failed with code 127
at
org\.godhuli\.rhipe\.RHMRHelper\.waitOutputThreads(RHMRHelper\.java:215)
at
org\.godhuli\.rhipe\.RHMRHelper\.mapRedFinished(RHMRHelper\.java:196)
at org\.godhuli\.rhipe\.RHMRMapper\.cleanup(RHMRMapper\.java:124)
at org\.godhuli\.rhipe\.RHMRMapper\.run(RHMRMapper\.java:65)
at
org\.apache\.hadoop\.mapred\.MapTask\.runNewMapper(MapTask\.java:621)
at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:305)
at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
" .
MapAttempt TASK_TYPE="CLEANUP" TASKID="task_201012091620_0009_m_000002"
TASK_ATTEMPT_ID="attempt_201012091620_0009_m_000002_0"
START_TIME="1291919223274"
TRACKER_NAME="tracker_ip-192-168-1-67\.local:localhost/127\.0\.0\.1:56007"
HTTP_PORT="50060" .
MapAttempt TASK_TYPE="CLEANUP" TASKID="task_201012091620_0009_m_000002"
TASK_ATTEMPT_ID="attempt_201012091620_0009_m_000002_0"
TASK_STATUS="SUCCESS" FINISH_TIME="1291919224621"
HOSTNAME="/default-rack/ip-192-168-1-67\.local" STATE_STRING="cleanup"
COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
Task TASKID="task_201012091620_0009_m_000002" TASK_TYPE="CLEANUP"
TASK_STATUS="SUCCESS" FINISH_TIME="1291919226258"
COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
Job JOBID="job_201012091620_0009" FINISH_TIME="1291919226258"
JOB_STATUS="FAILED" FINISHED_MAPS="0" FINISHED_REDUCES="0" .

On 12/09/2010 01:21 PM, Saptarshi Guha wrote:

[Yin Zihang]

got this

WARNING: ignoring environment value of R_HOME

*** buffer overflow detected ***:
/usr/local/lib64/R/library/Rhipe/libs/imperious.so terminated

======= Backtrace: =========
/lib/libc.so.6(__fortify_fail+0x37)[0x7fbadb7b0217]
/lib/libc.so.6(+0xfe0d0)[0x7fbadb7af0d0]
/lib/libc.so.6(+0xfd7cb)[0x7fbadb7ae7cb]
/lib/libc.so.6(__snprintf_chk+0x7a)[0x7fbadb7ae69a]
/usr/local/lib64/R/library/Rhipe/libs/imperious.so[0x411d95]
/lib/libc.so.6(__libc_start_main+0xfd)[0x7fbadb6cfc4d]
/usr/local/lib64/R/library/Rhipe/libs/imperious.so[0x4061f9]

======= Memory map: ========
00400000-00416000 r-xp 00000000 08:01 394780
/usr/local/lib64/R/library/Rhipe/libs/imperious.so
00615000-00616000 r--p 00015000 08:01 394780
/usr/local/lib64/R/library/Rhipe/libs/imperious.so
00616000-00617000 rw-p 00016000 08:01 394780
/usr/local/lib64/R/library/Rhipe/libs/imperious.so

00b33000-00b54000 rw-p 00b33000 00:00 0
[heap]
7fbadab13000-7fbadab51000 r-xp 00000000 08:01 213121
/lib/libncurses.so.5.7
7fbadab51000-7fbadad51000 ---p 0003e000 08:01 213121
/lib/libncurses.so.5.7
7fbadad51000-7fbadad55000 r--p 0003e000 08:01 213121
/lib/libncurses.so.5.7
7fbadad55000-7fbadad56000 rw-p 00042000 08:01 213121
/lib/libncurses.so.5.7
7fbadad56000-7fbadad58000 r-xp 00000000 08:01 213101
/lib/libdl-2.11.1.so
7fbadad58000-7fbadaf58000 ---p 00002000 08:01 213101
/lib/libdl-2.11.1.so
7fbadaf58000-7fbadaf59000 r--p 00002000 08:01 213101
/lib/libdl-2.11.1.so
7fbadaf59000-7fbadaf5a000 rw-p 00003000 08:01 213101
/lib/libdl-2.11.1.so
7fbadaf5a000-7fbadaf93000 r-xp 00000000 08:01 213154
/lib/libreadline.so.6.1
7fbadaf93000-7fbadb192000 ---p 00039000 08:01 213154
/lib/libreadline.so.6.1
7fbadb192000-7fbadb194000 r--p 00038000 08:01 213154
/lib/libreadline.so.6.1
7fbadb194000-7fbadb19a000 rw-p 0003a000 08:01 213154
/lib/libreadline.so.6.1
7fbadb19a000-7fbadb19b000 rw-p 7fbadb19a000 00:00 0
7fbadb19b000-7fbadb286000 r-xp 00000000 08:01 32769
/usr/lib/libgfortran.so.3.0.0
7fbadb286000-7fbadb485000 ---p 000eb000 08:01 32769
/usr/lib/libgfortran.so.3.0.0
7fbadb485000-7fbadb486000 r--p 000ea000 08:01 32769
/usr/lib/libgfortran.so.3.0.0
7fbadb486000-7fbadb487000 rw-p 000eb000 08:01 32769
/usr/lib/libgfortran.so.3.0.0
7fbadb487000-7fbadb488000 rw-p 7fbadb487000 00:00 0
7fbadb488000-7fbadb4b0000 r-xp 00000000 08:01 386725
/usr/local/lib64/R/lib/libRblas.so
7fbadb4b0000-7fbadb6af000 ---p 00028000 08:01 386725
/usr/local/lib64/R/lib/libRblas.so
7fbadb6af000-7fbadb6b0000 r--p 00027000 08:01 386725
/usr/local/lib64/R/lib/libRblas.so
7fbadb6b0000-7fbadb6b1000 rw-p 00028000 08:01 386725
/usr/local/lib64/R/lib/libRblas.so
7fbadb6b1000-7fbadb82b000 r-xp 00000000 08:01 213093
/lib/libc-2.11.1.so
7fbadb82b000-7fbadba2a000 ---p 0017a000 08:01 213093
/lib/libc-2.11.1.so
7fbadba2a000-7fbadba2e000 r--p 00179000 08:01 213093
/lib/libc-2.11.1.so
7fbadba2e000-7fbadba2f000 rw-p 0017d000 08:01 213093
/lib/libc-2.11.1.so
7fbadba2f000-7fbadba34000 rw-p 7fbadba2f000 00:00 0
7fbadba34000-7fbadba4c000 r-xp 00000000 08:01 213153
/lib/libpthread-2.11.1.so
7fbadba4c000-7fbadbc4b000 ---p 00018000 08:01 213153
/lib/libpthread-2.11.1.so
7fbadbc4b000-7fbadbc4c000 r--p 00017000 08:01 213153
/lib/libpthread-2.11.1.so
7fbadbc4c000-7fbadbc4d000 rw-p 00018000 08:01 213153
/lib/libpthread-2.11.1.so
7fbadbc4d000-7fbadbc51000 rw-p 7fbadbc4d000 00:00 0
7fbadbc51000-7fbadbc67000 r-xp 00000000 08:01 213111
/lib/libgcc_s.so.1
7fbadbc67000-7fbadbe66000 ---p 00016000 08:01 213111
/lib/libgcc_s.so.1
7fbadbe66000-7fbadbe67000 r--p 00015000 08:01 213111
/lib/libgcc_s.so.1
7fbadbe67000-7fbadbe68000 rw-p 00016000 08:01 213111
/lib/libgcc_s.so.1
7fbadbe68000-7fbadbeea000 r-xp 00000000 08:01 213119
/lib/libm-2.11.1.so
7fbadbeea000-7fbadc0e9000 ---p 00082000 08:01 213119
/lib/libm-2.11.1.so
7fbadc0e9000-7fbadc0ea000 r--p 00081000 08:01 213119
/lib/libm-2.11.1.so
7fbadc0ea000-7fbadc0eb000 rw-p 00082000 08:01 213119
/lib/libm-2.11.1.so
7fbadc0eb000-7fbadc1e1000 r-xp 00000000 08:01 239792
/usr/lib/libstdc++.so.6.0.13
7fbadc1e1000-7fbadc3e1000 ---p 000f6000 08:01 239792
/usr/lib/libstdc++.so.6.0.13
7fbadc3e1000-7fbadc3e8000 r--p 000f6000 08:01 239792
/usr/lib/libstdc++.so.6.0.13
7fbadc3e8000-7fbadc3ea000 rw-p 000fd000 08:01 239792
/usr/lib/libstdc++.so.6.0.13
7fbadc3ea000-7fbadc3ff000 rw-p 7fbadc3ea000 00:00 0
7fbadc3ff000-7fbadc6d5000 r-xp 00000000 08:01 386729
/usr/local/lib64/R/lib/libR.so
7fbadc6d5000-7fbadc8d5000 ---p 002d6000 08:01 386729
/usr/local/lib64/R/lib/libR.so
7fbadc8d5000-7fbadc8db000 r--p 002d6000 08:01 386729
/usr/local/lib64/R/lib/libR.so
7fbadc8db000-7fbadc8eb000 rw-p 002dc000 08:01 386729
/usr/local/lib64/R/lib/libR.so
7fbadc8eb000-7fbadc987000 rw-p 7fbadc8eb000 00:00 0
7fbadc987000-7fbadca60000 r-xp 00000000 08:01 281421
/usr/local/lib/libprotobuf.so.6.0.0
7fbadca60000-7fbadcc5f000 ---p 000d9000 08:01 281421
/usr/local/lib/libprotobuf.so.6.0.0
7fbadcc5f000-7fbadcc63000 r--p 000d8000 08:01 281421
/usr/local/lib/libprotobuf.so.6.0.0
7fbadcc63000-7fbadcc65000 rw-p 000dc000 08:01 281421
/usr/local/lib/libprotobuf.so.6.0.0
7fbadcc65000-7fbadcc66000 rw-p 7fbadcc65000 00:00 0
7fbadcc66000-7fbadcc86000 r-xp 00000000 08:01 213083
/lib/ld-2.11.1.so
7fbadce78000-7fbadce7f000 rw-p 7fbadce78000 00:00 0
7fbadce84000-7fbadce86000 rw-p 7fbadce84000 00:00 0
7fbadce86000-7fbadce87000 r--p 00020000 08:01 213083
/lib/ld-2.11.1.so
7fbadce87000-7fbadce88000 rw-p 00021000 08:01 213083
/lib/ld-2.11.1.so
7fbadce88000-7fbadce89000 rw-p 7fbadce88000 00:00 0
7fffe4e73000-7fffe4e88000 rw-p 7ffffffea000 00:00 0
[stack]
7fffe4ffe000-7fffe4fff000 r-xp 7fffe4ffe000 00:00 0
[vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
[vsyscall]
Aborted

[Yin Zihang]

It works

R version 2.12.0 (2010-10-15) ?
Copyright (C) 2010 The R Foundation for Statistical Computing
ISBN 3-900051-07-0
"Platform: x86_64-unknown-linux-gnu
(64-bit)

�R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

� Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

�Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

[Saptarshi Guha]

Now, in R,
R
library(Rhipe)
rhinit(TRUE,TRUE,buglevel=2000)

rhoptions(runner = c("R","CMD", rhoptions()$runner))

z=rhmr(map=expression(), N=3,
inout=c("lapp","seq"),ifolder="/tmp/x",mapred=list(mapred.job.tracker='local'))
rhex(z)

[Yin Zihang]

Running: /usr/local/hadoop-0.20.2/bin/hadoop jar
/usr/local/lib64/R/library/Rhipe/java/Rhipe.jar org.godhuli.rhipe.RHMR
/tmp/RtmpqnwAtH/rhipe2bbf0b0d
----------------------------------------
10/12/09 18:50:57 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=

10/12/09 18:50:58 INFO rhipe.RHMR: Tracking URL ----> http://localhost:8080/
10/12/09 18:50:58 INFO mapred.JobClient: Running job: job_local_0001
10/12/09 18:50:58 INFO mapred.MapTask: io.sort.mb = 100
10/12/09 18:50:58 INFO mapred.MapTask: data buffer = 79691776/99614720
10/12/09 18:50:58 INFO mapred.MapTask: record buffer = 262144/327680
mapred.input.file == null
10/12/09 18:50:58 INFO rhipe.RHMRHelper: Mapper:Started external
program:R CMD /usr/local/lib64/R/library/Rhipe/libs/imperious.so --slave
--silent --vanilla --max-ppsize=100000 --max-nsize=1G
10/12/09 18:50:58 INFO rhipe.RHMRHelper: Mapper:Started Error Thread
10/12/09 18:50:58 INFO rhipe.RHMRHelper: Mapper:Started Output Thread

*** buffer overflow detected ***:
/usr/local/lib64/R/library/Rhipe/libs/imperious.so terminated
======= Backtrace: =========

/lib/libc.so.6(__fortify_fail+0x37)[0x7fd56650e217]
/lib/libc.so.6(+0xfe0d0)[0x7fd56650d0d0]
/lib/libc.so.6(+0xfd7cb)[0x7fd56650c7cb]
/lib/libc.so.6(__snprintf_chk+0x7a)[0x7fd56650c69a]
/usr/local/lib64/R/library/Rhipe/libs/imperious.so[0x411d95]
/lib/libc.so.6(__libc_start_main+0xfd)[0x7fd56642dc4d]
/usr/local/lib64/R/library/Rhipe/libs/imperious.so[0x4061f9]

======= Memory map: ========
00400000-00416000 r-xp 00000000 08:01 394780
/usr/local/lib64/R/library/Rhipe/libs/imperious.so
00615000-00616000 r--p 00015000 08:01 394780
/usr/local/lib64/R/library/Rhipe/libs/imperious.so
00616000-00617000 rw-p 00016000 08:01 394780
/usr/local/lib64/R/library/Rhipe/libs/imperious.so

010a8000-010c9000 rw-p 010a8000 00:00 0
[heap]
7fd565871000-7fd5658af000 r-xp 00000000 08:01 213121
/lib/libncurses.so.5.7
7fd5658af000-7fd565aaf000 ---p 0003e000 08:01 213121
/lib/libncurses.so.5.7
7fd565aaf000-7fd565ab3000 r--p 0003e000 08:01 213121
/lib/libncurses.so.5.7
7fd565ab3000-7fd565ab4000 rw-p 00042000 08:01 213121
/lib/libncurses.so.5.7
7fd565ab4000-7fd565ab6000 r-xp 00000000 08:01 213101
/lib/libdl-2.11.1.so
7fd565ab6000-7fd565cb6000 ---p 00002000 08:01 213101
/lib/libdl-2.11.1.so
7fd565cb6000-7fd565cb7000 r--p 00002000 08:01 213101
/lib/libdl-2.11.1.so
7fd565cb7000-7fd565cb8000 rw-p 00003000 08:01 213101
/lib/libdl-2.11.1.so
7fd565cb8000-7fd565cf1000 r-xp 00000000 08:01 213154
/lib/libreadline.so.6.1
7fd565cf1000-7fd565ef0000 ---p 00039000 08:01 213154
/lib/libreadline.so.6.1
7fd565ef0000-7fd565ef2000 r--p 00038000 08:01 213154
/lib/libreadline.so.6.1
7fd565ef2000-7fd565ef8000 rw-p 0003a000 08:01 213154
/lib/libreadline.so.6.1
7fd565ef8000-7fd565ef9000 rw-p 7fd565ef8000 00:00 0
7fd565ef9000-7fd565fe4000 r-xp 00000000 08:01 32769
/usr/lib/libgfortran.so.3.0.0
7fd565fe4000-7fd5661e3000 ---p 000eb000 08:01 32769
/usr/lib/libgfortran.so.3.0.0
7fd5661e3000-7fd5661e4000 r--p 000ea000 08:01 32769
/usr/lib/libgfortran.so.3.0.0
7fd5661e4000-7fd5661e5000 rw-p 000eb000 08:01 32769
/usr/lib/libgfortran.so.3.0.0
7fd5661e5000-7fd5661e6000 rw-p 7fd5661e5000 00:00 0
7fd5661e6000-7fd56620e000 r-xp 00000000 08:01 386725
/usr/local/lib64/R/lib/libRblas.so
7fd56620e000-7fd56640d000 ---p 00028000 08:01 386725
/usr/local/lib64/R/lib/libRblas.so
7fd56640d000-7fd56640e000 r--p 00027000 08:01 386725
/usr/local/lib64/R/lib/libRblas.so
7fd56640e000-7fd56640f000 rw-p 00028000 08:01 386725
/usr/local/lib64/R/lib/libRblas.so
7fd56640f000-7fd566589000 r-xp 00000000 08:01 213093
/lib/libc-2.11.1.so
7fd566589000-7fd566788000 ---p 0017a000 08:01 213093
/lib/libc-2.11.1.so
7fd566788000-7fd56678c000 r--p 00179000 08:01 213093
/lib/libc-2.11.1.so
7fd56678c000-7fd56678d000 rw-p 0017d000 08:01 213093
/lib/libc-2.11.1.so
7fd56678d000-7fd566792000 rw-p 7fd56678d000 00:00 0
7fd566792000-7fd5667aa000 r-xp 00000000 08:01 213153
/lib/libpthread-2.11.1.so
7fd5667aa000-7fd5669a9000 ---p 00018000 08:01 213153
/lib/libpthread-2.11.1.so
7fd5669a9000-7fd5669aa000 r--p 00017000 08:01 213153
/lib/libpthread-2.11.1.so
7fd5669aa000-7fd5669ab000 rw-p 00018000 08:01 213153
/lib/libpthread-2.11.1.so
7fd5669ab000-7fd5669af000 rw-p 7fd5669ab000 00:00 0
7fd5669af000-7fd5669c5000 r-xp 00000000 08:01 213111
/lib/libgcc_s.so.1
7fd5669c5000-7fd566bc4000 ---p 00016000 08:01 213111
/lib/libgcc_s.so.1
7fd566bc4000-7fd566bc5000 r--p 00015000 08:01 213111
/lib/libgcc_s.so.1
7fd566bc5000-7fd566bc6000 rw-p 00016000 08:01 213111
/lib/libgcc_s.so.1
7fd566bc6000-7fd566c48000 r-xp 00000000 08:01 213119
/lib/libm-2.11.1.so
7fd566c48000-7fd566e47000 ---p 00082000 08:01 213119
/lib/libm-2.11.1.so
7fd566e47000-7fd566e48000 r--p 00081000 08:01 213119
/lib/libm-2.11.1.so
7fd566e48000-7fd566e49000 rw-p 00082000 08:01 213119
/lib/libm-2.11.1.so
7fd566e49000-7fd566f3f000 r-xp 00000000 08:01 239792
/usr/lib/libstdc++.so.6.0.13
7fd566f3f000-7fd56713f000 ---p 000f6000 08:01 239792
/usr/lib/libstdc++.so.6.0.13
7fd56713f000-7fd567146000 r--p 000f6000 08:01 239792
/usr/lib/libstdc++.so.6.0.13
7fd567146000-7fd567148000 rw-p 000fd000 08:01 239792
/usr/lib/libstdc++.so.6.0.13
7fd567148000-7fd56715d000 rw-p 7fd567148000 00:00 0
7fd56715d000-7fd567433000 r-xp 00000000 08:01 386729
/usr/local/lib64/R/lib/libR.so
7fd567433000-7fd567633000 ---p 002d6000 08:01 386729
/usr/local/lib64/R/lib/libR.so
7fd567633000-7fd567639000 r--p 002d6000 08:01 386729
/usr/local/lib64/R/lib/libR.so
7fd567639000-7fd567649000 rw-p 002dc000 08:01 386729
/usr/local/lib64/R/lib/libR.so
7fd567649000-7fd5676e5000 rw-p 7fd567649000 00:00 0
7fd5676e5000-7fd5677be000 r-xp 00000000 08:01 281421
/usr/local/lib/libprotobuf.so.6.0.0
7fd5677be000-7fd5679bd000 ---p 000d9000 08:01 281421
/usr/local/lib/libprotobuf.so.6.0.0
7fd5679bd000-7fd5679c1000 r--p 000d8000 08:01 281421
/usr/local/lib/libprotobuf.so.6.0.0
7fd5679c1000-7fd5679c3000 rw-p 000dc000 08:01 281421
/usr/local/lib/libprotobuf.so.6.0.0
7fd5679c3000-7fd5679c4000 rw-p 7fd5679c3000 00:00 0
7fd5679c4000-7fd5679e4000 r-xp 00000000 08:01 213083
/lib/ld-2.11.1.so
7fd567bd5000-7fd567bdd000 rw-p 7fd567bd5000 00:00 0
7fd567be2000-7fd567be4000 rw-p 7fd567be2000 00:00 0
7fd567be4000-7fd567be5000 r--p 00020000 08:01 213083
/lib/ld-2.11.1.so
7fd567be5000-7fd567be6000 rw-p 00021000 08:01 213083
/lib/ld-2.11.1.so
7fd567be6000-7fd567be7000 rw-p 7fd567be6000 00:00 0
7fff6fbd0000-7fff6fbe7000 rw-p 7ffffffe8000 00:00 0
[stack]
710/12/09 18:50:58 INFO rhipe.RHMRHelper: Mapper:MROutputThread done
10/12/09 18:50:58 INFO rhipe.RHMRHelper: Mapper:MRErrorThread done

java.lang.RuntimeException: RHMRMapRed.waitOutputThreads(): subprocess
failed with code 134

at org.godhuli.rhipe.RHMRHelper.waitOutputThreads(RHMRHelper.java:215)
at org.godhuli.rhipe.RHMRHelper.mapRedFinished(RHMRHelper.java:196)
at org.godhuli.rhipe.RHMRMapper.cleanup(RHMRMapper.java:124)
at org.godhuli.rhipe.RHMRMapper.run(RHMRMapper.java:65)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at

org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
10/12/09 18:50:58 WARN mapred.LocalJobRunner: job_local_0001

java.lang.RuntimeException: RHMRMapRed.waitOutputThreads(): subprocess
failed with code 134

at org.godhuli.rhipe.RHMRHelper.waitOutputThreads(RHMRHelper.java:215)
at org.godhuli.rhipe.RHMRHelper.mapRedFinished(RHMRHelper.java:196)
at org.godhuli.rhipe.RHMRMapper.cleanup(RHMRMapper.java:124)
at org.godhuli.rhipe.RHMRMapper.run(RHMRMapper.java:65)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at

org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
10/12/09 18:50:59 INFO mapred.JobClient: map 0% reduce 0%
10/12/09 18:50:59 INFO mapred.JobClient: Job complete: job_local_0001
10/12/09 18:50:59 INFO mapred.JobClient: Counters: 0

result:256
Error in readBin(f1, "raw", f2) : invalid 'n' argument

> 10/12/09 18:51:04 INFO mapred.LocalJobRunner:

[Saptarshi Guha]

wonder why the sprintf is causing a problem.
Try this version
http://ml.stat.purdue.edu/rhipe/tmp/Rhipe_0.65.1.tar.gz

[Yin Zihang]

I think I am using this version. Anyway, I will try. How do I uninstall
the old one?

[Saptarshi Guha]

Oh the version numbers are the same. i' ifdefed out some snprintf lines.
R CMD INSTALL .... will overwrite

[Yin Zihang]

After I installed, I just run

R
library(Rhipe)
rhinit(TRUE,TRUE,buglevel=2000)

rhoptions(runner = c("R","CMD", rhoptions()$runner))

z=rhmr(map=expression(), N=3,
inout=c("lapp","seq"),ifolder="/tmp/x",mapred=list(mapred.job.tracker='local'))
rhex(z)

I got this

Running: /usr/local/hadoop-0.20.2/bin/hadoop jar
/usr/local/lib64/R/library/Rhipe/java/Rhipe.jar org.godhuli.rhipe.RHMR

/tmp/Rtmp77iKPn/rhipee2d14a0
----------------------------------------
10/12/09 19:20:36 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized

10/12/09 19:20:36 INFO rhipe.RHMR: Tracking URL ----> http://localhost:8080/
10/12/09 19:20:36 INFO mapred.JobClient: Running job: job_local_0002
10/12/09 19:20:36 INFO mapred.MapTask: io.sort.mb = 100
10/12/09 19:20:37 INFO mapred.MapTask: data buffer = 79691776/99614720
10/12/09 19:20:37 INFO mapred.MapTask: record buffer = 262144/327680
mapred.input.file == null
10/12/09 19:20:37 INFO rhipe.RHMRHelper: Mapper:Started external
program:R CMD R CMD /usr/local/lib64/R/library/Rhipe/libs/imperious.so

--slave --silent --vanilla --max-ppsize=100000 --max-nsize=1G

10/12/09 19:20:37 INFO rhipe.RHMRHelper: Mapper:Started Error Thread
10/12/09 19:20:37 INFO rhipe.RHMRHelper: Mapper:Started Output Thread
10/12/09 19:20:37 INFO rhipe.RHMRHelper: Mapper:MROutputThread done
10/12/09 19:20:37 INFO rhipe.RHMRHelper: Mapper:MRErrorThread done
10/12/09 19:20:37 INFO mapred.MapTask: Starting flush of map output
10/12/09 19:20:37 INFO mapred.TaskRunner:
Task:attempt_local_0002_m_000000_0 is done. And is in the process of
commiting
10/12/09 19:20:37 INFO mapred.LocalJobRunner:
10/12/09 19:20:37 INFO mapred.TaskRunner: Task
'attempt_local_0002_m_000000_0' done.
10/12/09 19:20:37 INFO mapred.MapTask: io.sort.mb = 100
10/12/09 19:20:37 INFO mapred.MapTask: data buffer = 79691776/99614720
10/12/09 19:20:37 INFO mapred.MapTask: record buffer = 262144/327680
mapred.input.file == null
10/12/09 19:20:37 INFO rhipe.RHMRHelper: Mapper:Started external
program:R CMD R CMD /usr/local/lib64/R/library/Rhipe/libs/imperious.so

--slave --silent --vanilla --max-ppsize=100000 --max-nsize=1G

10/12/09 19:20:37 INFO rhipe.RHMRHelper: Mapper:Started Error Thread
10/12/09 19:20:37 INFO rhipe.RHMRHelper: Mapper:Started Output Thread
10/12/09 19:20:37 INFO mapred.JobClient: map 100% reduce 0%
10/12/09 19:20:37 INFO rhipe.RHMRHelper: Mapper:MROutputThread done
10/12/09 19:20:37 INFO rhipe.RHMRHelper: Mapper:MRErrorThread done
10/12/09 19:20:37 INFO mapred.MapTask: Starting flush of map output
10/12/09 19:20:37 INFO mapred.TaskRunner:
Task:attempt_local_0002_m_000001_0 is done. And is in the process of
commiting
10/12/09 19:20:37 INFO mapred.LocalJobRunner:
10/12/09 19:20:37 INFO mapred.TaskRunner: Task
'attempt_local_0002_m_000001_0' done.
10/12/09 19:20:37 INFO mapred.LocalJobRunner:
10/12/09 19:20:37 INFO mapred.Merger: Merging 2 sorted segments
10/12/09 19:20:37 INFO mapred.Merger: Down to the last merge-pass, with
0 segments left of total size: 0 bytes
10/12/09 19:20:37 INFO mapred.LocalJobRunner:
10/12/09 19:20:37 INFO mapred.TaskRunner:
Task:attempt_local_0002_r_000000_0 is done. And is in the process of
commiting
10/12/09 19:20:37 INFO mapred.LocalJobRunner: reduce > reduce
10/12/09 19:20:37 INFO mapred.TaskRunner: Task
'attempt_local_0002_r_000000_0' done.
10/12/09 19:20:38 INFO mapred.JobClient: map 100% reduce 100%
10/12/09 19:20:38 INFO mapred.JobClient: Job complete: job_local_0002
10/12/09 19:20:38 INFO mapred.JobClient: Counters: 14
10/12/09 19:20:38 INFO mapred.JobClient: FileSystemCounters
10/12/09 19:20:38 INFO mapred.JobClient: FILE_BYTES_READ=2760004
10/12/09 19:20:38 INFO mapred.JobClient: HDFS_BYTES_READ=121410
10/12/09 19:20:38 INFO mapred.JobClient: FILE_BYTES_WRITTEN=123859
10/12/09 19:20:38 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=2882316
10/12/09 19:20:38 INFO mapred.JobClient: Map-Reduce Framework
10/12/09 19:20:38 INFO mapred.JobClient: Reduce input groups=0
10/12/09 19:20:38 INFO mapred.JobClient: Combine output records=0
10/12/09 19:20:38 INFO mapred.JobClient: Map input records=3
10/12/09 19:20:38 INFO mapred.JobClient: Reduce shuffle bytes=0
10/12/09 19:20:38 INFO mapred.JobClient: Reduce output records=0
10/12/09 19:20:38 INFO mapred.JobClient: Spilled Records=0
10/12/09 19:20:38 INFO mapred.JobClient: Map output bytes=0
10/12/09 19:20:38 INFO mapred.JobClient: Combine input records=0
10/12/09 19:20:38 INFO mapred.JobClient: Map output records=0
10/12/09 19:20:38 INFO mapred.JobClient: Reduce input records=0
result:256
$state
[1] TRUE

$counters
$counters$FileSystemCounters
FILE_BYTES_READ FILE_BYTES_WRITTEN HDFS_BYTES_READ
HDFS_BYTES_WRITTEN
2760004 123859 121410
2882316

$counters$`Map-Reduce Framework`
Combine input records Combine output records Map input records
0 0 3
Map output bytes Map output records Reduce input groups
0 0 0
Reduce input records Reduce output records Reduce shuffle bytes
0 0 0
Spilled Records
0

$counters$job_time
[1] 2.008

On 12/09/2010 01:48 PM, Saptarshi Guha wrote:

[Yin Zihang]

It works out. Thank you very much.

On 12/09/2010 01:48 PM, Saptarshi Guha wrote:

[Saptarshi Guha]

lovely. install that version on all the nodes and remove
mapred.job.tracker='local'

[Yin Zihang]

When I try to R CMD INSTALL Rhipe

it says Error in dyn.load(file, DLLpath = DLLpath, ...) :
unable to load shared object
'/usr/local/lib64/R/library/Rhipe/libs/Rhipe.so':
libprotobuf.so.6: cannot open shared object file: No such file or
directory
ERROR: loading failed
* removing ‘/usr/local/lib64/R/library/Rhipe’

On 12/09/2010 12:30 PM, Saptarshi Guha wrote:

> No, not in R, run it on your current machine and on the work nodes.
> paste it into the terminal - as one line without newlines
>
> R_HOME=/ln/meraki/custom/lib64/R/ RHIPEWHAT=0 RHIPELOGFILE="/tmp/x"
> rhipe_outputformat_class=foo rhipe_combiner=0
> /ln/meraki/custom/lib64/R/library/Rhipe/libs/imperious.so
>
>
> On Thu, Dec 9, 2010 at 9:26 AM, Yin Zihang<yinzh...@gmail.com> wrote:
>
>> for b) I don't understand what you want me to do/
>>
>> R_HOME=/ln/meraki/custom/lib64/R/ (I set this as my environment varaible?
>> right now I think this is /usr/local/lib64/R/)
>>
>>
>> what's the following command?

>>
>> RHIPEWHAT=0 RHIPELOGFILE="/tmp/x"
>>
>> rhipe_outputformat_class=foo rhipe_combiner=0
>>

>> /ln/meraki/custom/lib64/R/library/Rhipe/libs/imperious.so
>>
>> I just run this in the worknode's terminal. Should I run this in R?
>>
>> On 12/09/2010 12:17 PM, Saptarshi Guha wrote:
>>
>> R_HOME=/ln/meraki/custom/lib64/R/ RHIPEWHAT=0 RHIPELOGFILE="/tmp/x"
>> rhipe_outputformat_class=foo rhipe_combiner=0
>> /ln/meraki/custom/lib64/R/library/Rhipe/libs/imperious.so

[Saptarshi Guha]

run
system("ldd /usr/local/lib64/R/library/Rhipe/libs/Rhipe.so")
is anything missing?

[Yin Zihang]

Syntax error near "ldd ?

[Yin Zihang]

I think no Rhipe.so in that folder.

On 12/12/2010 05:38 PM, Saptarshi Guha wrote:

[Saptarshi Guha]

Works for me

system("ldd //ln/meraki/custom/lib64/R/library/Rhipe/libs/Rhipe.so ")
libprotobuf.so.6 => /ln/meraki/custom/lib/libprotobuf.so.6 (0x00002aec36c6a000)
libz.so.1 => /usr/lib64/libz.so.1 (0x00002aec36f85000)
libR.so => /ln/meraki/custom//lib64/R/lib/libR.so (0x00002aec37199000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002aec37733000)
libm.so.6 => /lib64/libm.so.6 (0x00002aec37a33000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002aec37cb6000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aec37ec5000)
libc.so.6 => /lib64/libc.so.6 (0x00002aec380e0000)
libRblas.so => /ln/meraki/custom//lib64/R/lib/libRblas.so (0x00002aec38437000)
libg2c.so.0 => /usr/lib64/libg2c.so.0 (0x00002aec38663000)
libreadline.so.5 => /usr/lib64/libreadline.so.5 (0x00002aec38884000)
libncurses.so.5 => /usr/lib64/libncurses.so.5 (0x00002aec38ac1000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002aec38d1f000)
/lib64/ld-linux-x86-64.so.2 (0x00000038aea00000)

(my path is different, confirm your path has the Rhipe.so)

[Yin Zihang]

how can I get the Rhipe.so?

I don't have that. I just install R. There is no Rhipe library there.

[Yin Zihang]

After I installed R, there is no Rhipe.so file in the library.
Then after input the R CMD INSTALL RHIPE

it says Error in dyn.load(file, DLLpath = DLLpath, ...) :
unable to load shared object
'/usr/local/lib64/R/library/Rhipe/libs/Rhipe.so':
libprotobuf.so.6: cannot open shared object file: No such file or directory
ERROR: loading failed
* removing ‘/usr/local/lib64/R/library/Rhipe’

But without installing Rhipe, how can I get that?

On 12/12/2010 05:38 PM, Saptarshi Guha wrote:

[Saptarshi Guha]

Not sure if the syntax is right but try

ldconfig -s

Or could be - p which lists all the directories ( or libraries ? I
can't remember ) in loaders path. Is there a protobuf I'm that output.

The surest way is to write a little code and link with protobuf and
them ldd.

I can be more helpful in hour or sos time