Simple Examples?

[mtanquary]

I need a quick example for a presentation of Rhipe. I am able to run
the word count sample on the home page of Rhipe, but I'm not sure
about the results. I am running the latest Rhipe on the latest Hadoop
(CDH3beta). After running the sample, the /tmp/top seems to have just
a sequence file containing the lines of the text file with line
numbers (not counts). I may have missed something.

Also for the presentation, I am looking for something like a slide
showing architecture. If you have one all ready to go, I'd love to use
it. This is just a small group presentation exploring a new topic,
nothing formal.

Here is the code I used:

library(Rhipe)
rhinit()
m <- expression({
y <- strsplit(unlist(map.values)," ")
lapply(y,function(r) rhcollect(r,T))
})
r <- expression(
pre={
count=0
},
reduce={
count <- sum(as.numeric(unlist(reduce.values)),count)
},post={
rhcollect(reduce.key,count)
})


z=rhmr(map=m,reduce=r,comb=T,inout=c("text","sequence"),ifolder="/user/
hadoop/foo",ofolder="/tmp/
tof",mapred=list(mapred.job.tracker='local'))

rhex(z)

File contents:

line 1 of test
line 2 of test
rhipe wordcout test
this is rhipe sample
1 more line

Output:
result:256
$state
[1] TRUE

$counters
$counters$FileSystemCounters
FILE_BYTES_READ FILE_BYTES_WRITTEN HDFS_BYTES_READ
HDFS_BYTES_WRITTEN
920087 44266 43518
963957

$counters$combiner
bytesent
167

$counters$`Map-Reduce Framework`
Combine input records Combine output records Map input records
0 0 5
Map output bytes Map output records Reduce input groups
217 5 5
Reduce input records Reduce output records Reduce shuffle bytes
5 5 0
Spilled Records
10

$counters$job_time
[1] 2.014


Result says 256, but I'm not sure what that means.

Thanks for any guidance. If you have any other very quick examples
that might be helpful to present, that would be very much appreciated.

-M@

[Saptarshi Guha]

On Mon, Jan 31, 2011 at 3:11 PM, mtanquary <matt.t...@gmail.com> wrote:

I need a quick example for a presentation of Rhipe. I am able to run
the word count sample on the home page of Rhipe, but I'm not sure
about the results. I am running the latest Rhipe on the latest Hadoop
(CDH3beta). After running the sample, the /tmp/top seems to have just
a sequence file containing the lines of the text file with line
numbers (not counts). I may have missed something.

The code is wrong:

replace  y <- strsplit(unlist(map.values)," ") with  y <- unlist(strsplit(unlist(map.values)," "))

I can mail a slide later tonight.

The 'result is 256' is quite useless. I'll remove it one day.

Cheers

Saptarshi

(On github, RHIPE-Additions, you'll find examples using RHIPE to 'big' regression using biglm and stand alone)

 

[Saptarshi Guha]

If it's not too late, here are some slides.
Joy

[mtanquary]

Thank you Dr. Guha! This is great stuff!

-M@

On Feb 1, 12:22 pm, Saptarshi Guha <saptarshi.g...@gmail.com> wrote:
> If it's not too late, here are some slides.
> Joy
>

> On Mon, Jan 31, 2011 at 4:55 PM, Saptarshi Guha <saptarshi.g...@gmail.com>wrote:

>  notes_on_rhipe.pdf
> 1362KViewDownload