Using rhls, rhread within map expressions

[David]

Is it possible to use 'rhls' and 'rhread' within the mapper? I have
been successful in doing these in the console, and have checked file
permissions, executed shell commands (e.g. system("hadoop fs -ls /user/
testUser/testDir")), but have been unable to use these commands.

Could I be overlooking something? I am using version 0.65.1.

Thanks,

David

[Saptarshi Guha]

You're right. In the mapper, reducer, you can't even run
library(Rhipe), you shouldn't be able to at least.
And so no, rhread, rhls don't work. Only rhcounter, rhstatus, rhcollect.

[David]

Thanks - I tried loading Rhipe into the mapper, and discovered the joy
of protocol buffer collisions (or whatever it may be called). :)

I think I know what I should do, but I'd like to check:

Suppose that step A produces a bunch of files, basically the
serialized output. And many such sets are created, e.g. I have 100
input sets, each produces a unique directory of serialized output, and
the step A is actually a result of 100 jobs launched into Hadoop.

In step B, each of the 100 resulting directories of step A is used for
further processing.

Before scaling up, I was just producing 1 set, but that's a bundle of
a bunch of files, and was locally running rhread to produce a big list
object, which I then turned into a data.frame, and used for step B.
Now, however, I have a lot more such sets, and I want to either run
rhread within the mapper, or ... as I don't want to run this rhread
locally, I suppose this is where I learn the joys of reducers,
correct?

I am still getting my head around coding the reduce phase. In the 1
input set approach, I used lapply or ldply (from the plyr package) to
turn the list into the appropriate dataframe. In moving to a reduce
phase, instead of local aggregation, is there anything special to do
to ensure that the output data frame is generated with the correct
structure? Or, should I look into the combiner operations?

Thanks!

David

[David]

I think this issue is resolved - I just needed to stop doing local
aggregation and instead use reduces.

It occurred to me that I might hack my way through this by executing a
shell command to Rscript to launch Rhipe, run rhread, etc., but that's
bound to produce grief at some point. I'm not sure, but I suspect it
could potentially leave processes running on the compute nodes.

Thanks!

David