Running R/Rhipe Mapreduce jobs on Amazon elastic MapReduce

[bharath raj]

Hi,

We, in our project, are exploring the option of running mapreduce jobs
using Amazon elastic MapReduce on AWS. We already have these jobs
coded in R using Rhipe.But unfortunately couldn't find much help on
the Amazon elastic MapReduce page regarding jobs in R and Rhipe. Of
course I saw we can submit R jobs using Hadoop streaming work flow for
which we will need to have the mapper and reducer as seperate .R files
and that too completely coded in R.

The main problem I believe we will be facing in regards to running our
MR jobs on Amazon is that Rhipe (and R if it is not provided) will
have to be installed on cloud on all the instances(But the resource
allocation itself is dynamic,i.e.,we get a new instance every time and
hence would have to install Rhipe and its requisites every time).
Porting the entire MR from R to say Pig which is supported more on
Amazon elastic MapReduce would be very cumbersome and would prefer to
avoid that.

[Saptarshi Guha]

Hi

A user provided me with scripts to run Rhipe on EMR. Ill upload them over the weekend. Ill post an update to the mailing list.
Cheers
Sapsi

[bharath raj]

Hi,
Sorry my post above got submitted before I could finish it :(...

So please help with how to port R/Rhipe MR on AWS elastic MR if you
have done it before.

Thanks and Regards,
Bharath

[bharath raj]

Hi,

Thanks a lot for uploading the code to run RHIPE on EMR. If I am not
wrong the instructions you have given say how to start Rhipe on the
cluster and then start R and launch a job. I would like to know how to
launch the job from the AWS management console by creating a job flow
(using streaming utility). Or can I start the Rhipe cluster on command
line and then create a streaming job flow on the AWS management
console and launch it?

[Saptarshi Guha]

Hmm,
I've never done this before. Can't suggest anything here. Also, i've
never used the AWS management consoles to launch ajob.

[M Nakamura]

I would use an EC2 instance to control EMR cluster.

Masa Nakamura

On Mon, Aug 29, 2011 at 4:38 AM, bharath raj <bhara...@gmail.com> wrote:

[lei Li]

Dear authors,

I am new to the group, but i am really interested in using Rhipe in AWS EC2.   I searched the help document in Rhipe website but most of the links are not available any more. Are there some new scripts for the new version of Rhipe? Could you send me the scripts? leil...@gmail.com

Thanks,

Lei

在 2011年8月26日星期五UTC+10下午5时42分52秒，bharath raj写道：

[Saptarshi Guha]

The links are broken but here is some sample code. My recommendation is to use whirr.

here is a whirr.conf file

whirr.cluster-name=rhipetest
whirr.instance-templates=1 hadoop-jobtracker+hadoop-namenode,2 hadoop-datanode+hadoop-tasktracker
whirr.provider=aws-ec2
whirr.identity=${env:AWS_ACCESS_KEY_ID}
whirr.credential=${env:AWS_SECRET_ACCESS_KEY}
whirr.private-key-file=${sys:user.home}/.ssh/id_rsa
whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub
whirr.hardware-id=c3.xlarge
# rightscale-us-east/RightImage_Ubuntu_12.04_x64_v5.8.8.5.manifest.xml
whirr.image-id=us-east-1/ami-0769ee6e
whirr.location-id=us-east-1
# hadoop options
hadoop-mapreduce.mapred.child.ulimit=unlimited

Here is a script that you want to run across all the nodes(see whirr help for how to do this)

sudo apt-get install -y software-properties-common python-software-properties
sudo echo 'deb http://cran.fhcrc.org/bin/linux/ubuntu precise/' >  R.list
sudo cp R.list /etc/apt/sources.list.d/
sudo add-apt-repository -y ppa:webupd8team/java
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E084DAB9
sudo apt-get update
sudo apt-get -y install pkg-config
sudo apt-get -y install libprotobuf-dev
sudo apt-get install -y r-base-dev r-recommended r-cran-rodbc
sudo echo oracle-java7-installer shared/accepted-oracle-license-v1-1 select true | sudo /usr/bin/debconf-set-selections
sudo apt-get install -y oracle-java7-installer
sudo update-java-alternatives -s java-7-oracle
sudo apt-get install oracle-java7-set-default
sudo chmod -R 0777 /usr/local/lib/R/site-library
unset JAVA_HOME

R -e "install.packages('codetools',repos='http://cran.cnr.Berkeley.edu')"
R -e "install.packages('testthat',repos='http://cran.cnr.Berkeley.edu')"
R -e "install.packages('rJava',repos='http://cran.cnr.Berkeley.edu')"
sudo R CMD javareconf

export RHIPE_USE_CDH4=yes
export HADOOP_HOME=/usr
export HADOOP_LIBS=`hadoop classpath`
export HADOOP_CONF_DIR=/etc/hadoop/conf

you need to to also download Rhipe and install it on all the nodes too.

--

---
You received this message because you are subscribed to the Google Groups "rhipe" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rhipe+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.