Help using RMarkdown with RHadoop

28 views
Skip to first unread message

fishball

unread,
Sep 29, 2015, 1:46:53 AM9/29/15
to RHadoop

I am beginning to use RHadoop and have successfully run some simple mapreduce programs going line-by-line in R. I would now like to execute the same program as an R Markdown script that outputs to HTML or PDF.

However, I am getting the OutofMemoryError: Java heap space when I try to knit (run) any Rmd file. It does not matter how large the object I am passing to the DFS. 

Config-wise, I have a VM (8gb) with Ubuntu 14.04. I have already set my yarn-site.xml config to 4g RAM and set the RHadoop rmr.options("backend.parameters") to 4g for each of map and reduce parameters.

Again, I can run everything as individual commands but cannot knit the Rmd file.  This Java heap space error also occurs regardless if I use RStudio or the command line.

Is there some different config I need to setup for knitr and/or rmarkdown in conjunction with RHadoop/rmr? Thanks in advance.
Reply all
Reply to author
Forward
0 new messages