RHadoop (rmr2) job does not complete

[RK]

I installed RHadoop on Hortonworks sandbox 2.1. I used the instructions of:

- http://www.research.janahang.com/install-rhadoop-on-hortonworks-hdp-2-0/

- http://projects.revolutionanalytics.com/documents/rhadoop/rhadoopinstall/

 

I was facing the problem that MR-jobs did start but would not complete. The map progress on the terminal stayed at 0% (and in Ambari, it stuck at 5%).

 

I noticed that several other people were having the same problem and that they gave up:

- http://stackoverflow.com/questions/25753983/how-do-you-change-the-max-container-capability-in-hadoop-cluster
- http://stackoverflow.com/questions/26390290/hortonworks-data-platform-2-1-sandbox-unable-to-complete-a-very-simple-rhadoop

After several hours of struggling, I could finally fix it by changing memory settings. Especially the following resources were helpful:

- http://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/
- http://stackoverflow.com/questions/25753983/how-do-you-change-the-max-container-capability-in-hadoop-cluster
- https://groups.google.com/forum/#!topic/rhadoop/6HfzhBVMEhc
- http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.9.1/bk_installing_manually_book/content/rpm-chap1-11.html

The following settings worked for me:

- yarn.nodemanager.resource.memory-mb: 3072
- yarn.scheduler.minimum-allocation-mb: 512
- yarn.scheduler.maximum-allocation-mb: 3072
- yarn.nodemanager.vmem-pmem-ratio: 10
- mapreduce.map.memory.mb: 1024
- mapreduce.reduce.memory.mb: 1024
- yarn.app.mapreduce.am.resource.mb: 1024
- yarn.app.mapreduce.am.command-opts: -Xmx768m
- mapreduce.task.io.sort: 512
- mapreduce.map.java.opts: -Xmx768m
- mapreduce.reduce.java.opts: -Xmx768m

Notes:

- Before, I increased VirtualBox memory from 4096 to 5291

- When running R, I entered the following after loading rmr2:

    rmr.options(backend.parameters = list(
      hadoop = list(D = "mapreduce.map.memory.mb=1024", D = "mapreduce.reduce.memory.mb=1024")
    ))

I would like to know if my settings are optimal for HDP 2.1 Sandbox, or if further optimization is possible.

[Antonio Piccolboni]

Thanks for your report and list of links. These settings are complex for sure and not particularly friendly to non-java application like the ones we are running. In the latest rmr2 release we added a help entry, help(hadoop.settings), to try and collect information about them and how to set them, it's a first attempt and we'll use feedback to try and make progress. As far as your optimality question, it's way beyond what I am able to answer but I'd suggest that optimality depends on the application and even sometimes on the specific input. So my opinion is that it your question is ill-posed. Finally, the sandbox is a learning environment, before you start thinking about optimizing your application you should switch to a real cluster.