Hadoop not using the right number of reducers

135 views

Skip to first unread message

Debashis Das

unread,

Apr 22, 2013, 11:25:46 AM4/22/13

to rha...@googlegroups.com

I don’t have a reproducible example, just a question. I am running a MapReduce job (using a combiner) and I first ran it with stock Hadoop configuration and then with backend parameters option setting the number of reducers. I am aware that using the backend parameters to set number of mappers/reducers is not recommended and this option may not be available in the future versions of rmr. The problem is that I could never get it to use more than 1 reducer. Both approaches used 1 reducer. I have three unique keys as output from my map job and so I tried to force it to use three, which didn’t work. What am I doing wrong? Is this more of a Hadoop question than an rmr one? I know that 3 is not a high number of reduce input groups but expected Hadoop to use three. Even with a higher number of reduce groups I couldn’t get it to use more than one.

I am using version 2.0.2 of rmr and a virtual cluster with five nodes, 8-core, 16 GB RAM. No load on the cluster other than my job.

Thanks in advance!

Antonio Piccolboni

unread,

Apr 22, 2013, 2:36:28 PM4/22/13

to RHadoop Google Group

Hi,

it could be more of a hadoop question. Can you repro the problem with one of the examples that come with hadoop distros? Also, if you turn off the combiner, what happens? How many concurrent maps do you observe? Thanks

Antonio

--
post: rha...@googlegroups.com ||
unsubscribe: rhadoop+u...@googlegroups.com ||
web: https://groups.google.com/d/forum/rhadoop?hl=en-US
---
You received this message because you are subscribed to the Google Groups "RHadoop" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rhadoop+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Debashis Das

unread,

May 3, 2013, 3:46:34 PM5/3/13

to rha...@googlegroups.com, ant...@piccolboni.info

Hi Antonio,

I was waiting to run the same code on a physical MapR cluster using stock config to see how many reducers it uses. The same code that ran without any errors, with 10 different input datasets on a virtual cluster gave me the following error on the physical cluster right at the onset of the first MapReduce job:

13/05/03 15:06:29 INFO util.NativeCodeLoader: Loaded the native-hadoop library

13/05/03 15:06:29 INFO security.JniBasedUnixGroupsMapping: Using JniBasedUnixGroupsMapping for Group resolution

*** warn: MAPR_SUBNETS set to 192.168.26.13/32

packageJobJar: [/tmp/RtmpTLcthG/rmr-local-env8fb56ee32f0, /tmp/RtmpTLcthG/rmr-global-env8fb1479dae, /tmp/RtmpTLcthjob2336471312487051535.jar tmpDir=null

13/05/03 15:06:29 INFO fs.JobTrackerWatcher: Current running JobTracker is: ql1maprpr3.mi.corp.rockfin.com/192.168

13/05/03 15:06:30 WARN snappy.LoadSnappy: Snappy native library not loaded

13/05/03 15:06:30 INFO mapred.FileInputFormat: Total input paths to process : 1

13/05/03 15:06:30 INFO mapred.JobClient: Creating job's output directory at maprfs:/tmp/RtmpTLcthG/file8fb38a62fa7

13/05/03 15:06:30 INFO mapred.JobClient: Creating job's user history location directory at maprfs:/tmp/RtmpTLcthG/

13/05/03 15:06:31 INFO streaming.StreamJob: getLocalDirs(): [/tmp/mapr-hadoop/mapred/local]

13/05/03 15:06:31 INFO streaming.StreamJob: Running job: job_201303121101_0396

13/05/03 15:06:31 INFO streaming.StreamJob: To kill this job, run:

13/05/03 15:06:31 INFO streaming.StreamJob: /opt/mapr/hadoop/hadoop-0.20.2/bin/../bin/hadoop job -Dmapred.job.tra

13/05/03 15:06:31 INFO streaming.StreamJob: Tracking URL: http://ql1maprpr3.mi.corp.rockfin.com:50030/jobdetails.j

13/05/03 15:06:32 INFO streaming.StreamJob: map 0% reduce 0%

13/05/03 15:07:03 INFO streaming.StreamJob: map 100% reduce 100%

13/05/03 15:07:03 INFO streaming.StreamJob: To kill this job, run:

13/05/03 15:07:03 INFO streaming.StreamJob: /opt/mapr/hadoop/hadoop-0.20.2/bin/../bin/hadoop job -Dmapred.job.tra

13/05/03 15:07:03 INFO streaming.StreamJob: Tracking URL: http://ql1maprpr3.mi.corp.rockfin.com:50030/jobdetails.j

13/05/03 15:07:03 ERROR streaming.StreamJob: Job not successful. Error: NA

13/05/03 15:07:03 INFO streaming.StreamJob: killJob...

Streaming Command Failed!

Error in mr(map = map, reduce = reduce, combine = combine, in.folder = if (is.list(input)) { :

hadoop streaming failed with error code 1

Calls: mapreduce -> mr

Execution halted

Found 'Rscript does not exist' in the logs:

2013-05-03 15:10:50,769 INFO org.apache.hadoop.mapred.Child: JVM: jvm_201303121101_0397_m_205155044 pid: 31422

2013-05-03 15:10:51,070 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/mapr-hadoop/mapred/local/taskTracker/asingh2/jobcache/job_201303121101_0397/jars/rmr-local-env18e23ac907ef <- /tmp/mapr-hadoop/mapred/local/taskTracker/asingh2/jobcache/job_201303121101_0397/attempt_201303121101_0397_m_000000_1/work/rmr-local-env18e23ac907ef

2013-05-03 15:10:51,071 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/mapr-hadoop/mapred/local/taskTracker/asingh2/jobcache/job_201303121101_0397/jars/job.jar <- /tmp/mapr-hadoop/mapred/local/taskTracker/asingh2/jobcache/job_201303121101_0397/attempt_201303121101_0397_m_000000_1/work/job.jar

2013-05-03 15:10:51,072 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/mapr-hadoop/mapred/local/taskTracker/asingh2/jobcache/job_201303121101_0397/jars/rmr-streaming-map18e24735a88c <- /tmp/mapr-hadoop/mapred/local/taskTracker/asingh2/jobcache/job_201303121101_0397/attempt_201303121101_0397_m_000000_1/work/rmr-streaming-map18e24735a88c

2013-05-03 15:10:51,073 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/mapr-hadoop/mapred/local/taskTracker/asingh2/jobcache/job_201303121101_0397/jars/.job.jar.crc <- /tmp/mapr-hadoop/mapred/local/taskTracker/asingh2/jobcache/job_201303121101_0397/attempt_201303121101_0397_m_000000_1/work/.job.jar.crc

2013-05-03 15:10:51,074 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/mapr-hadoop/mapred/local/taskTracker/asingh2/jobcache/job_201303121101_0397/jars/rmr-global-env18e213a8df24 <- /tmp/mapr-hadoop/mapred/local/taskTracker/asingh2/jobcache/job_201303121101_0397/attempt_201303121101_0397_m_000000_1/work/rmr-global-env18e213a8df24

2013-05-03 15:10:51,104 INFO org.apache.hadoop.mapred.Child: Starting task attempt_201303121101_0397_m_000000_1

2013-05-03 15:10:51,105 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId=

2013-05-03 15:10:51,139 INFO org.apache.hadoop.mapreduce.util.ProcessTree: setsid exited with exit code 0

2013-05-03 15:10:51,146 WARN org.apache.hadoop.mapreduce.util.ProcfsBasedProcessTree: /proc/<pid>/status does not have information about swap space used(VmSwap). Can not track swap usage of a task.

2013-05-03 15:10:51,146 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.mapreduce.util.LinuxResourceCalculatorPlugin@4c53ab04

2013-05-03 15:10:51,493 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library not loaded

2013-05-03 15:10:51,599 ERROR org.apache.hadoop.streaming.PipeMapRed: configuration exception

java.io.FileNotFoundException: File /tmp/mapr-hadoop/mapred/local/taskTracker/asingh2/jobcache/job_201303121101_0397/attempt_201303121101_0397_m_000000_1/work/Rscript does not exist.

at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:395)

at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:257)

at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:738)

at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:713)

at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:185)

at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)

at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)

at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)

at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)

at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)

at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)

at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)

at org.apache.hadoop.mapred.Child$4.run(Child.java:270)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:396)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)

at org.apache.hadoop.mapred.Child.main(Child.java:264)

2013-05-03 15:10:51,612 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1

2013-05-03 15:10:51,626 WARN org.apache.hadoop.mapred.Child: Error running child

java.lang.RuntimeException: Error in configuring object

at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)

at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)

at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)

at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)

at org.apache.hadoop.mapred.Child$4.run(Child.java:270)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:396)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)

at org.apache.hadoop.mapred.Child.main(Child.java:264)

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)

... 9 more

Caused by: java.lang.RuntimeException: Error in configuring object

at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)

at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)

at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)

at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)

... 14 more

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)

... 17 more

Caused by: java.lang.RuntimeException: configuration exception

at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:228)

at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66)

... 22 more

Caused by: java.io.FileNotFoundException: File /tmp/mapr-hadoop/mapred/local/taskTracker/asingh2/jobcache/job_201303121101_0397/attempt_201303121101_0397_m_000000_1/work/Rscript does not exist.

at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:395)

at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:257)

at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:738)

at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:713)

at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:185)

... 23 more

2013-05-03 15:10:51,631 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task

I am clueless as to what is going on. I searched here, came across another post with similar issue:

https://groups.google.com/forum/#!msg/rhadoop/38JcBht8SiE/R6NRnPHKorMJ

I mentioned the Rscript error becaue you found that odd in the above thread, there could be other relevant errors in my logs. Based on this thread, it is more of a MapR issue.

Thanks in advance!!

Antonio Piccolboni

unread,

May 7, 2013, 1:09:08 PM5/7/13

to RHadoop Google Group

This error happens when R is not installed or not properly installed on one or more nodes. Rscript should be executable from any directory, which means installed into a directory in the PATH variable. Normally jobs are executed under a different user account than that of the person actually issuing the job, so you need to be sure that you make Rscript available to the right user, or every user, which is my usual approach. Also people sometimes configure only interactive shells changing the .bashrc file, but here we need to configure also non interactive or batch shells, using the .profile or .bash_profile files. I hope this helps

Antonio

Debashis Das

unread,

May 9, 2013, 2:58:37 PM5/9/13

to rha...@googlegroups.com, ant...@piccolboni.info

It does help, thanks a lot! This makes sense.

We have other issues going on with MapR (needed the latest update for M5) but this is good to know too.