Runtine issues after combining several MR jobs

[Claudio Hartmann]

Hi all,

after a major refactoring in my project I'm facing massive runtime changes and I'm a little clueless what causes them.

For development I divided my project into two MR-jobs:

1.

The first just did some calculations in the map phase an stored the result into the HDFS, no reduce phase.

2.

The second job read the data of the first job, and joined them with some additional information for subsequent comparison calculations and passed the results to the reducers where the comparisons were calculated and saved to the HDFS.

After making sure all the results and intermediate results are correct, I refactored my project. The map phases of both MR-jobs were joined, such that the calculations of the first mapper were joined with the data of the second mapper in one run an directly passed to the reducer. I expected this to be faster than the first set up since it saves the reading and writing to HDFS between the two MR-jobs. But unfortunately the opposite happened. A job lasting 29min in the old set up now takes now takes 66min. The extract of the log below shows mapping and reducing is done quite fast, progress is reported every few seconds. But finally the after  map 100%  reduce 100% it takes 52min to finish the job and I have no clue why. 

Is it the cleaning that takes so much longer because of the intermediate results that have to be cleared now and may not reside in the HDFS between the MR-jobs?

I'm using rmr2-2.2.0, are there any known issues which could cause that behavior and an upgrade will promise some improvement?

Hopefully the information I gave is enough to understand and determine the cause of the slowdown.

Best,

Claudio

14/03/04 10:28:54 INFO streaming.StreamJob:  map 88%  reduce 29%

14/03/04 10:29:00 INFO streaming.StreamJob:  map 89%  reduce 29%

14/03/04 10:29:06 INFO streaming.StreamJob:  map 90%  reduce 29%

14/03/04 10:29:10 INFO streaming.StreamJob:  map 91%  reduce 29%

14/03/04 10:29:17 INFO streaming.StreamJob:  map 92%  reduce 29%

14/03/04 10:29:18 INFO streaming.StreamJob:  map 92%  reduce 30%

14/03/04 10:29:23 INFO streaming.StreamJob:  map 93%  reduce 30%

14/03/04 10:29:27 INFO streaming.StreamJob:  map 93%  reduce 31%

14/03/04 10:29:28 INFO streaming.StreamJob:  map 94%  reduce 31%

14/03/04 10:29:34 INFO streaming.StreamJob:  map 95%  reduce 31%

14/03/04 10:29:39 INFO streaming.StreamJob:  map 96%  reduce 31%

14/03/04 10:29:45 INFO streaming.StreamJob:  map 97%  reduce 32%

14/03/04 10:29:49 INFO streaming.StreamJob:  map 98%  reduce 32%

14/03/04 10:29:56 INFO streaming.StreamJob:  map 99%  reduce 32%

14/03/04 10:30:02 INFO streaming.StreamJob:  map 100%  reduce 32%

14/03/04 10:30:04 INFO streaming.StreamJob:  map 100%  reduce 33%

14/03/04 10:30:52 INFO streaming.StreamJob:  map 100%  reduce 37%

14/03/04 10:30:53 INFO streaming.StreamJob:  map 100%  reduce 39%

14/03/04 10:30:54 INFO streaming.StreamJob:  map 100%  reduce 45%

14/03/04 10:30:55 INFO streaming.StreamJob:  map 100%  reduce 61%

14/03/04 10:30:56 INFO streaming.StreamJob:  map 100%  reduce 66%

14/03/04 10:30:57 INFO streaming.StreamJob:  map 100%  reduce 70%

14/03/04 10:30:58 INFO streaming.StreamJob:  map 100%  reduce 72%

14/03/04 10:30:59 INFO streaming.StreamJob:  map 100%  reduce 77%

14/03/04 10:31:01 INFO streaming.StreamJob:  map 100%  reduce 82%

14/03/04 10:31:02 INFO streaming.StreamJob:  map 100%  reduce 83%

14/03/04 10:31:04 INFO streaming.StreamJob:  map 100%  reduce 86%

14/03/04 10:31:06 INFO streaming.StreamJob:  map 100%  reduce 88%

14/03/04 10:31:07 INFO streaming.StreamJob:  map 100%  reduce 89%

14/03/04 10:31:08 INFO streaming.StreamJob:  map 100%  reduce 91%

14/03/04 10:31:11 INFO streaming.StreamJob:  map 100%  reduce 92%

14/03/04 10:31:12 INFO streaming.StreamJob:  map 100%  reduce 94%

14/03/04 10:31:15 INFO streaming.StreamJob:  map 100%  reduce 95%

14/03/04 10:31:29 INFO streaming.StreamJob:  map 100%  reduce 96%

14/03/04 10:31:32 INFO streaming.StreamJob:  map 100%  reduce 98%

14/03/04 10:33:32 INFO streaming.StreamJob:  map 100%  reduce 100%

14/03/04 11:25:05 INFO streaming.StreamJob: Job complete: job_201403040750_0001

14/03/04 11:25:05 INFO streaming.StreamJob: Output: /statData/SISout/ahs/testTargetFunction/error/errorAllAtts.csv

[Antonio Piccolboni]

I share your expectation that the combined job should be in general faster, all else being equal, than two separate jobs. Unfortunately I do not understand your attempted explanation. Not writing intermediate results is faster than writing them, it's a pretty general result in computing and life that not doing something takes shorter than doing it. The general principle in this group is that if you post a reproducible example you get help, if you don't you get sympathy and wild guesses at best. Your post is in the second category. 

Antonio

--
post: rha...@googlegroups.com ||
unsubscribe: rhadoop+u...@googlegroups.com ||
web: https://groups.google.com/d/forum/rhadoop?hl=en-US
---
You received this message because you are subscribed to the Google Groups "RHadoop" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rhadoop+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.