Thank you for your answer, you are right, I did see this error on my previous attempt, and made a fix similar to the one you linked, thanks!
If, however, I do use the mapreduce version of lemur with your fix, I get a strange behavior, here it is:
21:19:19,004 INFO FileInputFormat:237 - Total input paths to process : 1
21:19:19,172 INFO ProcessTree:63 - setsid exited with exit code 0
21:19:19,176 INFO Task:534 - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@189945c1
21:19:19,185 INFO MapTask:944 - io.sort.mb = 100
21:19:19,206 INFO MapTask:956 - data buffer = 79691776/99614720
21:19:19,208 INFO MapTask:957 - record buffer = 262144/327680
21:19:19,220 INFO WarcFileRecordReader:117 - file:/home/shlomi/test/CC-MAIN-20130516131833-00097-ip-10-60-113-184.ec2.internal.warc.wet.gz
21:19:19,223 INFO WarcFileRecordReader:122 - Compression enabled
21:19:27,648 INFO MapTask:1284 - Starting flush of map output
21:19:27,653 INFO Task:858 - Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
21:19:27,655 INFO LocalJobRunner:323 -
21:19:27,656 INFO Task:970 - Task 'attempt_local_0001_m_000000_0' done.
21:19:27,660 INFO Task:534 - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@5945d890
21:19:27,661 INFO LocalJobRunner:323 -
21:19:27,663 INFO Merger:390 - Merging 1 sorted segments
21:19:27,667 INFO Merger:473 - Down to the last merge-pass, with 0 segments left of total size: 0 bytes
21:19:27,667 INFO LocalJobRunner:323 -
21:19:27,670 INFO Task:858 - Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
21:19:27,671 INFO LocalJobRunner:323 -
21:19:27,671 INFO Task:1011 - Task attempt_local_0001_r_000000_0 is allowed to commit now
21:19:27,672 INFO FileOutputCommitter:173 - Saved output of task 'attempt_local_0001_r_000000_0' to /tmp/wet-ass
21:19:27,672 INFO LocalJobRunner:323 - reduce > reduce
21:19:27,673 INFO Task:970 - Task 'attempt_local_0001_r_000000_0' done.
Please notice the time jump between "compression enabled" and the next line, this is where I expect to see the mapper and reducer prints..
Nothing prints out and I get an empty result file with no errors...
any ideas?
Thanks,
Shlomi