Error on hadoop batch ingestion

[Marcelo Oikawa]

Hi, list.

I did a hadoop batch ingestion on Druid but I getting this error:

2016-11-16T20:00:24,544 INFO [localfetcher#17] org.apache.hadoop.mapreduce.task.reduce.LocalFetcher - fetcher#17 - MergeManager returned Status.WAIT ...
2016-11-16T20:00:24,545 INFO [localfetcher#17] org.apache.hadoop.mapreduce.task.reduce.LocalFetcher - fetcher#17 - MergeManager returned Status.WAIT ...
2016-11-16T20:00:24,545 INFO [localfetcher#17] org.apache.hadoop.mapreduce.task.reduce.LocalFetcher - fetcher#17 - MergeManager returned Status.WAIT ...
2016-11-16T20:00:25,921 WARN [Thread-53] org.apache.hadoop.mapred.LocalJobRunner - job_local633977191_0001
java.lang.Exception: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs
	at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462) ~[hadoop-mapreduce-client-common-2.3.0.jar:?]
	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529) [hadoop-mapreduce-client-common-2.3.0.jar:?]
Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs
	at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134) ~[hadoop-mapreduce-client-core-2.3.0.jar:?]
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376) ~[hadoop-mapreduce-client-core-2.3.0.jar:?]
	at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319) ~[hadoop-mapreduce-client-common-2.3.0.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_101]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_101]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_101]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_101]
	at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_101]
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/map_91.out
	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:402) ~[hadoop-common-2.3.0.jar:?]
	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) ~[hadoop-common-2.3.0.jar:?]
	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) ~[hadoop-common-2.3.0.jar:?]
	at org.apache.hadoop.mapred.MROutputFiles.getInputFileForWrite(MROutputFiles.java:206) ~[hadoop-mapreduce-client-core-2.3.0.jar:?]
	at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl$InMemoryMerger.merge(MergeManagerImpl.java:450) ~[hadoop-mapreduce-client-core-2.3.0.jar:?]
	at org.apache.hadoop.mapreduce.task.reduce.MergeThread.run(MergeThread.java:94) ~[hadoop-mapreduce-client-core-2.3.0.jar:?]
2016-11-16T20:00:26,213 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job - Job job_local633977191_0001 failed with state FAILED due to: NA
2016-11-16T20:00:26,287 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job - Counters: 33

Does anyone know why?

thanks

[Nishant Bangarwa]

Hi,
It seems the problem is that when Hadoop is trying to spill to disk, your nodes don't have enough disk space to hold all the intermediate output of the mappers. Make sure you have enough capacity on the disk. you can set the directory location on local disk by setting the property -  mapreduce.cluster.local.dir

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/CAD1mpAEeaN-%3DbPtH4fj32k5GdJVnemM68E2jkn2FHNcdsmmL_g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.