Hadoop-based Batch Ingestion Failed
171 views
Skip to first unread message
misslz...@gmail.com
Nov 3, 2016, 1:46:30 AM11/3/16
to Druid Development
I post a hadoop-based batch ingestion task to druid. It runs well but occurs errors at last.
Post json config as follow:
{
"type": "index_hadoop",
"spec": {
"dataSchema": {
"dataSource": "wikipedia",
"parser": {
"type": "hadoopyString",
"parseSpec": {
"format": "json",
"timestampSpec": {
"column": "time",
"format": "auto"
},
"dimensionsSpec": {
"dimensions": [
"channel",
"cityName",
"comment",
"countryIsoCode",
"countryName",
"isAnonymous",
"isMinor",
"isNew",
"isRobot",
"isUnpatrolled",
"metroCode",
"namespace",
"page",
"regionIsoCode",
"regionName",
"user"
],
"dimensionExclusions": [ ],
"spatialDimensions": [ ]
}
}
},
"metricsSpec": [
{
"type": "count",
"name": "count"
},
{
"type": "doubleSum",
"name": "added",
"fieldName": "added"
},
{
"type": "doubleSum",
"name": "deleted",
"fieldName": "deleted"
},
{
"type": "doubleSum",
"name": "delta",
"fieldName": "delta"
}
],
"granularitySpec": {
"type": "uniform",
"segmentGranularity": "HOUR",
"queryGranularity": "NONE",
"intervals": [
"2015-09-11/2015-09-12"
]
}
},
"ioConfig": {
"type": "hadoop",
"inputSpec": {
"type": "static",
"paths": "/home/bjduanweidong/druid/druid-0.9.1.1/quickstart/wikiticker-2015-09-12-sampled.json"
}
},
"tuningConfig": {
"type": "hadoop"
}
},
"hadoopDependencyCoordinates": [
"org.apache.hadoop:hadoop-client:2.3.0"
]
}
the indexing-log errors as follow:
2016-11-03T01:48:20,875 INFO [pool-19-thread-1] org.apache.hadoop.mapred.LocalJobRunner - 1 / 1 copied.
2016-11-03T01:48:20,875 INFO [pool-19-thread-1] org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2016-11-03T01:48:20,876 INFO [pool-19-thread-1] org.apache.hadoop.mapred.Merger - Merging 1 sorted segments
2016-11-03T01:48:20,877 INFO [pool-19-thread-1] org.apache.hadoop.mapred.Merger - Down to the last merge-pass, with 0 segments left of total size: 0 bytes
2016-11-03T01:48:20,877 INFO [pool-19-thread-1] org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl - Merged 1 segments, 2 bytes to disk to satisfy reduce memory limit
2016-11-03T01:48:20,878 INFO [pool-19-thread-1] org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl - Merging 1 files, 6 bytes from disk
2016-11-03T01:48:20,878 INFO [pool-19-thread-1] org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl - Merging 0 segments, 0 bytes from memory into reduce
2016-11-03T01:48:20,878 INFO [pool-19-thread-1] org.apache.hadoop.mapred.Merger - Merging 1 sorted segments
2016-11-03T01:48:20,878 INFO [pool-19-thread-1] org.apache.hadoop.mapred.Merger - Down to the last merge-pass, with 0 segments left of total size: 0 bytes
2016-11-03T01:48:20,879 INFO [pool-19-thread-1] org.apache.hadoop.mapred.LocalJobRunner - 1 / 1 copied.
2016-11-03T01:48:20,880 INFO [Thread-20] org.apache.hadoop.mapred.LocalJobRunner - reduce task executor complete.
2016-11-03T01:48:20,881 WARN [Thread-20] org.apache.hadoop.mapred.LocalJobRunner - job_local842174463_0001
java.lang.Exception: java.io.IOException: Mkdirs failed to create file:/user/ad_pangu/bjduanweidong/tmp/druid-indexing/wikipedia/2016-11-03T014806.753Z/0fcc92d530544690934e919a114d25a7/_temporary/0/_temporary/attempt_local842174463_0001_r_000000_0
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462) ~[hadoop-mapreduce-client-common-2.3.0.jar:?]
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529) [hadoop-mapreduce-client-common-2.3.0.jar:?]
Caused by: java.io.IOException: Mkdirs failed to create file:/user/ad_pangu/bjduanweidong/tmp/druid-indexing/wikipedia/2016-11-03T014806.753Z/0fcc92d530544690934e919a114d25a7/_temporary/0/_temporary/attempt_local842174463_0001_r_000000_0
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:438) ~[hadoop-common-2.3.0.jar:?]
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:424) ~[hadoop-common-2.3.0.jar:?]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:907) ~[hadoop-common-2.3.0.jar:?]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:888) ~[hadoop-common-2.3.0.jar:?]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:785) ~[hadoop-common-2.3.0.jar:?]
at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:132) ~[hadoop-mapreduce-client-core-2.3.0.jar:?]
at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:540) ~[hadoop-mapreduce-client-core-2.3.0.jar:?]
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:614) ~[hadoop-mapreduce-client-core-2.3.0.jar:?]
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) ~[hadoop-mapreduce-client-core-2.3.0.jar:?]
at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319) ~[hadoop-mapreduce-client-common-2.3.0.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[?:1.7.0_75]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[?:1.7.0_75]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[?:1.7.0_75]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[?:1.7.0_75]
at java.lang.Thread.run(Thread.java:745) ~[?:1.7.0_75]
2016-11-03T01:48:21,686 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job - Job job_local842174463_0001 failed with state FAILED due to: NA
2016-11-03T01:48:21,746 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job - Counters: 33
File System Counters
FILE: Number of bytes read=17106634
FILE: Number of bytes written=220970
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=39244
Map output records=0
Map output bytes=0
Map output materialized bytes=144
Input split bytes=320
Combine input records=0
Combine output records=0
Reduce input groups=0
Reduce shuffle bytes=144
Reduce input records=0
Reduce output records=0
Spilled Records=0
Shuffled Maps =24
Failed Shuffles=0
Merged Map outputs=24
GC time elapsed (ms)=30
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=253755392
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=0
2016-11-03T01:48:21,790 INFO [task-runner-0-priority-0] io.druid.indexer.JobHelper - Deleting path[/user/ad_pangu/bjduanweidong/tmp/druid-indexing/wikipedia/2016-11-03T014806.753Z/0fcc92d530544690934e919a114d25a7]
2016-11-03T01:48:21,811 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[HadoopIndexTask{id=index_hadoop_wikipedia_2016-11-03T01:48:06.584Z, type=index_hadoop, dataSource=wikipedia}]
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]
at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:204) ~[druid-indexing-service-0.9.1.1.jar:0.9.1.1]
at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:208) ~[druid-indexing-service-0.9.1.1.jar:0.9.1.1]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.9.1.1.jar:0.9.1.1]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.9.1.1.jar:0.9.1.1]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_75]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_75]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_75]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_75]
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_75]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_75]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_75]
at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_75]
at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:201) ~[druid-indexing-service-0.9.1.1.jar:0.9.1.1]
... 7 more
Caused by: com.metamx.common.ISE: Job[class io.druid.indexer.IndexGeneratorJob] failed!
at io.druid.indexer.JobHelper.runJobs(JobHelper.java:343) ~[druid-indexing-hadoop-0.9.1.1.jar:0.9.1.1]
at io.druid.indexer.HadoopDruidIndexerJob.run(HadoopDruidIndexerJob.java:94) ~[druid-indexing-hadoop-0.9.1.1.jar:0.9.1.1]
at io.druid.indexing.common.task.HadoopIndexTask$HadoopIndexGeneratorInnerProcessing.runTask(HadoopIndexTask.java:261) ~[druid-indexing-service-0.9.1.1.jar:0.9.1.1]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_75]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_75]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_75]
at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_75]
at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:201) ~[druid-indexing-service-0.9.1.1.jar:0.9.1.1]
... 7 more
2016-11-03T01:48:21,820 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_hadoop_wikipedia_2016-11-03T01:48:06.584Z] status changed to [FAILED].
2016-11-03T01:48:21,823 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
"id" : "index_hadoop_wikipedia_2016-11-03T01:48:06.584Z",
"status" : "FAILED",
"duration" : 8579
}
2016-11-03T01:48:21,832 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.server.coordination.AbstractDataSegmentAnnouncer.stop()] on object[io.druid.server.coordination.BatchDataSegmentAnnouncer@5048796
4].
What's wrong with the configuration? The tranquility batch ingestion runs well ,and use the HDFS deep storage.
Message has been deleted
David Lim
Nov 4, 2016, 1:31:32 PM11/4/16
to Druid Development
I just realized workingPath doesn't work for Hadoop indexing tasks and that you found a solution in another thread.
Here's the reference to the other thread: https://groups.google.com/forum/#!topic/druid-development/GNbQ6ISN7Ao
On Friday, November 4, 2016 at 11:23:16 AM UTC-6, David Lim wrote:
Here's the reference to the other thread: https://groups.google.com/forum/#!topic/druid-development/GNbQ6ISN7Ao
On Friday, November 4, 2016 at 11:23:16 AM UTC-6, David Lim wrote:
Looks like a file permission error:
java.lang.Exception: java.io.IOException: Mkdirs failed to create file:/user/ad_pangu/
bjduanweidong/tmp/druid-indexing/wikipedia/2016-11-03T014806.753Z/0fcc92d530544690934e919a114d25a7/_temporary/0/_temporary/attempt_local842174463_0001_r_000000_0
Does /user/ad_pangu/bjduanweidong/tmp exist and does the user for Druid have the permissions to write to here?
You can try setting the workingPath in the HadoopTuningConfig to something you have write permissions for.
Reply all
Reply to author
Forward
0 new messages