Hadoop batch ingestion failure

sumatheja dasararaju

unread,

Jul 27, 2017, 1:36:33 AM7/27/17

to Druid User

Hi,

I have been running into ingestion failures intermittently. Here is the stack trace from the overlord logs

[task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[HadoopIndexTask{id=index_hadoop_sales-rank-daily_2017-07-27T04:33:48.922Z, type=index_hadoop, dataSource=sales-rank-daily}]

java.util.NoSuchElementException

at java.util.ArrayList$Itr.next(ArrayList.java:854) ~[?:1.8.0_131]

at com.google.common.collect.Iterators.getOnlyElement(Iterators.java:297) ~[guava-16.0.1.jar:?]

at com.google.common.collect.Iterables.getOnlyElement(Iterables.java:285) ~[guava-16.0.1.jar:?]

at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:202) ~[druid-indexing-service-0.9.2.jar:0.9.2]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.9.2.jar:0.9.2]

at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.9.2.jar:0.9.2]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

2017-07-27T04:34:02,584 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_hadoop_sales-rank-daily_2017-07-27T04:33:48.922Z] status changed to [FAILED].

2017-07-27T04:34:02,586 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {

"id" : "index_hadoop_sales-rank-daily_2017-07-27T04:33:48.922Z",

"status" : "FAILED",

"duration" : 6574

}

Not sure where to get more information about the issue or how to go about debugging it. Attaching the ingestion spec file as well. Any help would be highly appreciated.

ingestion_spec.txt

sumatheja dasararaju

unread,

Jul 27, 2017, 1:58:11 AM7/27/17

to Druid User

Also, wanted to add that I am using druid version 0.9.2.

sumatheja dasararaju

unread,

Jul 27, 2017, 4:51:01 PM7/27/17

to Druid User

Our prod cluster stopped ingesting data due to this issue. Any pointers would really help. Thanks a lot!

On Wednesday, July 26, 2017 at 10:36:33 PM UTC-7, sumatheja dasararaju wrote:

sumatheja

unread,

Jul 28, 2017, 1:05:28 AM7/28/17

to druid...@googlegroups.com

The issue was due to metadata store. Once the mysql db was restarted, things started working normal.

--
You received this message because you are subscribed to a topic in the Google Groups "Druid User" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/druid-user/ZkG2mOXnzi0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to druid-user+unsubscribe@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/f3a64b06-4108-4a7a-bc72-3d895124de84%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.