Getting Exception : KAFKA Indexing Service

125 views
Skip to first unread message

Arpan Khagram

unread,
Jan 9, 2017, 1:09:35 AM1/9/17
to Druid User
Hi Team, We are getting one exception while running KAFKA indexing job for one of the data type we have.

PFB the exception -

2017-01-06T13:18:05,469 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[KafkaIndexTask{id=index_kafka_yu

konSEGMENT_cb3d8d61b79f8f8_fommniod, type=index_kafka, dataSource=yukonSEGMENT}]

com.metamx.common.ISE: Could not allocate segment for row with timestamp[2017-01-05T12:00:00.000Z]

        at io.druid.indexing.kafka.KafkaIndexTask.run(KafkaIndexTask.java:442) ~[?:?]

        at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.9.2-rc2.jar:0.9.2-

rc2]

        at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.9.2-rc2.jar:0.9.2-

rc2]

        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_91]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_91]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_91]

        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]

2017-01-06T13:18:05,475 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_kafka_yukonSEGMENT_cb3d8d61b79f8f8_fommniod] status cha

nged to [FAILED].


Also attaching the JSON of KAFKA Indexing Service. CAn you please let me know what could have gone wrong ?


Thanks in advance.


Regards,

Arpan Khagram

+91 8308993200

job_kafka_indexing.json

Nishant Bangarwa

unread,
Jan 9, 2017, 2:38:21 AM1/9/17
to Druid User
I guess, you might have run batch index job for the interval you are trying to ingest the data. Fwiw, the segments generated by batch index task tasks are by default non-extendable i.e any data arriving for that time period will lead to above exception with kafka indexing task. 
If you are using 0.9.2, you can set forceExtendableShardSpecs to true in tuningConfig for batch index task to make the segments extendable. (docs - http://druid.io/docs/latest/ingestion/tasks.html)


--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/0791ff05-932b-42b3-89c2-3e73ed4f577b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Arpan Khagram

unread,
Feb 8, 2017, 7:33:35 AM2/8/17
to Druid User
Hi Nishant, We were trying to run KAFKA Indexing Service and not Batch job.
Reply all
Reply to author
Forward
0 new messages