Error: Could not allocate segment

[Joaquín Silva]

Hello, I'm having this error when I try to start a kafka-indexing service task

ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[KafkaIndexTask{id=index_kafka_sep-druid_e7dd9410a20390d_fknochmf, type=index_kafka, dataSource=sep-druid}]
com.metamx.common.ISE: Could not allocate segment for row with timestamp[2016-12-26T11:00:10.000-03:00]
	at io.druid.indexing.kafka.KafkaIndexTask.run(KafkaIndexTask.java:442) ~[?:?]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.9.2.jar:0.9.2]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.9.2.jar:0.9.2]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_111]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
2016-12-26T17:36:09,055 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_kafka_sep-druid_e7dd9410a20390d_fknochmf] status changed to [FAILED].
2016-12-26T17:36:09,057 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
  "id" : "index_kafka_sep-druid_e7dd9410a20390d_fknochmf",
  "status" : "FAILED",
  "duration" : 5530
}

The task was working fine until I restarted all the services (coordinator, broker, middleManager, overlord and hisrotical). 

This is my spec:

https://gist.github.com/goaquin/d0e67de2a89afa644821b768eb61097d

Druid version: 0.9.2

Regards,

Joaquín Silva

[Ben Vogan]

Hi Joaquin,

This has happened to me after running a hadoop indexing job on the same data source.  Unless you are using 0.9.2 and have forceExtendableShardSpecs in the tuningConfig of your spec the segments produced by hadoop jobs cannot be expanded and any data arriving for that time period will cause the kafka indexing task to fail.  If you are still on 0.9.1.1 then you can configure the lateMessageRejectionPeriod in the ioConfig of your supervisor spec to drop messages older than 'x'.  In this way you can always be sure that it is safe to re-index data older than 'x' without breaking your real time ingestion.

Regards,

--Ben

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+unsubscribe@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/bc086c6f-f98f-47b1-9939-56d2924ea7df%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

BENJAMIN VOGAN | Data Platform Team Lead

shopkick

 

The indispensable app that rewards you for shopping.

[Ryne Yang]

yes, reading through the source code:

IndexerSQLMetadataStorageCoordinator.java:

if (existingChunks
    .stream()
    .flatMap(holder -> StreamSupport.stream(holder.getObject().spliterator(), false))
    .anyMatch(chunk -> !chunk.getObject().getShardSpec().isCompatible(shardSpecFactory.getShardSpecClass()))) {
  // All existing segments should have a compatible shardSpec with shardSpecFactory.
  return null;

this will force the segments to have the same shardSpec.

and I've noticed that in our case, we used a different shardSpec during reindexing. 

which caused the problem.

make sure you use the same shardSpec for your kafka ingestion and reindexing.

otherwise this will cause ingestion to fail and it's hard to fix. 

R

On Monday, December 26, 2016 at 4:40:22 PM UTC-5, Ben Vogan wrote:

Hi Joaquin,

This has happened to me after running a hadoop indexing job on the same data source.  Unless you are using 0.9.2 and have forceExtendableShardSpecs in the tuningConfig of your spec the segments produced by hadoop jobs cannot be expanded and any data arriving for that time period will cause the kafka indexing task to fail.  If you are still on 0.9.1.1 then you can configure the lateMessageRejectionPeriod in the ioConfig of your supervisor spec to drop messages older than 'x'.  In this way you can always be sure that it is safe to re-index data older than 'x' without breaking your real time ingestion.

Regards,

--Ben

On Mon, Dec 26, 2016 at 2:46 PM, Joaquín Silva <joaquin.si...@gmail.com> wrote:

Hello, I'm having this error when I try to start a kafka-indexing service task

ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[KafkaIndexTask{id=index_kafka_sep-druid_e7dd9410a20390d_fknochmf, type=index_kafka, dataSource=sep-druid}]
com.metamx.common.ISE: Could not allocate segment for row with timestamp[2016-12-26T11:00:10.000-03:00]
	at io.druid.indexing.kafka.KafkaIndexTask.run(KafkaIndexTask.java:442) ~[?:?]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.9.2.jar:0.9.2]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.9.2.jar:0.9.2]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_111]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
2016-12-26T17:36:09,055 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_kafka_sep-druid_e7dd9410a20390d_fknochmf] status changed to [FAILED].
2016-12-26T17:36:09,057 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
  "id" : "index_kafka_sep-druid_e7dd9410a20390d_fknochmf",
  "status" : "FAILED",
  "duration" : 5530
}

The task was working fine until I restarted all the services (coordinator, broker, middleManager, overlord and hisrotical). 

This is my spec:

https://gist.github.com/goaquin/d0e67de2a89afa644821b768eb61097d

Druid version: 0.9.2

Regards,

Joaquín Silva

--
You received this message because you are subscribed to the Google Groups "Druid User" group.

To unsubscribe from this group and stop receiving emails from it, send an email to druid...@googlegroups.com.

To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/bc086c6f-f98f-47b1-9939-56d2924ea7df%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.