some kafka index services can't ingest data

[xin....@shopee.com]

There are a lot of kafka index services, we have submit them successfully, but a few hours later, there are some kis tasks stopped ingesting data and no related running tasks, I had confirmed new data produced to these topics, what caused this issue? anyone have the same problems?thank you, the exception are below in overlord

2018-09-12T19:29:13,130 ERROR [KafkaSupervisor-shopee_my_s0__campaign_microsite] io.druid.indexing.kafka.supervisor.KafkaSupervisor - KafkaSupervisor[shopee_my_s0__campaign_microsite] failed to handle notice: {class=io.druid.indexing.kafka.supervisor.KafkaSupervisor, exceptionType=class java.lang.IllegalStateException, exceptionMessage=This consumer has already been closed., noticeClass=RunNotice}

java.lang.IllegalStateException: This consumer has already been closed.

at org.apache.kafka.clients.consumer.KafkaConsumer.ensureNotClosed(KafkaConsumer.java:1611) ~[kafka-clients-0.10.2.0.jar:?]

at org.apache.kafka.clients.consumer.KafkaConsumer.acquire(KafkaConsumer.java:1622) ~[kafka-clients-0.10.2.0.jar:?]

at org.apache.kafka.clients.consumer.KafkaConsumer.assignment(KafkaConsumer.java:759) ~[kafka-clients-0.10.2.0.jar:?]

at io.druid.indexing.kafka.supervisor.KafkaSupervisor.getOffsetFromKafkaForPartition(KafkaSupervisor.java:1886) ~[druid-kafka-indexing-service-0.12.0.jar:0.12.0]

at io.druid.indexing.kafka.supervisor.KafkaSupervisor.getOffsetFromStorageForPartition(KafkaSupervisor.java:1839) ~[druid-kafka-indexing-service-0.12.0.jar:0.12.0]

at io.druid.indexing.kafka.supervisor.KafkaSupervisor.generateStartingOffsetsForPartitionGroup(KafkaSupervisor.java:1821) ~[druid-kafka-indexing-service-0.12.0.jar:0.12.0]

at io.druid.indexing.kafka.supervisor.KafkaSupervisor.createNewTasks(KafkaSupervisor.java:1712) ~[druid-kafka-indexing-service-0.12.0.jar:0.12.0]

at io.druid.indexing.kafka.supervisor.KafkaSupervisor.runInternal(KafkaSupervisor.java:825) ~[druid-kafka-indexing-service-0.12.0.jar:0.12.0]

at io.druid.indexing.kafka.supervisor.KafkaSupervisor$RunNotice.handle(KafkaSupervisor.java:574) ~[druid-kafka-indexing-service-0.12.0.jar:0.12.0]

at io.druid.indexing.kafka.supervisor.KafkaSupervisor$2.run(KafkaSupervisor.java:366) [druid-kafka-indexing-service-0.12.0.jar:0.12.0]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

2018-09-12T19:29:13,140 ERROR [KafkaSupervisor-shopee_ph_s0__campaign_microsite] io.druid.indexing.kafka.supervisor.KafkaSupervisor - No task in [[index_kafka_shopee_ph_s0__campaign_microsite_92c63c0ed0737cb_nehnbobo]] succeeded before the completion timeout elapsed [PT1800S]!: {class=io.druid.indexing.kafka.supervisor.KafkaSupervisor}

2018-09-12T19:29:13,140 ERROR [KafkaSupervisor-shopee_ph_s0__campaign_microsite] io.druid.indexing.kafka.supervisor.KafkaSupervisor - KafkaSupervisor[shopee_ph_s0__campaign_microsite] failed to handle notice: {class=io.druid.indexing.kafka.supervisor.KafkaSupervisor, exceptionType=class java.lang.NullPointerException, exceptionMessage=null, noticeClass=RunNotice}

java.lang.NullPointerException

[Jonathan Wei]

This looks like some of Kafka ingestion bugs that were fixed after 0.12.0:

https://github.com/apache/incubator-druid/pull/5902

https://github.com/apache/incubator-druid/issues/5900

Can you try running 0.12.2 or 0.12.3-rc1?

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+unsubscribe@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/44eed95f-d71c-423a-a246-eec95860c84b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[xin....@shopee.com]

ok. thank you so much, I have been using 0.12.0. As I see that, the codes only  updated in the kafkaSupervisor, so I just need to update the druid-kafka-indexing-service-0.12.0.jar to druid-kafka-indexing-service-0.12.2.jar, anything else would be ok, is that right?

[Jonathan Wei]

You should upgrade all the jars, since there are also changes to "indexing-service" and other Druid packages between 0.12.0 -> current version.

Thanks,

Jon

--

You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+unsubscribe@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/6cbd8104-12da-4d72-bac8-d06503265c35%40googlegroups.com.

[Xin Zhang]

understand, thank you

To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.

To post to this group, send email to druid...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/6cbd8104-12da-4d72-bac8-d06503265c35%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Druid User" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/druid-user/WxVVkDy7KYQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to druid-user+...@googlegroups.com.

To post to this group, send email to druid...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/CAG0p_PFYqW-ZYxR3dCQ6ibOBJT8_6ny0Ra7Y_ZcbqD5HKqS_MQ%40mail.gmail.com.