Kafka Ingestion Failed Tasks

[Abinash Kumar]

Hi Team,

I have been seeing failures with Kafka Ingestion. The tasks are in 'Failed'

 state. The supervisor  is in 'UNHEALTHY' state. The ingestion is not complete and none of the new tasks are successful.

I have my  doubts on the consistent logging of 'Flushed in-memory data for segment ...' but I am not sure what to make out of it.  Attached Log for reference.

The configuration is 'conf/druid/single-server/micro-quickstart' . The machine has 8GB ram. There is enough space on HDD to assimilate data. Historical usage in only 0.5% at the moment.

Please help to debug this. Thanks for your time.

[Mark Herrera]

Hi,

Sorry to hear that your ingestion isn't complete, and thanks for attaching the log.

I looked for warnings in the log. One particular warning jumped out:

2022-03-12T22:35:24,476 WARN [main] org.apache.druid.indexing.common.config.TaskConfig - Batch processing mode argument value is null or not valid:[null], defaulting to[CLOSED_SEGMENTS]

I wonder if you hit this issue?

Regarding the "Flushed in-memory data for segment ..." message, it looks like it's part of the Appenderator class, and it looks like it returns the number of rows persisted.

Best,

Mark