Failed Supervisors

16 views
Skip to first unread message

Suprabhat

unread,
Jun 18, 2024, 7:23:48 AMJun 18
to Druid User

Hi All, we found unparsable events being discarded. Following are the logs:
 

  • ERROR [MonitorScheduler-0] org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor - [17,305] unparseable events discarded.
  • ERROR [MonitorScheduler-0] org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor - [14,281] unparseable events discarded.
  • ERROR [MonitorScheduler-0] org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor - [14,104] unparseable events discarded.
  • ERROR [MonitorScheduler-0] org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor - [14,415] unparseable events discarded.
  • ERROR [MonitorScheduler-0] org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor - [15,378] unparseable events discarded.
  • ERROR [MonitorScheduler-0] org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor - [14,633] unparseable events discarded.
  • ERROR [MonitorScheduler-0] org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor - [15,145] unparseable events discarded.
  • ERROR [MonitorScheduler-0] org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor - [15,243] unparseable events discarded.
  • ERROR [MonitorScheduler-0] org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor - [14,619] unparseable events discarded.

While error messages in metadata are following:

  • status:"FAILED", errorMsg:"Canceled as unknown task. See middleManager or indexer logs for more details."
  • status:"FAILED", errorMsg:"No task in the corresponding pending completion taskGroup[1] succeeded before completion timeout ela..."
  • status:"FAILED", errorMsg:"Canceled as unknown task. See middleManager or indexer logs for more details."
  • status:"FAILED", errorMsg:"Canceled as unknown task. See middleManager or indexer logs for more details."
  • status:"FAILED", errorMsg:"Canceled as unknown task. See middleManager or indexer logs for more details."

My question is : Can these much of unparsable events lead to task failure which in turns lead to failure of corresponding supervisor?

If these are not the exact reasons, what else can be the possibility for this error?

Thank you for the help. 

Suprabhat

unread,
Jun 18, 2024, 7:30:08 AMJun 18
to Druid User
Also, received following error log with unparseable events:

ERROR [MonitorScheduler-0] org.apache.druid.indexing.common.stats.TaskRealtimeMetricsMonitor - [13,058] unparseable events discarded.

ERROR [task-runner-0-priority-0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Encountered exception in run() before persisting. java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[?:1.8.0_382] at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[?:1.8.0_382] at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.possiblyPause(SeekableStreamIndexTaskRunner.java:1309) ~[druid-indexing-service-25.0.0.jar:25.0.0] at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:581) ~[druid-indexing-service-25.0.0.jar:25.0.0] at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:266) ~[druid-indexing-service-25.0.0.jar:25.0.0] at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.runTask(SeekableStreamIndexTask.java:151) ~[druid-indexing-service-25.0.0.jar:25.0.0] at org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:169) ~[druid-indexing-service-25.0.0.jar:25.0.0] at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:477) ~[druid-indexing-service-25.0.0.jar:25.0.0] at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:449) ~[druid-indexing-service-25.0.0.jar:25.0.0] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_382] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_382] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_382] at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_382]
Reply all
Reply to author
Forward
0 new messages