Native batch indexing task failed silently when disk is out of space?

102 views
Skip to first unread message

Didip Kerabat

unread,
Mar 10, 2021, 11:00:47 AM3/10/21
to Druid User
Hi folks,

I noticed that when both middleManager and historical are running out of space, the native batch indexing task would fail silently without error.

This is the tail end of my index log. The log just stopped abruptly without any error indication.

2021-03-09T19:19:06,272 INFO [[index_parallel_mydata_lljejpfd_2021-03-09T18:37:07.667Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed in-memory data for segment[mydata_2021-02-07T00:00:00.000Z_2021-02-08T00:00:00.000Z_2021-03-09T18:37:07.684Z] spill[11] to disk in [506] ms (2,490 rows).
2021-03-09T19:19:06,292 INFO [[index_parallel_mydata_lljejpfd_2021-03-09T18:37:07.667Z]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed in-memory data with commit metadata [null] for segments: mydata_2021-02-07T00:00:00.000Z_2021-02-08T00:00:00.000Z_2021-03-09T18:37:07.684Z

I checked my logs on middleManager and historical and I saw nothing, it would be nice to see an error message there.

I set both of my logs with INFO level, should I change that to DEBUG?


Peter Marshall

unread,
Mar 23, 2021, 8:02:23 AM3/23/21
to Druid User
Could be a good one for a PR.  OOI are your logs going to local or are they going to, say, S3?

Didip Kerabat

unread,
Mar 24, 2021, 10:06:01 AM3/24/21
to Druid User
Looks like the middlemanager peon logs are stored temporarily on local disk and then after task is successful, they are uploaded to S3.

Didip Kerabat

unread,
Mar 24, 2021, 10:31:35 AM3/24/21
to Druid User
I believe I found the root cause of this issue and it's not Druid's fault.

Kubernetes was killing the pod because middleManager ran out of disk space.

Peter Marshall

unread,
Apr 16, 2021, 6:06:23 AM4/16/21
to druid...@googlegroups.com
Ohhhh...!!

Well, you helped me anyway because now I understand how the logs get generated and when :D. I will work with some docs authors to add more info in it :) 

--
You received this message because you are subscribed to a topic in the Google Groups "Druid User" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/druid-user/r4stRVPV_zA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to druid-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/d11300bd-4778-4290-aece-c9e5198f0c7bn%40googlegroups.com.


--
Peter Marshall
Apache Druid® Community Technology Evangelist


Reply all
Reply to author
Forward
0 new messages