Getting error in loading segments after restarting data server

[TEJAS]

Hi all,
I have increased the hard disk capacity of data server, changed data/historical/runtime.propertis and re-started.

After that when i checked in historical.log i am getting below statement

 org.apache.druid.server.coordination.SegmentLoadDropHandler - Loading segment[1590/64677]

it will complete loading all the segment and create a file hs_err_pid277556.log in druid home folder and started loading segment from segment-cache folder from 0.

The same thing is repeating again and again and each time it will create new log file with pid.

In console I can see like below

Also in coordinator-overlord.log i can see below error

2021-07-07T13:19:30,263 ERROR [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Tier[_default_tier] has no servers! Check your cluster configuration!: {class=org.apache.druid.server.coordinator.rules.LoadRule}

2021-07-07T13:19:30,263 ERROR [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Tier[_default_tier] has no servers! Check your cluster configuration!: {class=org.apache.druid.server.coordinator.rules.LoadRule}

2021-07-07T13:19:30,263 ERROR [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Tier[_default_tier] has no servers! Check your cluster configuration!: {class=org.apache.druid.server.coordinator.rules.LoadRule}

2021-07-07T13:19:30,263 ERROR [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Tier[_default_tier] has no servers! Check your cluster configuration!: {class=org.apache.druid.server.coordinator.rules.LoadRule}

2021-07-07T13:19:30,263 ERROR [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Tier[_default_tier] has no servers! Check your cluster configuration!: {class=org.apache.druid.server.coordinator.rules.LoadRule}

2021-07-07T13:19:30,263 ERROR [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Tier[_default_tier] has no servers! Check your cluster configuration!: {class=org.apache.druid.server.coordinator.rules.LoadRule}

2021-07-07T13:19:30,263 ERROR [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Tier[_default_tier] has no servers! Check your cluster configuration!: {class=org.apache.druid.server.coordinator.rules.LoadRule}

2021-07-07T13:19:30,263 ERROR [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Tier[_default_tier] has no servers! Check your cluster configuration!: {class=org.apache.druid.server.coordinator.rules.LoadRule}

2021-07-07T13:19:30,263 ERROR [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Tier[_default_tier] has no servers! Check your cluster configuration!: {class=org.apache.druid.server.coordinator.rules.LoadRule}

Can anyone please help me understanding the error?

Regards,

Tejas

[Vaibhav Vaibhav]

Hi Tejas,

The error you see in the coordinator log (2021-07-07T13:19:30,263 ERROR [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Tier[_default_tier] has no servers!) - indicates that your Historical is down.

Based on what you have described about historicals , it seems that you are hitting a similar issue being discussed recently here -
https://mail.google.com/mail/u/0/#inbox/FMfcgzGkZGlBzhvTknsNRmFGvpGZJHpp?compose=DmwnWsdMWppfqgshNlvPPvdcVJCtpRlrlvvRMGfNtfkqHNNpLxcVKrkwBKQpmpSVrzLVCXZnctjQ

Please read that thread and try the suggestions, I think that should help in your case as well.

Regards,
Vaibhav

::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------

The contents of this e-mail and any attachments are confidential and intended for the named recipient(s) only.E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents(with or without referred errors) shall therefore not attach any liability on the originator or redBus.com. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of redBus.com. Any form of reproduction, dissemination, copying, disclosure, modification,distribution and / or publication of this message without the prior written consent of authorized representative of redbus.com is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.Before opening any email and/or attachments, please check them for viruses and other defects.

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/35184b91-8d67-4d5f-824c-b0ecedb06e1fn%40googlegroups.com.

[Vaibhav Vaibhav]

Most probably you may need to adjust -
Linux system limits -  (No of open files,max_map_count) and maybe a historical heap.

https://druid.apache.org/docs/latest/operations/basic-cluster-tuning.html#linux-limits

Further, Historical logs could provide you more details for historical startup issues. On the other side , It seems that you have a lot of segments to load and I am guessing there are possibly huge no of tiny segments (few Kbs to Few MBs in size), If that's the case, You may also want to compact them to get decent-sized segments ( ~500 MB / 5 Million rows per segment) which can drastically reduce the total segment to load and also improve overall query performance and historical stability.

Regards,
Vaibhav