Historical storage size

Skip to first unread message

Donovan Owen

Sep 17, 2021, 5:16:06 PMSep 17
to Druid User
Hello Druid Community,

I am needing some guidance on what to do with my Historical service that is filling up. I currently have a little over 10Gb assigned to my Historical and it is almost filled up (see screen capture attached). 
What happens if this fills completely up?
I've read that there is a setting, druid.server.maxsize, that can be changed but that it's usually not necessary. So how do I get more space, or do I need more space? If I don't need more space what changes do I need to consider?
Currently there are 5 supervisors stream ingesting data, about 2 years of data, with Druid version 0.20.0.
I've also attached the historical portion of my deploy for reference.
Thanks for your time.
historical usage.PNG

Didip Kerabat

Sep 30, 2021, 7:30:38 PMSep 30
to Druid User
Add another historical node to get more space.

Donovan Owen

Oct 8, 2021, 6:24:12 PMOct 8
to Druid User
Thanks for you reply. I'm wondering what happens when that node gets filled up, do I just add another node and another and another as they fill up. Is there a better way to scale this so I don't need to keep an eye on it and keep adding nodes?

Sergio Ferragut

Oct 11, 2021, 10:54:38 AMOct 11
to Druid User
A few thoughts:

- Have you also looked into optimization of the space you are currently using?
- One other thing to consider is TTL for your data? Do you need 2 years? Perhaps you can start dropping some older segments. Take a look at https://druid.apache.org/docs/latest/operations/rule-configuration.html#permanently-deleting-data.

If that is not enough to reduce space usage on your historical, then it is likely that you do need more nodes because the size of your data and the space used by your workloads requires that much space.

Hope this helps.
Reply all
Reply to author
0 new messages