Populate Old Segments From Deep Storage

[sm...@aterlo.com]

Hello,

I wanted to know how to populate old segments from deep storage if query data is not available on the local cache. For example, we have 2 days data segments on local disk cached but what if we want to query data from 3 or 4 days ago?

Thank you

[Akhilesh Murthy]

In that case you want to check your retention rules for the datasource, using that you can load the data from HDFS to your historicals. You may increase your retention rules to get the interval of data you are looking for.

Regards,

Akhilesh

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/ebd79a80-34c8-4f36-a7e2-44b9d7e7430an%40googlegroups.com.

--

Regards,

Akhilesh

[sm...@aterlo.com]

But in my case data is not available because of space issues. Total data for 2 days comes to around 48GB and disk space is 50GB. This means it will store 2 days of data only on disk but in some cases, we need to query data from before 2 days. 

[Peter Marshall]

Data must be loaded onto Historicals in order for it to be queryable - simply because it's the Historicals that actually do the querying.

The data loaded by Historicals is controlled by the coordinator - you set the load / drop rules accordingly.

https://druid.apache.org/docs/latest/operations/rule-configuration.html#load-rules

If it's data you will not query often, you might want to spin up a second tier - you can then just bring this up when you want to query data that's older than you usually look at, and then take it back offline afterwards.  You could also use muuuuuch cheaper hardware :)