Druid query behavior for older data that has been dropped from historical nodes

[Rahul Aggarwal]

I will appreciate any help with the following question on a specific druid usage pattern.

The pattern is as follows.

Deep storage is setup as S3; druid coordinator rules are setup to load data for the latest one month and drop older data; a query is then made for data that is older than one month.

In this scenario how will the druid broker know which historical nodes to forward the query to and further how will it take into account the current capacity of the historical nodes?

I have gone through the documentation, the druid white paper and previous postings in this google group, but am not clear about the behavior.

Much thanks,

rahul

[Gian Merlino]

Hi Rahul,

I think you are missing one thing: data that is not loaded in advance on historical nodes cannot be queried. So in your scenario, where a query is made for a time range that has been dropped, what happens is you will just get no results at all.

Gian

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+unsubscribe@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/cbe8d632-7c38-4ac8-a4c7-8b8dcb205c35%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rahul Aggarwal]

Hi Gian,

Thank you for the response.

I was hoping that druid can be configured to load data on historical nodes on demand. 

My use case is one where we will have > 10TB of data over a year and will accumulate year over year. Last month's data (~ 1TB) needs to be available for querying on a fairly regular basis and hence needs to be loaded in advance on the historical nodes.

The rest of the data (> 10 TB) will be queried only once in a while. I was hoping that druid will allow some sort of an on-demand cache behavior for querying this data, where historical nodes could be told to load this data by the coordinator based on an incoming query and the data will time out after a configurable time (for example when the coordinator applies the rules next) from the historical node segment cache.

Given that this is not the case what is the recommended deployment configuration for dealing with this sort of usage pattern? 

Loading all the data on historical nodes local storage is not a good option given the cost of local storage vs deep storage. Sounds like that is required? Tiering historical node clusters could help a bit but doesn't quite address the issue.

thanks,

rahul

--
You received this message because you are subscribed to a topic in the Google Groups "Druid User" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/druid-user/B7Mr2I7qcSY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to druid-user+unsubscribe@googlegroups.com.

To post to this group, send email to druid...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/CACZNdYCRZcDbPjpoxHuoZ6fQ5PeddNJ_EKdc83LsxPctQRW6Rw%40mail.gmail.com.

[Kyle Boyle]

Druid attempts to address this use case with tiers of historical nodes. Add a couple of cheap nodes with slow cheap storage for your really old data on a slow tier.

Load Rules > http://druid.io/docs/latest/operations/rule-configuration.html

Kyle

On Tuesday, February 20, 2018 at 3:57:53 PM UTC-4, Rahul Aggarwal wrote:

Hi Gian,

Thank you for the response.

I was hoping that druid can be configured to load data on historical nodes on demand. 

My use case is one where we will have > 10TB of data over a year and will accumulate year over year. Last month's data (~ 1TB) needs to be available for querying on a fairly regular basis and hence needs to be loaded in advance on the historical nodes.

The rest of the data (> 10 TB) will be queried only once in a while. I was hoping that druid will allow some sort of an on-demand cache behavior for querying this data, where historical nodes could be told to load this data by the coordinator based on an incoming query and the data will time out after a configurable time (for example when the coordinator applies the rules next) from the historical node segment cache.

Given that this is not the case what is the recommended deployment configuration for dealing with this sort of usage pattern? 

Loading all the data on historical nodes local storage is not a good option given the cost of local storage vs deep storage. Sounds like that is required? Tiering historical node clusters could help a bit but doesn't quite address the issue.

thanks,

rahul

On Mon, Feb 19, 2018 at 11:37 PM, Gian Merlino <gi...@imply.io> wrote:

Hi Rahul,

I think you are missing one thing: data that is not loaded in advance on historical nodes cannot be queried. So in your scenario, where a query is made for a time range that has been dropped, what happens is you will just get no results at all.

Gian

On Sat, Feb 17, 2018 at 12:10 PM, Rahul Aggarwal <rahul....@gmail.com> wrote:

I will appreciate any help with the following question on a specific druid usage pattern.

The pattern is as follows.

Deep storage is setup as S3; druid coordinator rules are setup to load data for the latest one month and drop older data; a query is then made for data that is older than one month.

In this scenario how will the druid broker know which historical nodes to forward the query to and further how will it take into account the current capacity of the historical nodes?

I have gone through the documentation, the druid white paper and previous postings in this google group, but am not clear about the behavior.

Much thanks,

rahul

--
You received this message because you are subscribed to the Google Groups "Druid User" group.

To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.

To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/cbe8d632-7c38-4ac8-a4c7-8b8dcb205c35%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Druid User" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/druid-user/B7Mr2I7qcSY/unsubscribe.

To unsubscribe from this group and all its topics, send an email to druid-user+...@googlegroups.com.

To post to this group, send email to druid...@googlegroups.com.

[santosh yadav]

Hi Kyle,

We have same use case that Rahul mentioned. I do not see how slow tier would help, if slow tier nodes have to store data locally and cannot rely on deep storage to retrieve segments on-demand. Am I missing anything? Thanks for the help.

[Kyle Boyle]

Santosh,

I was merely trying to point out that you can configure a slow tier with different hardware specs to serve less important data. You are right that you can't answer queries directly from deep storage.

EG for historicals in the slow tier, you could purchase the cheapest local disk type available.

Kyle

[santosh yadav]

Thanks Kyle. I wonder if it is something that Druid would like it to be a feature? We won't mind to do the DEV work if it fits in Druid product roadmap. Would be great to hear comments from other Druid committers / maintainers. Thanks. 

[Gian Merlino]

IMO, this feature makes sense in theory but would be challenging for some reasons,

1) Druid's query operators are all implemented to work on memory-mapped data, and it would be a big effort to have them work on streams of data from deep storage.

2) You could imagine instead, having Druid download the deep storage data locally, memory map it, query it, and then delete it, but this would probably have a lot of overhead.

3) You could also imagine doing a cache for (2), which actually might work although there could be a lot of thrashing.

If one of these three approaches is an exciting feature for you I encourage bringing it up on the Druid dev list (d...@druid.apache.org). You can subscribe by emailing dev-su...@druid.apache.org.

[Santosh Yadav]

Got it, thanks. Does metadata store keep track of segments on deep storage? I admit I don’t fully understand how data on deep storage is tracked and how indexing works in Druid, but a naive thought would be to keep track of segment locations and indexes of data stored on deep storage and load only that data when a query needs to access that data. It will probably overload meta store and may have to have a different meta store for deep storage. Please feel free to correct me if I got it completely wrong. Thanks.

Sent from my iPhone

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/c412a0dd-2f53-4398-b4d0-b515a34cf30c%40googlegroups.com.

[Muthu Lalapet]

Hi Santosh,

Yes Druid metadata store keeps track of the segments in the deep storage. However for druid to query this deep storage during the query time will add a lot of overhead a) reading from deep storage b) memory map it c) query the same and d) then delete the local segment file. These would add a lot of latency to the query. Like Gian pointed out this implementation could be very challenging.

Regards,

Muthu Lalapet.

To unsubscribe from this group and stop receiving emails from it, send an email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/c412a0dd-2f53-4398-b4d0-b515a34cf30c%40googlegroups.com.

[Gian Merlino]

Hey Santosh,

Yes, Druid keeps track of segment locations on deep storage using the metadata store. It could in theory use this to load data on demand. You could imagine putting Druid Historicals in a mode where they "swap" segments in and out of deep storage rather than downloading everything locally ahead of time. This would be like my suggestion (3) above. If it didn't thrash too much it could work.

Btw, one reason we haven't implemented this yet is that the thought was that storage is cheap, and compute is expensive, and most of the cost of running Druid Historicals is compute anyway. So it shouldn't be too costly or burdensome to require enough disk to store any of the data you might want to query. This assumption might be invalid if there is a _lot_ of data that you are not normally querying, but would do so from time to time.

If you want to continue this discussion I definitely suggest bringing it to the dev list. It's the most appropriate forum for discussing future evolutions of Druid's capabilities.

Gian

To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+unsubscribe@googlegroups.com.

[Santosh Yadav]

Thanks Gian!

Sent from my iPhone

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/6470b4f9-a78f-43e6-81d9-867aa3fe0f1e%40googlegroups.com.

[Santosh Yadav]

Thanks Muthu!

Sent from my iPhone

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/e59e8a39-5ae2-4891-bf33-43736750c55c%40googlegroups.com.