Getting rolling Aggregates in Druid

[Surya Vijayaraghavan]

Hey guys,

For the data I have currently, I answer a use case like - Give me the number of page views on an hourly basis. 

The result I get is

TimeStamp                    | Page View

===========================

2018-07-18T00:00:00Z  |  10

2018-07-18T01:00:00Z  |  5

2018-07-18T02:00:00Z  |  0

2018-07-18T03:00:00Z  |  15

The usecase that I want to solve is - Give me the total number page views till that hour. That means, I want to do an overall aggregate of my page views. So the result should look like - 

TimeStamp                    | Page View

============================

2018-07-18T00:00:00Z  |  10

2018-07-18T01:00:00Z  |  15

2018-07-18T02:00:00Z  |  15

2018-07-18T03:00:00Z  |  30

Is there any way I can achieve this? 

[Tan Huynh]

This is a great question.  I have a similar use case where I need to calculate the cumulative sum of a metric.    

[Pratik Khadloya]

+1 I have a similar use case.

[Atul Mohan]

Hello,

We're working on releasing a custom query type called RollingAverage query as a separate extension into the community. This extension should be able to solve your usecases.

Thanks,

Atul

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/9bab1bfe-46b2-4e10-869a-c1bfa6f54f6f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Atul Mohan

[Surya Vijayaraghavan]

Thanks Atul! 

Please let me know if you can point me to a repo or whether I can contribute it. 

[Pratik Khadloya]

Thats great to know! Thanks a lot!

[no...@singular.net]

That might be really useful to us as well! we'd love to hear additional information and possibly contribute.

[Charles Allen]

Thanks Atul!

Have you considered doing a org.apache.druid.query.PostProcessingOperator instead of a whole new query type?

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/3c56b49c-c42c-480d-9fbb-1e091abe5b99%40googlegroups.com.

[Eyal Yurman]

Hi Charles,

Our implementation doesn't use PostProcessingOperator but is leveraging the groupBy query internally.

We've been running this query type for a while in production and I'm glad to see there is a demand for it in other companies,

I'll try to open the pr this week.

[Tan Huynh]

Hi Eyal,

Can you describe the algorithm that you are using?   Are you using nested groupby to get the previous segment total?

Thanks

Tan

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/7ccc1907-44ff-49b5-b67a-fd838f11d43a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Tan Huynh

[Eyal Yurman]

Hi,

I've opened a new issue so you can review the concept of what we implemented:

https://github.com/apache/incubator-druid/issues/6320

I'm working on the user-guide (documentation) and hope to submit a pr later this week.

On Wednesday, August 8, 2018 at 10:01:38 AM UTC-7, Surya Vijayaraghavan wrote:

[Eyal Yurman]

(sorry for the multiple replies)

Hi,

I've opened a new issue so you can review the concept of what we implemented:

https://github.com/apache/incubator-druid/issues/6320

I'm working on the user-guide (documentation) and hope to submit a pr later this week.

[Eyal Yurman]

Hi, I published a pull request:

https://github.com/apache/incubator-druid/pull/6430

The original issue:

https://github.com/apache/incubator-druid/issues/6320

[Pratik Khadloya]

Thanks a lot Eyal! Does this work with hyperUnique or Theta Sketch columns?

[Eyal Yurman]

Yes, it works with Theta Sketch columns.

On Thu, Oct 11, 2018 at 10:24 AM Pratik Khadloya <tisp...@gmail.com> wrote:

Thanks a lot Eyal! Does this work with hyperUnique or Theta Sketch columns?

--

You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/cb8ff3e4-1c61-4df2-8bc5-1d285b3ebc37%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Best regards,

Eyal Yurman
+972-54-3056315