Do we have to set rollup=false to disable rollup for kafka indexing?

[quangtrung tran]

By default, rollup=true is set for batch indexing. Therefore we need to set rollup=false to disable it. 

How about kafka indexing? is it enable by default and if yes, can we disable it?

[Nishant Bangarwa]

yes, you can specify rollup=false as part of granularitySpec for kafka indexing service also. 

On Thu, 24 Nov 2016 at 09:24 quangtrung tran <tranquang...@gmail.com> wrote:

By default, rollup=true is set for batch indexing. Therefore we need to set rollup=false to disable it. 

How about kafka indexing? is it enable by default and if yes, can we disable it?

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/8d0cb4a6-4f6c-479c-82f9-c9c14c11a8f2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Fangjin Yang]

Note that this only works in 0.9.2+

On Thursday, November 24, 2016 at 1:54:30 AM UTC-8, Nishant Bangarwa wrote:

yes, you can specify rollup=false as part of granularitySpec for kafka indexing service also. 

On Thu, 24 Nov 2016 at 09:24 quangtrung tran <tranquang...@gmail.com> wrote:

By default, rollup=true is set for batch indexing. Therefore we need to set rollup=false to disable it. 

How about kafka indexing? is it enable by default and if yes, can we disable it?

--
You received this message because you are subscribed to the Google Groups "Druid User" group.

To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+unsubscribe@googlegroups.com.

[Linbo Jin]

Hi guys,

I have two questions:

1. So it will not work even if we set rollup=false for kafka indexing service on druid 0.9.2, it will always roll-up rows?

2. About the kafka indexing service roll-up feature: I know "across different taskDurations (tasks), new task will create new segment vertically without modifying old segments created by old tasks if these segments span same granularity time period." Then if one row A is generated in previous taskDuration, which is already pushed to deep storage and a new row B (should roll-up with row A) is generated inside this taskDuration, will kafka indexing service roll-up them? One row is published and another is delayed into current taskDuration.

Best wishes,

By Linbo

On Friday, December 16, 2016 at 7:34:02 AM UTC+8, Fangjin Yang wrote:

Note that this only works in 0.9.2+

On Thursday, November 24, 2016 at 1:54:30 AM UTC-8, Nishant Bangarwa wrote:

yes, you can specify rollup=false as part of granularitySpec for kafka indexing service also. 

On Thu, 24 Nov 2016 at 09:24 quangtrung tran <tranquang...@gmail.com> wrote:

By default, rollup=true is set for batch indexing. Therefore we need to set rollup=false to disable it. 

How about kafka indexing? is it enable by default and if yes, can we disable it?

--
You received this message because you are subscribed to the Google Groups "Druid User" group.

To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.

[Gian Merlino]

Hey Linbo,

1. If you set rollup to false on druid 0.9.2+ then the Kafka indexing service (or any other indexing method) won't roll up rows.

2. Yeah, if that happens, the rows will not be rolled up after stream ingestion. If you want to roll them up, you could do it by running a batch reindexing job on the same interval. In general, Druid streaming ingestion methods don't guarantee perfect rollup at ingestion time. This is fine if you view rollup as a space saving optimization, since it will be applied at query time too.

Gian

To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+unsubscribe@googlegroups.com.

To post to this group, send email to druid...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/394075c1-c6ad-498a-b2b6-a17f5ecb10b2%40googlegroups.com.

[Jason Cheow]

Hey Gian,

"it will be applied at query time too" - just want to clarify that the rollup at query time uses the aggregation specified by the original ingestion spec's `metricsSpec` attribute? And not the aggregation specified by the query?

Regards,

Jason

Gian

[Gian Merlino]

It uses the aggregation specified by the query, although it only "works right" if you specify the same aggregation that was used at ingestion time. By "works right" I mean acts just as if things were fully rolled up.

Gian

To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+unsubscribe@googlegroups.com.

To post to this group, send email to druid...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/64a13b77-fe42-4cd5-ae5a-a19666f2b38f%40googlegroups.com.