Storing repaired rollups on read

[BalaKrishna Mumdluri]

I have situation where the rollups are processed far behind the ingestion of metrics. So the query for metrics till recent time stamp trying to repair rollups on read to fill the missed data points from raw tables. This may be good if the query for a metric is executed once or twice, but if it has be executed repeatedly repair on read will be an additional latency to serve data points to client. So we have decided to see if the repaired rollups on read can be saved in corresponding table. Fortunately it is quite easy to trigger rollup for a metric for requested granularity.

Let us say if the query for metrics of 20 min granularity has some missed data points and they got repaired during read and persisted only in 20min granularity table. Does it has any implication on actual rollup service? Any pointers would be appreciated.

[Chandra Addala]

I think when the actual rollup happens, they data just gets overridden. But I dont know if I would persist rollups on read data. When the query is executed repeatedly, there is a chance there are different number of data points every time we do rollup on read, as the data is probably still coming thru. So I dont know what benefit u get by persisting.

[BalaKrishna Mumdluri]

When actual rollup happens if it just overrides the existing one, that's fine. As long as it does not affect the actual rollup and it's state that is okay. 

Storing would benefit in a sense that repair on read (is costly as it reads data from metrics_full table by splitting the requested time range into equal time ranges based on requested granularity, so involves many calls to database.) We want to avoid so many calls to database where latency to show chart on GUI is key.