backfill recording rules creates duplicate metrics on 2 hour border

[Jonathan Tougas]

I seem to be getting overlapping/duplicated metrics when trying to backfill recording rules. Working off of this article: https://jessicagreben.medium.com/prometheus-fill-in-data-for-new-recording-rules-30a14ccb8467

my recording rule is:
```
      - record: s8_es_syslog_query_count_total5:rate5m
        expr: rate(s8_es_syslog_query_count_total{query_name="all"}[5m])
```

I create blocks and copy them into a prometheus server. Then there are overalaping/duplicate metrics in the prometheus ui, always on a 2 hour border, but not always every 2 hours.
Here's an example overlap in a text friendly format:

$ promtool query range http://10.10.10.1:9090 's8_es_syslog_query_count_total5:rate5m{instance="10.10.10.59"}' --start=1626998350 --end=1626998450 --step=50s
s8_es_syslog_query_count_total5:rate5m{instance="10.10.10.59", job="es-syslog-query", name="server1", query_name="all", roles="foo"} =>
0.008333333333333333 @[1626998400]
0 @[1626998450]
s8_es_syslog_query_count_total5:rate5m{instance="10.10.10.59", job="es-syslog-query", name="server1", query_name="all", roles="foo"} =>
0.008333333333333333 @[1626998350]
0.008333333333333333 @[1626998400]
0.008333333333333333 @[1626998450]

So the same labelset is returned twice at timestamps ending in 400 and 450 (with differing values at 450 to boot).

This seems somewhat random in that the overlap does not occur at the same place if I run the same data through the same procedure a second time.

Here's a screenshot of the problem as seen in the default prometheus ui.

These tests were performed using prometheus and promtool from the

prom/prometheus:v2.29.1 docker image.

Is this a bug? Am I holding it wrong?

[Jess G]

Thanks for the bug. At first glance I'm not sure why this would be happening. Let me investigate and I will get back to you.

[Jess G]

This may be similar to this issue: 

https://github.com/prometheus/prometheus/issues/9288

Going to try to reproduce today and debug.

[Jonathan Tougas]

Thank you! If I can help in any way let me know.