Counter increase graph and totals

[Nick]

Hi, hope you're doing great.

I have a counter metric like:

mercury_site_views_total{site="A",node="n2"} 14

mercury_site_views_total{site="B",node="n5"} 31

mercury_site_views_total{site="C",node="n1"} 9

Metrics are exported for multiple sites, and multiple nodes within each site. Metrics are scraped every 5 minutes. All metrics reset (blank) once a day.

Now, I'm trying to build Prometheus queries via Grafana for two things:

1) Graph showing total views for each site within the selected time range - query (in non-Instant mode) I've got seems to be fine I guess:

sum by (site) ( increase(mercury_site_views_total[6m]) > 0 or (mercury_site_views_total != 0 unless mercury_site_views_total offset $__interval) )

2) Table listing total views for each site and node combination within the selected time range - query (in Instant mode) I've got doesn't seem to be working (missing some metrics):

sum by (site, node) ( increase(mercury_site_views_total[$__range]) > 0 or (mercury_site_views_total != 0 unless mercury_site_views_total offset $__range) )

I'll really appreciate if someone can review and suggest improvements for these queries. I've been struggling with getting them right for the last two days, but without luck.

Thank you!

[Brian Candler]

> All metrics reset (blank) once a day.

Why?  These are not true counters, neither are they gauges.

Can you turn them into true counters?  That is, counters which only increment but don't reset?  You will get more accurate answers to queries that way.

[Nick]

Thanks for your reply.

> Can you turn them into true counters?  That is, counters which only increment but don't reset? 

Unfortunately, counter metrics can't be persisted due to dynamic nature of node labels, and storage restrictions in our case.

> You will get more accurate answers to queries that way.

Query results for our use case don't have to be completely accurate. We can work with +/-10% variance. I realized that increase() may not be suitable, specially for summed-up Table data, but I'm wondering if max_over_time() may be more suitable?

Any tips or suggestions are appreciated.