Getting value of gauge at day x and time y

[Oskar Oskarsson]

Hi,

I am currently creating some metrics for a legacy application. I cannot (sadly) change the code, I can only query the database to get some information about the state of the application. So I created a small "proxy application" which queries the database and exposes the results as prometheus metrics.

I have one query which returns me two values A and B which are two related object types. Over time, objects of type A will become B, so the metric here is a gauge.

Currently I'm exporting

object{type="A"} 123

object{type="B"} 456

My problem is that the object counts are only valid per day. So on the next day, I'll get values from A and B, but they are not related to the values of yesterday.

So when the value of A at 23:59 is 100, and at 00:02 it is 120, then it did not increase by 20, in fact it got is more like it turned to 0 at 00:00 and then increased to 120.

What I need:

I need an alert which fires if the value of A / B is under a threshold after 3:00 a.m. I also need to know the "end" values of each day to compare todays values with yesterdays and so on.

I tried to fullfill this by adding a "date" tag like this:

object{type="A", date="2020-10-01"} 123

object{type="B", date="2020-10-01"} 456

object{type="A", date="2020-10-02"} 10

object{type="B", date="2020-10-02"} 4

On each day change, I set the "old" metrics to 0.

So max_over_time(object{date!=""}[5d]) will give me a nice overview here.

However, the approach makes alerting hard as prometheus might fetch the wrong (zeroed) metrics. I solved this application wise by unregistering the old metrics.

All of this feels terribly wrong to be honest. Is there a better way to solve this?

Thanks in advance!