Compare current metric value and the previous one

[aistl...@gmail.com]

Hello,

We discussed my problem with grobie in https://riot.im/app/#/room/#prometheus:matrix.org and he said that I can write it here.

So our usecase is the following:

We're collecting metrics with e.g. 15-seconds interval.

If current value of the metric is not equal to the previous one, we need to send a notification to an external system about that. So this notification should contain current metric value and some set of labels.

We've thought about two ways of solving this problem:

1. Alert manager rules.

Theoretically it's possible to write rules which conditions will be true if e.g. delta(metric_name[15s]) != 0. Then it would possible to send a notification from alert manager side.

But here is at least two problems.

• $value macro would be equal to this aggregated metric value rather than to the source value. So I don't know how to send current metric value.
• scrape interval can be changed from time to time and it seems that in our case it's quite difficult to update alert rules in time.

Also I don't think that alerting is a good way to do this feature...

2.   remote_write

Send metric values to our own web service (e.g. java application) and store required metrics in our own DB or cache. So if new value is not equal to the previous one, we can send a notification.

We thought that remote_write is a tool which can be used for metric sending via http. But it uses application/x-protobuf content type so it seems that we couldn't get metrics as a simple json... Also grobie has said

using the remote storage interface certainly sounds like overkill

but honestly I don't understand why.

So what would be the best solution for our use case? Or prometheus is not intended for previous and current metric values comparison?

Thanks in advance!

[Conor Broderick]

I'm not very experienced with this particular topic but I came across a Stack Overflow question containing an answer that appears to solve your issue of including the the actual metric value in your alert.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f52fffd0-08d0-4a44-ab43-d7f071d334cc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Conor Broderick

www.robustperception.io