need a help with PromQL

[Dmitry Kundo]

Hello,

I'm using process-exporter (https://github.com/ncabatoff/process-exporter) for monitoring running processes on hundreds of servers. There is a known problem that if the process exporter hasn't seen a certain process since the exporter started, it won't report it as "missing" - actually there will be no metric at all for this process.

So the alert:  namedprocess_namegroup_num_procs(groupname="AAA") == 0 won't fire for such servers.

Found a solution suggesting using absent function for checking this situation but I'm not sure how to make it work because I don't understand how to express the relevant labels set. Each such metric also has "server_id" label which identifies a specific server.

What I need is to express this:

namedprocess_namegroup_num_procs(groupname="AAA") == 0 or  absent( namedprocess_namegroup_num_procs (groupname="AAA", server_id="???????")   

The monitored servers are a dynamic group (EC2 auto-scaling instances), so I don't know all possible servers in advance,  and anyway there are hundreds of them.

Regards,

Dmitry

[Brian Brazil]

On Thu, 21 Oct 2021 at 08:51, Dmitry Kundo <dku...@gmail.com> wrote:

Hello,

I'm using process-exporter (https://github.com/ncabatoff/process-exporter) for monitoring running processes on hundreds of servers. There is a known problem that if the process exporter hasn't seen a certain process since the exporter started, it won't report it as "missing" - actually there will be no metric at all for this process.

So the alert:  namedprocess_namegroup_num_procs(groupname="AAA") == 0 won't fire for such servers.

Found a solution suggesting using absent function for checking this situation but I'm not sure how to make it work because I don't understand how to express the relevant labels set. Each such metric also has "server_id" label which identifies a specific server.

What I need is to express this:

namedprocess_namegroup_num_procs(groupname="AAA") == 0 or  absent( namedprocess_namegroup_num_procs (groupname="AAA", server_id="???????")   

https://www.robustperception.io/absent-alerting-for-scraped-metrics is the technique you want here. So

up{job="myjob"} unless ignoring(groupname, server_id) namedprocess_namegroup_num_procs{job="myjob",groupname="AAA"} > 0

Brian

 

The monitored servers are a dynamic group (EC2 auto-scaling instances), so I don't know all possible servers in advance,  and anyway there are hundreds of them.

Regards,

Dmitry

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CADxb1xW4tVzCq5SMtXCUAF7%2BCdCuGffLZmsRSNVqM1KOBaO5Kw%40mail.gmail.com.

--

Brian Brazil

www.robustperception.io