targetmissing alert is fooled by two endpoints on one job

[Mario Cornaccini]

hi,

i have a job 'node-tools'ansible',

with two endpoints.. one for each node- and shell exporter.

in prometheus/targets i see the same set of labels for each endpoint.

for testing i stoppped the node exporter, but the alert is based on the following expr:
up{} == 0

on the graph i can see that the up{}==0 expr has value 0 for a few seconds, then gaps;

when i remove the ==0 i can see it goes from 0 to 1.

so it seems to me that the other (shell) exporter mixes into the up metric;

and that is because the endpoint have the same labels, right ?

so in my scrape defintion i need to specify one differing label for the shell exporter..

i could make an exporter label, setting it to 'node'/'shell' i guess..

or how do you guys handle that?

TIA; cheers,

fil

[Stuart Clark]

They can't have identical labels. Even if the job label is the same the
instance label should be different.

--
Stuart Clark

[Mario Cornaccini]

in our case we had indeed identical 'instance' labels i believe, because we had relabeling rules in place, rewriting it from the ip-address:port style prometheus does as default to the hostname.
we now changed it back, so the instance label is the unique ip:port style,
and introduced a hostname label; (just for documentation, hope it helps someone)

cheers,

fil