get unique list of instances from prometheus api

[Richie]

I would like to pull a line by line list of unique instances from agent_build_info and save it to a file for the purpose of cross checking via script against a list of known instances that should be monitored (reporting purposes).

so in short is it possible to create a query that equivalent of select distinct instance from agent_build_info --(were it a traditional table) ?

-Richie

[Richie]

or i suppose i could process the results as they come from the api via python

[Brian Candler]

If you are happy to list all the labels of interest (i.e. everything apart from the 'job' and 'instance' labels and any other labels you've added yourself):

count by (goversion,version,branch,revision) (agent_build_info)

Or you can use "group by" instead of "count by" to get a value of 1 for everything (although personally I think the counts are useful!)

[Brian Candler]

Aha, I just discovered you can do:

count without (instance, job) (agent_build_info)
group without (instance, job) (agent_build_info)

That should be what you want :-)

[Richie]

Brian your suggestions ultimately helped as it introduced me a new concept.  I don't think I conveyed what I wanted properly.   Put another way I was looking for  'a list of unique values for the key "instance".  Still not sure if that's the best way to articulate it.

Based on your suggestions this is what I came up with

group by (instance) (agent_build_info)

Ideally the port ':12345' could be stripped, but what is generated from the above is plenty sufficient.

Thanks for the help.

[Richie]

came up with the following

label_replace(group by (instance) (agent_build_info), "instance", '$1', "instance", '(.+):12345')

[Brian Candler]

Glad it's working. A couple of ideas:

(1) You said "Put another way I was looking for  'a list of unique values for the key "instance""

The simplest way to do that is to query 'up' or 'up{job="foo"}'.  This is a metric synthesised by Prometheus with one entry for every scrape, and hence every job/instance combination.  It also tells you whether the scrape was successful or not.

(2) In general, it's a good idea to keep the port number out of the instance label in the first place, even though Prometheus puts it there by default if you don't set the instance label explicitly.

Apart from making dashboards and the like clearer, it makes it much easier to write join queries which combine the results of different exporters on the same host.

https://www.robustperception.io/controlling-the-instance-label
https://groups.google.com/forum/#!topic/prometheus-users/I4dVtwI5YrM
https://groups.google.com/forum/#!msg/prometheus-users/PPwF-2CQiSc/FsSvfCm3AwAJ