Prometheus query in python

[Sesha kumari]

Hello All,

I want to write Prometheus query in python. I want to store the output of unreachable instances in a variable.

I found one blog where we can fetch all metrics. But I want to write query, but I am not able to write. Could you please help me out?

I want to write a query to find the unreachable servers in python.

Here is the code which I found:

import datetime
import time
import requests  

PROMETHEUS = 'http://localhost:9090/'

end_of_month = datetime.datetime.today().replace(day=1).date()

last_day = end_of_month - datetime.timedelta(days=1)
duration = '[' + str(last_day.day) + 'd]'

response = requests.get(PROMETHEUS + '/metrics',
  params={
    'query': 'max_over_time(up{instance=~"$unreachbale_instance",job="node_exporter"}[$interval]) $reachability and up',
    'time': time.mktime(end_of_month.timetuple())})
results = response.json()['data']['result']

print('{:%B %Y}:'.format(last_day))
for result in results:
  print(' {metric}: {value[1]}'.format(**result))

could anyone please help me?

Thanks & regards,

Bharath Kumar.

[Brian Candler]

You need to read the Prometheus HTTP API documentation, then get it to work using curl, before you write your Python version.

Your first problem is that you are querying the '/metrics' endpoint of prometheus.  That only gives you metrics about prometheus itself - internal information about the performance of its time series database for example.  There is a separate endpoint for querying metrics data which is stored in the database.

To check for "unreachable instances", I suggest you start with a simple query like up==0.  Do this using an instant query.  Example:

curl -gsS 'http://localhost:9090/api/v1/query?query=up==0'

You can then make this query more sophisticated to meet your requirements, and when you're happy with it, convert into Python.

Your second problem is that you are putting nonsense in your queries like "$unreachbale_instance" [sic] and "$interval".  I am guessing you've been doing stuff like this in Grafana?  But Grafana substitutes these placeholders with real values *before* sending the query to Prometheus' API.  If prometheus sees these $ values then it will treat them literally as those strings, and in particular "$interval" is not a valid value for a range vector selector.  To see what syntax is allowed in a query, read the query documentation, and test your queries out in the Prometheus web interface.

I note that you have calculated a suitable value and stored it in a python variable:

    duration = '[' + str(last_day.day) + 'd]'

... but you have not actually used this value in your query anywhere.

[Sesha kumari]

Thanks for your quick around. Could you please share any reference? I am not able to understand the flow to follow

[Brian Candler]

My post includes hyperlinks to several relevant pieces of documentation. If you can't see them, then please view the post online at https://groups.google.com/g/prometheus-users/c/uWzjulLOcig/m/SxWeMkMSBAAJ where you should find the links are clickable.

I've also spoon-fed you an exact curl command you can run to send a query via HTTP to the API.  Have you tried running it?  Then have you tried converting it into Python?