How to query a pushgateway for metrics via Python?

[Peter Toft]

I have a case where I probably have to push say 200 metrics from Python to a pushgateway->prometheus->grafana system - it rocks,
However the 200 metrics might later be reduced to 170 metrics, where at that point I need to query the pushgateway for the obtained metrics - and delete the 30 ones that we did not want any more. They day after it might be 10 that I need to remove. And add a few others.
I can manually go to the pushgateway and press "Delete Group" at http://localhost:9091/# - it works, but I really like to

• automate
• handling done from Python

Anyone who has examples for the following - preferably within Python

• Obtain a Python list or dict of metrics -> "my_metrics"
• Query the pushgateway for the known metrics -> "pg_metrics"
• Traverse the "pg_metrics" and delete the metrics that are not in "my_metrics"

There are discussions on something similar e.g. in
https://www.metricfire.com/blog/prometheus-pushgateways-everything-you-need-to-know/
prometheus/pushgateway#77

[Peter Toft]

A variant that I could use - in lack of better - would be to delete all metrics from Python

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9208f69f-095f-481c-9110-6c7c3a34da61n%40googlegroups.com.

--

Peter Toft <peter...@gmail.com>

[Stuart Clark]

On 18/11/2020 21:31, Peter Toft wrote:

A variant that I could use - in lack of better - would be to delete all metrics from Python

Den ons. 18. nov. 2020 kl. 11.46 skrev Peter Toft <peter...@gmail.com>:

I have a case where I probably have to push say 200 metrics from Python to a pushgateway->prometheus->grafana system - it rocks,
However the 200 metrics might later be reduced to 170 metrics, where at that point I need to query the pushgateway for the obtained metrics - and delete the 30 ones that we did not want any more. They day after it might be 10 that I need to remove. And add a few others.
I can manually go to the pushgateway and press "Delete Group" at http://localhost:9091/# - it works, but I really like to

• automate
• handling done from Python

Anyone who has examples for the following - preferably within Python

• Obtain a Python list or dict of metrics -> "my_metrics"
• Query the pushgateway for the known metrics -> "pg_metrics"
• Traverse the "pg_metrics" and delete the metrics that are not in "my_metrics"

There are discussions on something similar e.g. in
https://www.metricfire.com/blog/prometheus-pushgateways-everything-you-need-to-know/
prometheus/pushgateway#77

As per the API documentation you can delete a metric - https://github.com/prometheus/pushgateway#delete-method

You can also wipe all data - https://github.com/prometheus/pushgateway#admin-api

[Peter Toft]

Thanks Stuart. 

When I read these sections I thought that these operations require curl and the command line.

I can certainly use this, but it is not Python functions that can be used directly. Did I miss

something?

Best

Peter

--

Peter Toft <peter...@gmail.com>

[Stuart Clark]

On 21/11/2020 13:05, Peter Toft wrote:

Thanks Stuart. 

When I read these sections I thought that these operations require curl and the command line.

I can certainly use this, but it is not Python functions that can be used directly. Did I miss

something?

If you are using the Python client library there is the delete_from_gateway method, but otherwise you can just use the requests library (or urllib) to call the API - the calls are just standard HTTP(s) requests.

Best

Peter

Den tor. 19. nov. 2020 kl. 18.52 skrev Stuart Clark <stuart...@jahingo.com>:

On 18/11/2020 21:31, Peter Toft wrote:

A variant that I could use - in lack of better - would be to delete all metrics from Python

Den ons. 18. nov. 2020 kl. 11.46 skrev Peter Toft <peter...@gmail.com>:

I have a case where I probably have to push say 200 metrics from Python to a pushgateway->prometheus->grafana system - it rocks,
However the 200 metrics might later be reduced to 170 metrics, where at that point I need to query the pushgateway for the obtained metrics - and delete the 30 ones that we did not want any more. They day after it might be 10 that I need to remove. And add a few others.
I can manually go to the pushgateway and press "Delete Group" at http://localhost:9091/# - it works, but I really like to

• automate
• handling done from Python

Anyone who has examples for the following - preferably within Python

• Obtain a Python list or dict of metrics -> "my_metrics"
• Query the pushgateway for the known metrics -> "pg_metrics"
• Traverse the "pg_metrics" and delete the metrics that are not in "my_metrics"

There are discussions on something similar e.g. in
https://www.metricfire.com/blog/prometheus-pushgateways-everything-you-need-to-know/
prometheus/pushgateway#77

As per the API documentation you can delete a metric - https://github.com/prometheus/pushgateway#delete-method

You can also wipe all data - https://github.com/prometheus/pushgateway#admin-api

--

Peter Toft <peter...@gmail.com>

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAAW%2Bni-jeS1tB-jTJ%2B-V6z-0%2B%3DyqA5nmQun5LkGOpY-FdY9KpA%40mail.gmail.com.

[Chris Siebenmann]

> I have a case where I probably have to push say 200 metrics from
> Python to a pushgateway->prometheus->grafana system - it rocks,
> However the 200 metrics might later be reduced to 170 metrics, where
> at that point I need to query the pushgateway for the obtained metrics
> - and delete the 30 ones that we did not want any more. They day after
> it might be 10 that I need to remove. And add a few others.

If all of the metrics are pushed at the same time by the same program,
there is an easy way to do this automatically with Pushgateway. What you
want to do is to publish to a unique (and constant) group, and then use
PUT instead of POST to publish. When you use PUT to publish metrics to a
Pushgateway group, Pushgateway removes all other metrics in that group
for you, regardless of their names.

If you have multiple Python bits that must publish separately, some of
which don't always generate metrics, you can do a variant of this. Each
Python bit publishes to a separate Pushgateway group with PUT, and if
they have no metrics to generate this time, they PUT an empty set of
metrics (or a single dummy metric to say 'I ran'), which will delete
their past metrics.

You can extend this to automatically clean out Pushgateway groups that
haven't been pushed to recently enough with some scripting. First, make
a query to Prometheus for some metric that reflects it being stale, such
push_time_seconds values that are sufficiently old (and fall within the
groups that you want to purge). Then, run through each group and either
PUT an empty update to it or use DELETE to delete it entirely.

- cks
PS: For more discussion of the PUT versus POST difference with
Pushgateway, you can read
https://utcc.utoronto.ca/~cks/space/blog/sysadmin/PrometheusPushgatewayDropMetrics