Historical exposed metrics memorization

[Daniele Pardo]

Hello,

I found out the follow unexpected beahviour between my prometheus db and customized client_python exporter:

I've tried to delete metrics from prometheus db using follow Web API commands:

• curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]=<name_metric>
• curl -X POST -g "http://localhost:9090/api/v1/admin/tsdb/clean_tombstones

All the time series metric diseppeared from prometheus db but after few seconds, probably after the new scraping iteration, all the time series was restored. (I tried with a two hour time series, very short).
So, the questions is: was the timeseries SAVED in the client_python exporter (buffer, cache, registry etc) and re-exposed every time to prometheus or the problem is at prometheus db side?

Is it a configurable option?

Thank you,
Daniele.

[Brian Candler]

Whenever prometheus scrape an exporter, it automatically creates any timeseries which don't exist (i.e. unique combination of metric name + labels)

Therefore if your exporter is continuing to expose those timeseries - which you can easily check with a curl scrape - they will be recreated.

Things like gauges and counters set in a prometheus client library will normally persist unless you delete them.  Otherwise: (a) they wouldn't work when being scraped by multiple prometheus servers, and (b) they would not be present when the next scrape comes along - which would result in the timeseries going "stale".  Remember that the prometheus client library does not know how often it's going to get scraped, and by whom.

[Daniele Pardo]

So, it is necessary to delete the timeseries not only at prometheus db side but also at prometheus client exporter side otherwise they will be recreated.
I am supposing the prometheus client exporter saved only the last value for each metric (gauges, counter or others..) until it was not updated with a new set value while,  actually all the values previously exposed persist on the prometheus client.

Right?

[Brian Candler]

Yes.  Use "curl" to hit the exporter URL directly, to see what's going on.

[Daniele Pardo]

I did it.

I used curl <address_of_exporter> and i see only the last exposed value for each metric. So, I don't understand how is possible that when I delete a timeseries in prometheus db, during the following scraping iteration prometheus db will be able to scrape also the historical metrics values (that should previosly been deleted from server) and not only the last metrics value in real time. 

[Brian Candler]

On Wednesday, 8 April 2020 13:31:49 UTC+1, Daniele Pardo wrote:

I used curl <address_of_exporter> and i see only the last exposed value for each metric. So, I don't understand how is possible that when I delete a timeseries in prometheus db, during the following scraping iteration prometheus db will be able to scrape also the historical metrics values (that should previosly been deleted from server) and not only the last metrics value in real time. 

I wouldn't expect that.  Are you sure the delete happened? What response do you get from the curl which deletes the series?  (Add -v to curl to see the result code)