Hisorical data

[BHARATH KUMAR]

Hello All,

I deleted some instances from the prometheus.yml file. But still, I am able to see those servers' historical data. What should I do so that those instances which I removed, those instances historical data should not have appeared?

Example:

I am finding the unreachable servers for 30 days

 Query : max_over_time(up{instance=~"$instance",job="node_exporter"}[30d]) == 0

There are some servers that were unreachable for the last 30 days. Among these servers, I deleted a few of them from prometheus.yml. But still, I am able to see those servers in the output. 

What should I do so that I didn't get these servers' details?

Thanks & regards,

Bharath Kumar.

[Brian Candler]

https://www.robustperception.io/deleting-time-series-from-prometheus/

However, it's still true that you *were* monitoring those servers, and this history may be useful  You don't have to throw away the data.  Just limit your query like this:

max_over_time(up{instance=~"$instance",job="node_exporter"}[30d]) == 0 and up

"up" is a metric that's always present for any scrape job, and absent if not.  So this query will filter out anything which is not actively being scraped (within the last 5 minutes)

[BHARATH KUMAR]

Query:  up{instance=~"$unreachbale_instance"} and up

This query is giving me some instances which were already removed from promethues.yml

Is there any mistake I am doing?

[Stuart Clark]

On 26/09/2022 13:10, BHARATH KUMAR wrote:

Query:  up{instance=~"$unreachbale_instance"} and up

This query is giving me some instances which were already removed from promethues.yml

That's expected.

The jobs in prometheus.yaml and the targets referenced are just what Prometheus should scrape "now". You can add and remove targets whenever you want, which will change what new data is collected. However it has nothing to do with the ongoing storage of data within the timeseries database. The main configuration for local storage is the number of days (or total storage) values, which default to storing 14 days of data. So by default if you stop scraping something today it would continue to exist (with all previous metrics) for another two weeks. Note also that any remote read system could also continue to return data depending on how that external system was configured.

If you have something stored in the local Prometheus store (rather than something you are using remote read with) and don't want to wait until it expires (14 days or whatever you configured) you can use the deletion API.

-- 
Stuart Clark

[BHARATH KUMAR]

Thanks for your reply.

If I use deletion API will it delete all the stored data or will it delete the instances data I deleted in the Prometheus.yml file.

Thanks & regards,

Bharath Kumar.

[Stuart Clark]

On 26/09/2022 14:33, BHARATH KUMAR wrote:
> Thanks for your reply.
>
> If I use deletion API will it delete all the stored data or will it
> delete the instances data I deleted in the Prometheus.yml file.

It will delete whatever you tell it to delete:

https://prometheus.io/docs/prometheus/latest/querying/api/#delete-series

--
Stuart Clark

[Brian Candler]

On Monday, 26 September 2022 at 13:10:57 UTC+1 chembakay...@gmail.com wrote:

Query:  up{instance=~"$unreachbale_instance"} and up

This query is giving me some instances which were already removed from promethues.yml

Is it still the same after waiting at least 5 minutes?