How to delete the historical data

[BHARATH KUMAR]

Hello All,

I installed node exporter in multiple instances. Now I deleted one of those instance(this is unreachable instance) from prometheus.yml file. When I am trying to query up == 0, still I am able to see this instance in the database. 

How to delete that particular instance in database? Any help?

Thanks in advance.

Thanks & regards,

Bharath Kumar 

[Shelten]

Remove the target from prometheus.yml file as well

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/58e88b61-58be-441c-9fa8-a89a534d180dn%40googlegroups.com.

[Michael Kogelman]

The default retention is set to 15d iirc, so once that ages out it wil disappear. You can also use promtool to manually delete out data if you really wish.

--

[BHARATH KUMAR]

I removed it from the prometheus.yml file... I am still getting that instance while querying.

[BHARATH KUMAR]

could you please help me with the exact command?

[Brian Candler]

Use Google. Enter "prometheus delete timeseries", and you will find:

https://www.robustperception.io/deleting-time-series-from-prometheus/

This gives the exact command.

[BHARATH KUMAR]

I enabled this flag (--web.enable-admin-api) and executed this command  curl -XPOST http://localhost:9090/api/v1/admin/tsdb/clean_tombstones 

Still, I am able to see the deleted instances while executing the prometheus query.  

[Brian Candler]

You're not cleaning tombstones, you're deleting timeseries. This means that the endpoint needs to be

/api/v1/admin/tsdb/delete_series?match[]={instance="foo"}

just like the page I linked you to says.

[BHARATH KUMAR]

curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={instance="instance:port"}'

Is the above query correct? If so, I am able to see those instances' values that were removed from the prometheus.yml file. Will it take time to reflect on those changes?

[Brian Candler]

Looks OK to me, if your instance labels are of the form name:port.  It should take effect pretty much instantaneously.

You could add -v to the the curl command line, to see the response code (204 / 400 etc)

Of course, the metrics will reappear if you are still scraping them, which might be because:

- you've changed prometheus.yml without sending a HUP signal to reload the config

- you've changed prometheus.yml, you've sent a HUP signal, but the new config has a syntax error (in which case prometheus will keep running with the old config)

Check with: journalctl -eu prometheus