Remove metric completely

30 views
Skip to first unread message

Sergey Urushkin

unread,
Apr 14, 2020, 7:18:27 AM4/14/20
to Prometheus Users
Hello,
I'm wondering is there a way to remove metric completely.
After running :
  /api/v1/admin/tsdb/delete_series?match[]={__name__=~"bad_metric_.*"}
  /api/v1/admin/tsdb/clean_tombstones
And even restarting prometheus, they are still appearing in autocompletion in the prometheus and grafana ui (while all values are removed because of the admin-api calls).
Is there a way to clean/refresh/rescan metrics for autocompletion. Or should I file an issue?

Brian Brazil

unread,
Apr 14, 2020, 8:26:52 AM4/14/20
to Sergey Urushkin, Prometheus Users
On Tue, 14 Apr 2020 at 12:18, Sergey Urushkin <urush...@gmail.com> wrote:
Hello,
I'm wondering is there a way to remove metric completely.
After running :
  /api/v1/admin/tsdb/delete_series?match[]={__name__=~"bad_metric_.*"}
  /api/v1/admin/tsdb/clean_tombstones
And even restarting prometheus, they are still appearing in autocompletion in the prometheus and grafana ui (while all values are removed because of the admin-api calls).

That should work, but it didn't always in all versions.
 
Is there a way to clean/refresh/rescan metrics for autocompletion. Or should I file an issue?

This isn't a bug per-se, auto complete and so on can return false positives for various implementation reasons.

--

Sergey Urushkin

unread,
Apr 14, 2020, 8:47:37 AM4/14/20
to Prometheus Users
Forgot to mention:
# prometheus --version
prometheus, version 2.17.1 (branch: HEAD, revision: ae041f97cfc6f43494bed65ec4ea4e3a0cf2ac69)
  build user:       root@806b02dfe114
  build date:       20200326-16:18:19
  go version:       go1.13.9

What builds are not affected then?

вторник, 14 апреля 2020 г., 15:26:52 UTC+3 пользователь Brian Brazil написал:

Brian Brazil

unread,
Apr 14, 2020, 8:49:52 AM4/14/20
to Sergey Urushkin, Prometheus Users
On Tue, 14 Apr 2020 at 13:47, Sergey Urushkin <urush...@gmail.com> wrote:
Forgot to mention:
# prometheus --version
prometheus, version 2.17.1 (branch: HEAD, revision: ae041f97cfc6f43494bed65ec4ea4e3a0cf2ac69)
  build user:       root@806b02dfe114
  build date:       20200326-16:18:19
  go version:       go1.13.9

What builds are not affected then?

That should work then, you're probably still ingesting this metric.

Brian
 

вторник, 14 апреля 2020 г., 15:26:52 UTC+3 пользователь Brian Brazil написал:
On Tue, 14 Apr 2020 at 12:18, Sergey Urushkin <urus...@gmail.com> wrote:
Hello,
I'm wondering is there a way to remove metric completely.
After running :
  /api/v1/admin/tsdb/delete_series?match[]={__name__=~"bad_metric_.*"}
  /api/v1/admin/tsdb/clean_tombstones
And even restarting prometheus, they are still appearing in autocompletion in the prometheus and grafana ui (while all values are removed because of the admin-api calls).

That should work, but it didn't always in all versions.
 
Is there a way to clean/refresh/rescan metrics for autocompletion. Or should I file an issue?

This isn't a bug per-se, auto complete and so on can return false positives for various implementation reasons.

--

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/fb227510-89c3-4024-aba6-21ff4747f48a%40googlegroups.com.


--

Sergey Urushkin

unread,
Apr 14, 2020, 9:20:08 AM4/14/20
to Prometheus Users
I'm absolutely sure it's

вторник, 14 апреля 2020 г., 15:49:52 UTC+3 пользователь Brian Brazil написал:
On Tue, 14 Apr 2020 at 13:47, Sergey Urushkin <urus...@gmail.com> wrote:
Forgot to mention:
# prometheus --version
prometheus, version 2.17.1 (branch: HEAD, revision: ae041f97cfc6f43494bed65ec4ea4e3a0cf2ac69)
  build user:       root@806b02dfe114
  build date:       20200326-16:18:19
  go version:       go1.13.9

What builds are not affected then?

That should work then, you're probably still ingesting this metric.
I'm pretty sure it's not so, querying autocompleted metric gives "No datapoints found." or "Empty query result" (in modern prometheus ui) (on all timeline). I see that this metric doesn't exist when scraping it by curl. There is no relabeling of _name_ (while some other labels are relabeled).
I've also tried to remove particular metric via "match[]=bad_metric_1", this doesn't help too.

Brian
 

вторник, 14 апреля 2020 г., 15:26:52 UTC+3 пользователь Brian Brazil написал:
On Tue, 14 Apr 2020 at 12:18, Sergey Urushkin <urus...@gmail.com> wrote:
Hello,
I'm wondering is there a way to remove metric completely.
After running :
  /api/v1/admin/tsdb/delete_series?match[]={__name__=~"bad_metric_.*"}
  /api/v1/admin/tsdb/clean_tombstones
And even restarting prometheus, they are still appearing in autocompletion in the prometheus and grafana ui (while all values are removed because of the admin-api calls).

That should work, but it didn't always in all versions.
 
Is there a way to clean/refresh/rescan metrics for autocompletion. Or should I file an issue?

This isn't a bug per-se, auto complete and so on can return false positives for various implementation reasons.

--

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages