delete metrics from prometheus 2.0

[robu...@gmail.com]

Hi all,

I'm trying out prometheus 2.0 and like to delete some metrics that I by accident ingested with a wrong number calculation

I have some metrics exported via text_file/node_exporter  like this:

...

d_zpool_errors{zpool="<poolname>"} 0

d_zpool_scrub{zpool="<poolname>"} 0

d_zpool_repaired{zpool="<poolname>"} <number>

d_zfs_used{device="<poolname>"} <number>

d_zfs_available{device="<poolname>"} <number>

...

all fine and pretty 

But now I would like to delete ALL metrics matching "^(d_zpool_|d_zfs_).*" 

(or if regex wouldn't work here, I would loop through the list of relevant metric names and delete it one by one)

regardless of any label.

I tried various approaches from what I found on the net but neither worked,

I than tried to delete the prometheus job, but even that failed:

curl -XDELETE -g 'http://localhost:9090/api/v1/series?match[]={job="prometheus"}'

{"status":"error","errorType":"internal","error":"not implemented"}

Has the syntax changed in prometheus 2.0

(on a 1.7.1 instance this command works)

or am I'm missing something obvious?

And yes, the prometheus job is running, I can query it and have it in the web console

best regards and TIA

[Maxim Filatov]

Looks like you forgot to enable admin API.

Quote from manual (https://prometheus.io/docs/operating/security/):

As of Prometheus 2.0, the --web.enable-admin-api flag controls access to the administrative HTTP API which includes functionality such as deleting time series. This is disabled by default. If enabled, administrative and mutating functionality will be accessible under the /api/*/admin/ paths. The --web.enable-lifecycle flag controls HTTP reloads and shutdowns of Prometheus. This is also disabled by default. If enabled they will be accessible under the /-/reload and /-/quit paths.

--

Best regards,

Maxim Filatov

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/1375b28b-c812-420c-be8e-8aa197617990%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[robu...@gmail.com]

thnx, 

yupp forgot that flag, but now that I enabled it I'm getting:

curl -XDELETE -g 'http://localhost:9090/api/v1/admin/series?match[]={job="prometheus"}'

Method Not Allowed

curl -XDELETE -g 'http://localhost:9090/api/v1/series?match[]={job="prometheus"}'

{"status":"error","errorType":"internal","error":"not implemented"}

my environment file is:

ARGS="--config.file=/etc/prometheus/prometheus.yml \

      --log.level=info \

      --web.enable-admin-api \

      --web.enable-lifecycle \

      --web.console.libraries=/opt/prometheus-2.0.0.linux-amd64/console_libraries \

      --web.console.templates=/opt/prometheus-2.0.0.linux-amd64/consoles \

      --storage.tsdb.path=/var/lib/prometheus/metrics \

      --storage.tsdb.retention=36d"

seems I still missing something

[Simon Pasquier]

Hello,

I think that the path is http://localhost:9090/api/v2/admin/tsdb/delete_series

Cheers,

Simon

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/73bbd594-8168-401b-9258-47f7664a9eb3%40googlegroups.com.

[robu...@gmail.com]

hm, that sounds like a reasonable idea, but:

curl -XDELETE -g 'http://localhost:9090/api/v2/admin/tsdb/delete_series?match[]={job="prometheus"}'

Method Not Allowed

Any other idea is most welcome!

[robu...@gmail.com]

Is that the right syntax to delete metrics from prometheus 2.0

 

curl -XDELETE -g 'http://localhost:9090/api/v2/admin/tsdb/delete_series?match[]={job="prometheus"}'

Method Not Allowed

still doesn't work here,

admin api flag is set and prometheus is running 

TIA

[Simon Pasquier]

Pls see the answer provided last week: https://groups.google.com/d/msg/prometheus-users/ToMKsb9fYp8/az6afuX3CgAJ

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d95bbf28-8d6e-4133-a6f9-0bdb58cd8da4%40googlegroups.com.

[robu...@gmail.com]

Ah thnx ...

okay will wait for the documentation ...

[Ben Kochie]

Or better, write it up and submit a PR. :-D

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2e783520-bbea-4883-894b-b54b2aade640%40googlegroups.com.

[Hector Fabio Aguilar]

Anyone knows where is the documentation. I validated the related link but there also mention the documentation will be created "soon". That was in november.

[Ben Kochie]

The Prometheus docs source is here: https://github.com/prometheus/docs

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/1f1460a2-7ffc-492b-9591-1e5e12572a6d%40googlegroups.com.