keep metric using metric_relabel_configs not working

[Ankita Khot]

 - job_name: 'abc'

    metrics_path: /prometheus

    scheme: https

    static_configs:

    - targets: ['test_vm']

    tls_config:

     insecure_skip_verify: true

    metric_relabel_configs:

     - source_labels: [__name__]

       regex:         "jvm_memory_used_bytes"

       action:        keep

I am trying to keep only "jvm_memory_used_bytes" metric using metric_relabel_configs but on the dashboard I am able to see all the metrics. I also changed 'action' to 'drop' but "jvm_memory_used_bytes" is not getting dropped.

Is the above configuration correct ? Is there a way to filter the metrics at the source, so that while using the curl command for metric endpoint we will be able to see only the required metrics.

[Brian Candler]

> Is there a way to filter the metrics at the source, so that while using the curl command for metric endpoint we will be able to see only the required metrics.

That would be something you configure in whatever exporter you are using.  For example, node_exporter has flags to turn various metrics on and off.

I don't know why your metric relabelling isn't dropping metrics.  Perhaps the metrics you see are not from job "abc"? Did you HUP prometheus after changing the config?

[Ankita Khot]

Its working fine but with a delay of 3-4 mins. What probably could be the reason for this delay? 

[Brian Candler]

If a pre-existing metric just vanishes, the most recent data point hangs around for a while.

https://www.robustperception.io/staleness-and-promql

[Ankita Khot]

ohk.. thanks!

On Fri, Feb 28, 2020 at 9:32 PM Brian Candler <b.ca...@pobox.com> wrote:

If a pre-existing metric just vanishes, the most recent data point hangs around for a while.

https://www.robustperception.io/staleness-and-promql

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e2e20a27-ab8b-44de-b6a2-a5ff9a347b71%40googlegroups.com.

[Ankita Khot]

In Prometheus UI within the dropdown that says 'insert metric at cursor' it shows the previous scraped metrics as well. How to remove those? I would like it to show only the metrics which i am keeping through 'metric-relabel-config'. 

On Friday, February 28, 2020 at 2:14:50 PM UTC+5:30, Ankita Khot wrote:

[Brian Candler]

On Monday, 2 March 2020 06:45:56 UTC, Ankita Khot wrote:

In Prometheus UI within the dropdown that says 'insert metric at cursor' it shows the previous scraped metrics as well. How to remove those? I would like it to show only the metrics which i am keeping through 'metric-relabel-config'. 

They'll eventually expire, when the retention period is reached (15 days by default).  But if you want to get rid of them before then, you can explicitly delete all the timeseries you don't want:

https://prometheus.io/docs/prometheus/latest/querying/api/#delete-series

https://www.robustperception.io/deleting-time-series-from-prometheus