Metric Relabeling

98 views
Skip to first unread message

rksakt...@gmail.com

unread,
May 7, 2021, 6:01:58 AM5/7/21
to Prometheus Users
Hi All,

Our Java applications are exposing exposing application metrics using micrometer framework and we have the http_server_requests_bucket getting created based on the response time of the requests. Since the number of microservices are in 1000's we are already having more than 1 million time-series. We analysed and found the above time series metrics is having more than 500k+  time-series. So we want to keep only the time series which has below bucket values and drop the rest. If we try doing the below all the other time series like (process uptime, JVM related statistics are also getting dropped).

We can use drop and provide regex of other values(https://www.robustperception.io/why-are-prometheus-histograms-cumulative) but they the list to blacklist is very long so we want to whitelist the required values. Any option to include additional check to apply the below metric label only for time series with name http_server_requests_bucket or any other options available. Looking forward for some help

- source_labels: [le]
action: keep
regex: '(\+Inf|0.1|0.25|0.5|0.75|1.0|2.5|5.0|7.5|10|1)'

Regards
Sakthi

Ben Kochie

unread,
May 7, 2021, 6:25:41 AM5/7/21
to rksakt...@gmail.com, Prometheus Users
If you're using micrometer, you can configure the buckets at the application side.

https://micrometer.io/docs/concepts

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/1a080aaf-0db3-4ebe-a3b5-3f7ec5a223can%40googlegroups.com.

Sakthi Raam

unread,
May 7, 2021, 9:44:47 AM5/7/21
to Ben Kochie, Prometheus Users
Hi Ben, Thanks for the information. This requires configuration change which application teams are reluctant to do so hence looking to drop them on Prometheus end before storing it.

Thank You
Regards
Sakthi
--

Best regards

R.K.Sakthi Raam

*: rksakt...@gmail.com

': +65-9185-0354

Linkedin

Sakthi Raam

unread,
May 18, 2021, 2:51:46 AM5/18/21
to Prometheus Users
Hi All, Any help on this plz. Thank You

Regards
Sakthi
Reply all
Reply to author
Forward
0 new messages