Feature Interest - reduce cardinality of node_cpu_seconds_total

[bob.c...@gmail.com]

Hello,

I have an environment where we monitor many nodes, with many cores. This drives a very large number of series coming from node_cou_seconds_total that presents an unnecessary load on Prometheus and Cortex. Don't care about per-core metrics in this case.

I'm about to start a PR to add a command-line option to omit the cpu label from this metric. 

Is this something that would be considered?

Thanks

[daniel.s...@ionos.com]

This can already be achieved with a combination of recording rules and metric_relabel_configs to drop the series that you don't want.

[Julius Volz]

On Thu, Dec 16, 2021 at 12:09 AM daniel.s...@ionos.com <daniel.s...@ionos.com> wrote:

This can already be achieved with a combination of recording rules and metric_relabel_configs to drop the series that you don't want.

Not quite though. You first have to store all the data in Prometheus in order to be able to use a recording rule to aggregate it down. And since Prometheus's local TSDB doesn't have selective retention yet (e.g. keep detailed data only for a short time), you'd then have to have at least two Prometheus servers, one just pulling the aggregates from the other and having longer retention...

So in general to me the feature request makes sense, but I'm not a Node Exporter maintainer. I'd just try filing a feature request / discussion there at https://github.com/prometheus/node_exporter/issues and see what people say.

 

On Wednesday, December 15, 2021 at 7:51:31 PM UTC+1 bob.c...@gmail.com wrote:

Hello,

I have an environment where we monitor many nodes, with many cores. This drives a very large number of series coming from node_cou_seconds_total that presents an unnecessary load on Prometheus and Cortex. Don't care about per-core metrics in this case.

I'm about to start a PR to add a command-line option to omit the cpu label from this metric. 

Is this something that would be considered?

Thanks

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/38fbf8a5-4079-47f1-9d6f-b30cefd17d6dn%40googlegroups.com.

--

Julius Volz

PromLabs - promlabs.com