Understanding why High Cardinality is Problematic & Solutions for High Cardinality

[Al]

I realize why high label cardinality is problematic in Prometheus although I have yet to come across a blog post or article that really explains the lower level details as to why it's problematic.  It would be great to have this information so that I could then better understand the limitations when compared to other database systems such as Postgres and kdb.  I would appreciate it if anyone could share with me any information relevant to this topic.  

I would also like to know some commonly used strategies for offloading high-cardinality metrics (maybe even just a subset of metrics from a /metrics endpoint) to other database systems which don't have issues with this such as Elasticsearch, Postgres, kdb, etc.  I'm currently using Elasticsearch in these cases although curious to know what others are doing.

Thank you

[Brian Brazil]

On Thu, 16 Jan 2020 at 11:53, Al <alain.l...@gmail.com> wrote:

I realize why high label cardinality is problematic in Prometheus although I have yet to come across a blog post or article that really explains the lower level details as to why it's problematic.  It would be great to have this information so that I could then better understand the limitations when compared to other database systems such as Postgres and kdb.  I would appreciate it if anyone could share with me any information relevant to this topic.  

https://www.robustperception.io/cardinality-is-key

Brian

 

I would also like to know some commonly used strategies for offloading high-cardinality metrics (maybe even just a subset of metrics from a /metrics endpoint) to other database systems which don't have issues with this such as Elasticsearch, Postgres, kdb, etc.  I'm currently using Elasticsearch in these cases although curious to know what others are doing.

Thank you

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/336f776f-fc1c-4b26-a23d-6870df041eee%40googlegroups.com.

--

Brian Brazil

www.robustperception.io

[Al]

Thanks so much for the quick response Brian. I have previously come across this article, which is great, although I would really like to understand the implications at a lower level of things such as:
- memory and cpu costs
- how this might affect the WAL, especially when you have to restart Prometheus or it crashes
- etc

For example, I’ve been using the postgres_expoter and I’ve had to change things a bit as some metrics from pg_stat_statements have a very high cardinality for labels such as ”query”. These Prometheus instances were reaching ~49GB mem and then crashing as they were evicted due to memory constraints (in kubernetes). I’m working on a solution to offload these metrics to Elasticsearch, although I having a more in depth understanding of what’s going on will help me explain to management the reason for the limitations.

Thanks again!

[Ben Kochie]

The memory use of Prometheus for large indexes should be substantially improved with the 2.15 release.

I also have a pg_stat_statements collection which generates around 5k queryids and takes up quite a lot of resources on our Prometheus server. These metrics take up about 80% of one server's metrics.

Upgrading to Prometheus 2.15 cut the memory use of this server in half.

I also just recently did some optimization to our postgres_exporter query config to reduce our ingestion:

https://gitlab.com/gitlab-cookbooks/gitlab-exporters/merge_requests/124

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b18fe9ed-0c76-4d47-b836-ee63dcfb5d8f%40googlegroups.com.

[Brian Candler]

There is a nice document on the internal design of prometheus 2.x storage here:

https://fabxc.org/tsdb/

Roughly speaking:

- prometheus tries to put all the data points of a time series next to each other on disk: this allows it to do delta compression between them.

- each unique bag of labels defines a distinct time series

- therefore, if you have 1 million different values for a label, you will get 1 million different timeseries

- there are implications for buffering to write each timeseries in their own blocks, keeping track of all the distinct timeseries in RAM etc.

- queries may also end up having to aggregate across millions of time series, requiring lots of small reads

- in the limit you'd end up with billions of timeseries with one point in each, which makes no sense

You'll find similar cardinality issues with "tags" in influxdb.