Impact of high cardinality

58 views
Skip to first unread message

Girish Aher

unread,
Jan 22, 2021, 6:28:11 PM1/22/21
to Prometheus Users
Hello Folks,
A relatively new prometheus adopter here. Howdy!

I have been reading through the terrific articles on Prometheus 2.x TSDB here:

As I understand, the effect of high label cardinality (>1000 values for a labelname) is high memory usage. Assuming I can scale up the memory, are there any other effects of high label cardinality that one needs to be aware of?

Could the ingestion/query performance be affected due to high label cardinality (despite having plenty of memory)? I gather from above articles that a map[labename][]postingOffset is maintained in memory per block and for a match query, it requires iterating through the []postingOffset sequentially comparing label value. Would this contribute to high query latencies?

fyi - we are doing 4 million unique time series today with an ingestion rate of about 135K/sec. Pretty happy so far with the performance! :)



--Girish

Ben Kochie

unread,
Jan 23, 2021, 5:11:42 AM1/23/21
to Girish Aher, Prometheus Users
Many of the issues are related to query-time performance. It depends a lot on the total cardinality at query time.

For example, we have a couple of metrics with high total cardinality due to the number of label combinations. One histogram metric has about 100k total metrics after you multiple the number of instances times the number of buckets times the number of labels on those buckets.

In order to produce graphs and SLI/SLO reporting, we run a pretty standard `rate(histogram_bucket[1m])` recording rule. This rule evaluation takes 30 seconds to complete. Thankfully Prometheus TSDB has isolation for queries, so even if we have overlapping rule evaluations, we don't break anything. But it's still quite CPU and memory expensive to run these rules.

Another thing to note about high cardinality is regexp matching `=~` is a full table scan. So every query has to query all values in the label index. This can start to take a long time when you have 10s of thousands of values in a single label. Say the full scan takes 10ms. That's fine for a single instant query, but Prometheus evaluation is a bit different. Each point on the graph is a unique instant evaluation. So if you have a 500 data-point graph, that's now 5 seconds of query time spent parsing labels.

Ingestion is typically the easy part, querying is the hard part.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/972e5772-0001-4fd1-8906-383884c70d63n%40googlegroups.com.

Brian Brazil

unread,
Jan 23, 2021, 5:57:20 AM1/23/21
to Ben Kochie, Girish Aher, Prometheus Users
On Sat, 23 Jan 2021 at 10:11, Ben Kochie <sup...@gmail.com> wrote:
Many of the issues are related to query-time performance. It depends a lot on the total cardinality at query time.

For example, we have a couple of metrics with high total cardinality due to the number of label combinations. One histogram metric has about 100k total metrics after you multiple the number of instances times the number of buckets times the number of labels on those buckets.

In order to produce graphs and SLI/SLO reporting, we run a pretty standard `rate(histogram_bucket[1m])` recording rule. This rule evaluation takes 30 seconds to complete. Thankfully Prometheus TSDB has isolation for queries, so even if we have overlapping rule evaluations, we don't break anything. But it's still quite CPU and memory expensive to run these rules.

Another thing to note about high cardinality is regexp matching `=~` is a full table scan. So every query has to query all values in the label index. This can start to take a long time when you have 10s of thousands of values in a single label. Say the full scan takes 10ms.

 
That's fine for a single instant query, but Prometheus evaluation is a bit different. Each point on the graph is a unique instant evaluation. So if you have a 500 data-point graph, that's now 5 seconds of query time spent parsing labels.

The index lookup is only done once for given selector for a range (i.e. graph) query. The principle applies though for the evaluation process after that, you'd have 500 evaluations of say 100k series each with a rate() on 5 samples, taking the rough rule of thumb of 10M samples processed per second that'd be around 25s for the query.

Brian

 

Ingestion is typically the easy part, querying is the hard part.

On Sat, Jan 23, 2021 at 12:28 AM Girish Aher <giris...@gmail.com> wrote:
Hello Folks,
A relatively new prometheus adopter here. Howdy!

I have been reading through the terrific articles on Prometheus 2.x TSDB here:

As I understand, the effect of high label cardinality (>1000 values for a labelname) is high memory usage. Assuming I can scale up the memory, are there any other effects of high label cardinality that one needs to be aware of?

Could the ingestion/query performance be affected due to high label cardinality (despite having plenty of memory)? I gather from above articles that a map[labename][]postingOffset is maintained in memory per block and for a match query, it requires iterating through the []postingOffset sequentially comparing label value. Would this contribute to high query latencies?

fyi - we are doing 4 million unique time series today with an ingestion rate of about 135K/sec. Pretty happy so far with the performance! :)



--Girish

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/972e5772-0001-4fd1-8906-383884c70d63n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages