Does Prometheus recommend exposing 2M timeseries per scrape endpoint?

[tejaswini vadlamudi]

I have a use case where a particular service (that can be horizontally scaled to desired replica count) exposes a 2 Million time series. Prometheus might expect huge resources to scrape such service (this is normal). But I'm not sure if there is a recommendation from the community on instrumentation best practices and maximum count to expose.

Thanks,

Teja

[Stuart Clark]

Two million time series returned from a single scrape request?

That's way out of the expected ballpark for Prometheus, and also sounds
totally outside what I'd expect from any metrics system.

Would you be able to explain a bit more about what you are wanting to be
able to achieve and we can suggest an alternative to Prometheus/metrics
that would help?

--
Stuart Clark

[tejaswini vadlamudi]

Thanks Stuart, this expectation is coming as a legacy requirement for a Telco cloud-native application that has huge cardinality (3000 values for a label-1) and little dimensionality (2 labels) for 300 metrics.

Is there any recommendation like not more than 1k or 10k series per endpoint?

Br, Teja

[Stuart Clark]

On 14/06/2022 12:32, tejaswini vadlamudi wrote:
> Thanks Stuart, this expectation is coming as a legacy requirement for
> a Telco cloud-native application that has huge cardinality (3000
> values for a label-1) and little dimensionality (2 labels) for 300
> metrics.
>
> Is there any recommendation like not more than 1k or 10k series per
> endpoint?

The general expectation is that each Prometheus server would have no
more than a few million time series in total. The normal use case is to
have 10s/100s of jobs & targets, each exposing 100s/1000s of time
series, rather than a single target exposing significant number of time
series itself.

If this is a single application what is the need for that level of
cardinality? For example something like a high volume e-commerce system
with millions of users might only need labels with say a cardinality of
a few 10s/100s each (for example HTTP status code, section of website,
etc.). What is it about this system that you think it needs very high
cardinality labels?

--
Stuart Clark

[tejaswini vadlamudi]

Telco system, that creates series per Access Point Name (APN) name as label value (similar to a user-id or a pod-IP but very much important to record)

/Teja

[l.mi...@gmail.com]

Total number of time series scraped would be more important I think, so you also need to know how many targets you'll have.

I had Prometheus servers scraping 20-30M time series total and that was eating pretty much all memory on server with 256GB ram.

In general when doing capacity planning we expect 4KB of memory per time series for base Go memory, and then we need to double that for garbage collector (you can try to tweak GOGC env variable to trade some cpu for less gc memory overhead).

With 25M time series 4KB per series means 100GB of Go allocations, and 200GB to account for garbage collector, which usually fits 256GB.
But we do run a huge number of services, so Prometheus will scrape lots of targets and get a small number of metrics from each.

You want to scrape 2M from a single target and that means Prometheus will have to request, read and parse a huge response body, this might require more peak memory and it might be slow, so your scrape interval would have to allow for that.
Another thing to remember is churn - if your time series have labels that keep changing all the time then you might run out of memory, since everything that prometheus scrapes (even only once) ends up in memory until it persists data to disk, which is by default every 2h AFAIR. If the list of values of your APN is not a fixed set and you keep seeing random values over time, then that will accumulate in memory, so your capacity planning would have to take into account how many unique values of APN (and other values) are there and if this is going to grow over time. That's assuming you want to stick with a single prometheus instance, if you can shard your scrapes then you can scale horizontally.

It's always hard to give a concrete answer to question like this since it all depends, but it's usually a matter of having enough memory, cpu is typically (in my environment at least) less important.

[Matthias Rampke]

One place where time series of this magnitude from a single target are unfortunately common is kube-state-metrics (KSM). On a large cluster, I see almost 1M metrics. Those are relatively cheap because they are nearly constant and compress well, but I believe there was quite some work in that project to make scraping work well from the target side. This includes playing with compression - depending on your network it may be faster to stream uncompressed, than to compress and uncompress.

In summary, 2M time series from a single target is unusual but not without precedent. Look at KSM for issues that they encountered and possible solutions.

/MR

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9c494e40-fe19-4252-a9f0-5d024c04d8b4n%40googlegroups.com.