Suggested limit for max number of series per Prometheus instance?

[Al]

I'm currently on-boarding many more metrics and hosts to our Prometheus infrastructure and I wanted to know when is it advised to shard out metrics into a separate instance?  I have multiple shards (or groups) of Prometheus instances although one of the groups of instances has over 3 million total series in it, even though its ingestion rate doesn't really go above a comfortable 100k series/second.  I know that a single instance can typically handle up to 1 million series per second, although I can't seem to find clear guidelines with regards to the total number of series per Prometheus host.   Some of the metrics have a relatively higher cardinality, thus causing the higher number of series.   I'd appreciate any advice you can provide me with.

Thanks!

[Brian Brazil]

On Tue, 19 May 2020 at 14:23, Al <alain.l...@gmail.com> wrote:

I'm currently on-boarding many more metrics and hosts to our Prometheus infrastructure and I wanted to know when is it advised to shard out metrics into a separate instance?  I have multiple shards (or groups) of Prometheus instances although one of the groups of instances has over 3 million total series in it, even though its ingestion rate doesn't really go above a comfortable 100k series/second.  I know that a single instance can typically handle up to 1 million series per second, although I can't seem to find clear guidelines with regards to the total number of series per Prometheus host.   Some of the metrics have a relatively higher cardinality, thus causing the higher number of series.   I'd appreciate any advice you can provide me with.

The limit currently appears to be somewhere in the low tens of millions of head series.

Brian

 

Thanks!

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/608224ac-f7fc-4dbb-9b18-0e7ed4f6e604%40googlegroups.com.

--

Brian Brazil

www.robustperception.io