Kubernetes and large number of time series

[Joe Kemp]

I am deploying a Kubernetes environment and hope to use Prometheus to monitor all of my pods.  I will have hundreds of pods each generating its own set of metrics with a unique kubernetes_pod_name label.  I am trying to figure out how the caution statement about creating too many time series due to multiple key-value label pairs (https://prometheus.io/docs/practices/naming/#labels) is applicable in a Kubernetes environment.  We do multiple deployments every day which will potentially create a thousand unique kubernetes_pod_name labels per day.  This doesn't even include autoscaling impacts.  I don't see a way to follow the caution statements advice in my environment.

Am I missing a workaround for this issue?  I don't think there is any other option than to apply a unique label to each pod's set of metrics.  What is the real impact with this scenario?  Is storage space preallocated for each unique time series, will aggregation across this number of time series be too CPU intensive, will I need large amounts of RAM to handle the promql queries? 

Thanks,

-Joe

[Julius Volz]

Yes, that should be fine for most use cases. You'll want to label your metrics by pod (either with your kubernetes pod name label, or with the host:port "instance" label that Prometheus automatically attaches to all scraped series).

Overall, you have a budget of a few million time series in a big Prometheus server, so you can do the combinatorial math of where things would end up in your case. Individual labels going into the multi-10k cardinality range can still be problematic when you query for the involved metrics (the query result sets can grow very large). And don't stuff things like email addresses or public internet IP numbers etc. into labels, as those can of course grow into the many millions...

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.