How to Optimize Prometheus High memory？

[Liu Chang]

[prometheus]# ./bin/prometheus --version

prometheus, version 2.19.2

./bin/prometheus --config.file=./conf/prometheus.yml --web.listen-address="0.0.0.0:8089" --storage.tsdb.retention.size=200GB

Killed process 203738 (prometheus) total-vm:241826816kB, anon-rss:185585992kB, file-rss:0kB

We have a lot of metrics(> 10000) and a lots of instances (> 1000)，how to optimize prometheus memory usage？Thanks~

[Stuart Clark]

To reduce memory usage you would need to reduce the number of time series being scraped. This is often done by sharding - different Prometheus servers for different groups of targets or different applications.