Need help on Prometheus load testing for capacity planning

[rs vas]

We are planning to attempt to generate the load on the Prometheus instance so that we can identify the right EC2 instance family to run a specific load.

We have tried https://github.com/ncabatoff/prombench but without any success, since this has not been touched since 4 years, not compatible with the latest Prometheus version.

Our requirement is: we would like to generate some load using a synthetic exporter and target to expose x number metrics on a y number of targets for x number of hours to identify the right instance size.

Does one have any suggestions to run a tool from an EC2 instance which can expose targets? If possible can it also hit Prometheus with some queries...?

Appreciate any inputs on the same.

rs vas

[Brian Candler]

I'd be inclined to take the opposite approach: use your production exporters, guess the initial instance size, and iterate based on how under- or over-utilized it is.   It only takes a minute to switch an instance from one type to another in EC2.

As a starting point for your guess, you can use this:

https://www.robustperception.io/how-much-ram-does-prometheus-2-x-need-for-cardinality-and-ingestion

Your real metrics are going to be a better test than any synthetic set of metrics.  Remember also that you can scrape the same targets from multiple prometheus instances, so you can spin up a second prometheus for testing without touching the existing one, if you have one.

[rs vas]

Hey Brian, that link is great to start with, thanks for sharing it. Very helpful.

If we had to test with some synthetic set of exporters with some configurable metrics, Do we really have anything we can quickly set-up on amazon ec2 and get that load tested? Just thought of finding out still... I am sure many of you must have done it already on this forum...

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/1c6e391b-d06b-40fb-a088-67624b559596o%40googlegroups.com.