Suggested hardware for production installation

[Jan Algermissen]

Hi,

what hardware/VM specifications would you recommend for a production installation of Prometheus (roughly thousands of services, hundreds of DB etc. nodes)?

I would guess the focus should be on RAM and network throughput, both the more the better. Is that a correct way to approach the sizing, or do I need to consider other factors?

Jan

[Ben Kochie]

That's a good question.  It mostly comes down to how many individual metrics and how many samples per second you plan to ingest.  The number of actual targets isn't as big an issue as the scrapes are cheap, a simple http GET, but the sample ingestion takes some work.

RAM is a big factor

* It limits how much data you can crunch with queries

* It limits how much data can be buffered before writing to the disk storage

Network throughput is not a huge issue.  A single server with millons of timeseries and 100k samples/second only needs a few megabits/second.

CPU is important, a large server can easily use many cores.

For example, a prometheus server configured to monitor just node_exporter metrics:

* ~1700 nodes

* ~1400 metrics/node

* ~2.3M in-memory series

* ~78k samples/second

This server uses about 45GB of ram, and typically uses about 5 CPUs

It also needs about 5GB/day of storage space (SSD in this case) with varbit encoding.

We could probably get away with a lot less ram, but it allows for very large historical queries.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ivaquero...@gmail.com]

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.

>
> For more options, visit https://groups.google.com/d/optout.

Thanks for your answer, Ben.

Do you know what are the requirements for the Alert Manager component, given these assumptions for node_exporter metrics?

[Ben Kochie]

The alertmanager is a very small service, it only requires about 50MiB of memory and very little cpu. It scales with how many alerts you send, but it would take a huge amount of alerts to cause any load.

To post to this group, send email to prometheus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/5610afe4-5fcb-467b-97a9-cef01e8f8e82%40googlegroups.com.

[Ben Kochie]

Note, this thread contains obsolete scaling information. The numbers here are from Prometheus 1.x. Prometheus 2.x has a completely different memory, cpu, and IO profile.

[mrserg...@gmail.com]

Hello, Ben!
Could you please put here scaling information of Prometheus 2.x? Whta is hte memory, cpu, and IO profile for Prometheus 2.x?
Your previos posts were very helpfull for me.
Thanks in advance!