Basic Pointers on Initial Deployment for Newbs

[Kelsey Cummings]

Greetings, I'm looking to replace a legacy application using RRDs with Prometheus and I'm looking for some guidance on what a deployment would probably end up looking like.

We need to scale up to ~8m samples, on 300 and 1200 second intervals and need to keep a years worth of data, with or without down-sampling.  If it makes any difference, it's probable that we'll need to make use of the push gateway.

Ignoring HA, it seems like this is well within what a single instance with 128 or 256G of ram could handle with a few TB of local storage but the docs seem to strongly encourage  use of some other system to achieve reliable long term storage.

Queries will mostly be limited to single/few series graphing for 24 hours but we are interested in alerting functionality on poll/push (eg, last N samples over/under threshold).

Should we just put this on a decently sized host with local redundant ssd or should we really go down the road of smaller instances with thanos.io with an object store backend?

Any insights are greatly appreciated.

[Brian Candler]

The minimum useful scrape time for Prometheus is 2 minutes (120 seconds).  This is because Prometheus treats timeseries which are older than 5 minutes as "stale", i.e. gone away.  Using a 2 minute scrape interval means that even if you miss one scrape, the timeseries won't go stale.

When you say "we'll need to make use of the push gateway", you should reconsider this.  Whatever the problem is, the push gateway is almost always NOT the right solution.  Pushgateway is just a cache that stores the last pushed value; it exists for one-shot scripts which vanish after execution but want to stash a value away to be scraped later (e.g. cron jobs).  If you describe your use case, someone can help you find a better solution.

Despite what the docs say, people have reported using prometheus for long-term storage successfully.  There are a whole range of options with remote_write that you can use too.  Thanos with object store will certainly give you very reliable persistent long-term storage, and will do downsampling too (although this is mainly to improve performance of queries over long time ranges, rather than to reduce the storage size).

If you're only graphing over 24 hour periods, and alerting over the last N samples, native prometheus should be fine: the alerting rules will mainly be looking at the "head" chunks which remain in RAM.

However, 8 million series for a single prometheus instance might be pushing it a bit too much.  I would be more comfortable with sharding across a few smaller instances than having one monster instance.

[Kelsey Cummings]

On 10/13/2020 12:42 AM, Brian Candler wrote:
> The minimum useful scrape time for Prometheus is 2 minutes (120
> seconds).

Ah, interesting. The 300 second to staleness is not configurable?
Polling the devices faster is not practical. All of the samples will be
coming out of an existing poller - in some cases it takes ~ 10 minutes
to the poller to complete taking the stats off a single device (limited
by the device management plane's pitiful CPU). We could write our own
cache layer that would allow prometheus to poll, oversampling if
necessary, if that's architecturally better than using the push gateway.

> Despite what the docs say, people have reported using prometheus for
> long-term storage successfully.

You'd never know it. ;)

> However, 8 million series for a single prometheus instance might be
> pushing it a bit too much.  I would be more comfortable with sharding
> across a few smaller instances than having one monster instance.

Any hints on what server specs would be required for an instance that
big? Projecting costs and what scale out is always a bit tricky when
you've never run an application before.

-K

[Brian Candler]

On Tuesday, 13 October 2020 19:39:08 UTC+1, Kelsey Cummings wrote:

On 10/13/2020 12:42 AM, Brian Candler wrote:
> The minimum useful scrape time for Prometheus is 2 minutes (120
> seconds).
Ah, interesting.  The 300 second to staleness is not configurable?

There is a global option "--query.lookback-delta"

 

Polling the devices faster is not practical. All of the samples will be
coming out of an existing poller - in some cases it takes ~ 10 minutes
to the poller to complete taking the stats off a single device (limited
by the device management plane's pitiful CPU).  We could write our own
cache layer that would allow prometheus to poll, oversampling if
necessary, if that's architecturally better than using the push gateway.

For devices which are slow or expensive to poll: I'd suggest polling them in a separate process (e.g. a cronjob at 30 minute intervals or whatever), write the data somewhere, and then scrape the written data.  node-exporter's textfile-collector is very convenient for this.

Amongst other things, it means you can scrape from multiple prometheus servers for HA without increasing the load on your target systems.  Also, the textfile-collector exposes a metric which is the timestamp of the file, so you can easily alert if the file hasn't been updated for more than a certain amount of time.

 

Any hints on what server specs would be required for an instance that
big?  Projecting costs and what scale out is always a bit tricky when
you've never run an application before.

I'll leave others to comment, not having built one that big myself.  There is this calculator for RAM though:

https://www.robustperception.io/how-much-ram-does-prometheus-2-x-need-for-cardinality-and-ingestion

You could also look at prometheus -> (remote write) -> VictoriaMetrics.  This has good support for backup and restore of long-term data, and would let you keep your prometheus retention short (just enough for alerting history really).  Blog:

https://medium.com/@valyala/speeding-up-backups-for-big-time-series-databases-533c1a927883

[Kelsey Cummings]

Thanks for the pointers Brian, I appreciate the help.
-K