HA setup for VM for HA Prometheus.

[Yagyansh S. Kumar]

Hi. So, I have a vanilla Prometheus setup with 50 day retention period and 4.6TB of persistent disk is used for that. Now, I want to have a HA Prometheus but I don't want to maintain 2 4.6TB disks(one for each Prometheus instance) which is why I am turning to VM setup to rescue. I'll keep a minimal retention of my Prometheus instances and let VM have a retention of 50 days. But the problem is that once my VM is setup, the VM node will become a single point of failure for me, for which also let say I spin up another node and ask one Prometheus instance to write data to VM1 and another to VM2, but in this case again I'll have to maintain 2 huge persistent disks(This time one for each VM node).

Is there any way to bypass this? Can I use shared disk for my 2 VM nodes? Or can you suggest a better setup here?

I am totally new to VM, please don't mind if the query sounds silly.

Thanks in advance.

[hage...@gmail.com]

Hi Yagyansh,

> Is there any way to bypass this? Can I use shared disk for my 2 VM nodes?

The point of HA or having a replica is that data will be duplicated. The duplication gives you a protection from accidental disk corruption or human error (accidental data corruption). Using shared disk for both replicas kills the point as I see it.

> Or can you suggest a better setup here?

Setup is fine from my point of view and is similar to what described here https://github.com/VictoriaMetrics/VictoriaMetrics#high-availability

But here's my thoughts regarding it:

1. It is likely that disk usage will be reduced after migrating data to VM from Prometheus. Users report different compression improvements from percents to hundreds of percents. To check what the benefit will be in your case I propose to try the migration tool we developed specifically for this purpose - https://github.com/VictoriaMetrics/vmctl#migrating-data-from-prometheus. Pls also see the articles about migration details here https://github.com/VictoriaMetrics/vmctl#articles.

2. In case if you don't want to maintain a huge disk - use a cluster version of VM https://github.com/VictoriaMetrics/VictoriaMetrics#scalability-and-cluster-version. It would allow to shard data into parts and reduce disk size-per-shard. It is ok to run multiple VM clusters for HA purpose.

3. In case if you use GCP persistent disks https://cloud.google.com/compute/docs/disks/#pdspecs - there is a very low risk to get the disk corruption (see disks internal implementation details). So the only thing you might want to get protection from - human error. And it can be achieved by setting the backup process via https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmbackup/README.md

4. Please note that VM the smallest retention unit is month - it has no way to set days or hours yet. It means you won't be able to set retention to 50d - it will be a month or two. Pls see more details here https://github.com/VictoriaMetrics/VictoriaMetrics#retention

Best regards,

Roman

[Yagyansh S. Kumar]

Thanks a lot, Roman.

[Yagyansh S. Kumar]

Is there any formula to estimate the amount of disk space required for a certain retention period?

[hage...@gmail.com]

The formula is almost the same as for Prometheus - see here https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects

Required disk space depends on compression, and compression depends on data you store. The unit of data in VM is represented as data points (or samples) - pairs of timestamp and value. Data points are then grouped into sorted blocks of data with different compression techniques applied. See more details here https://medium.com/faun/victoriametrics-achieving-better-compression-for-time-series-data-than-gorilla-317bc1f95932. 

So, let's imagine the following situation. You have 100 targets to scrape by Prometheus and every target you scrape returns 1000 data points (every line you see on page `/metrics` for your targets is a data point) and your scrape interval is 15s. Then number of data points Prometheus collects will be the following:

for 1m: 100 (targets) * 1000 (data points per target) * 4 (scrapes in a minute) = 400000 = 400K

for 1d: 24 * 60 * 400K = 576000000  = 576KK

for 30d: 30 * 576KK = 17280KK

Based on the first link "Prometheus uses only around 1-2 bytes per sample". So for Prometheus occupied disk space for 30d will be the following:

1B-per-sample: 17280KK * 1 B = 17.28 GB

2B-per-sample: 17280KK * 2 B  = 34.6 GB

VM users report the compression rate from 0.4B to 1.2B per data point. So calculations will be the following:

0.4B-per-sample: 17280KK * 0.4 B = 6.9 GB

1.2B-per-sample: 17280KK * 1.2 B = 20.7 GB

Pls see https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/CaseStudies for public case studies and numbers they report.

Also see details about VM retention here https://github.com/VictoriaMetrics/VictoriaMetrics#retention

Metrics and dashboards listed here https://github.com/VictoriaMetrics/VictoriaMetrics#monitoring will show the current number of stored data points, compression rate, disk space usage and a lot of other useful details. 

My recommendation is to setup a single version of VM, migrate via vmctl some part of Prometheus data and see how it goes. The dashboard will show details about disk usage, so you'll have a real numbers you can decide no.

[Yagyansh S. Kumar]

Thanks a lot.

What happens to my alerting rules and Alertmanager though? How are they gonna work with VM coming into the picture?

[hage...@gmail.com]

There are multiple ways for alerting, but it is ok to mix them though.

1. Use vmalert service for rules evaluation - https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmalert

Please see the list of features supported. In two words, you can feed vmalert with rules you already have and it will work.

Also see the limitations of vmalert - https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmalert#limitations.

vmalert becomes very handy when you start to store in VM data from more than 1 Prometheus HA pairs to get the global view.

2. Use 3rd party project Promxy for rules evaluation - https://github.com/jacksontj/promxy#how-do-i-use-alertingrecording-rules-in-promxy 

3. Retain Prometheus retention to cover your alerts and leave it as is. This is the most reliable way, because rules evaluation happens inside of one process and can't be interrupted by network issues. 

The rule of thumb is usually to evaluate the rules as close as possible to the data source. So if you can afford enough retention in your Prometheus - do so. If you want a global view or rules evaluation over long periods - use vmalert or promxy.

[Yagyansh S. Kumar]

Thanks a lot.

How much is "enough retention"? Can we estimate it?

[hage...@gmail.com]

It depends only on rules you want to evaluate. For example, alerting rule `increase(http_errors[6h]) > 0` will need 6h of retention to give a correct result. And if Prometheus is configured to minimal retention of 2h - it won't make much sense to evaluate rules which touch more than 2h of data.

[Yagyansh S. Kumar]

Got it. Thanks a lot. I am anyways thinking of keeping up to 3-4 days of retention on Prometheus instances.

I have started testing by ingesting the data from one of my Prometheus instances to VM.

Sorry to bug you but I have a couple of silly things in my mind.
1. If an alert is 3 days old and my Prometheus retention is 2 days, in this case, the alert will be generated again, as a new alert right? I mean, of course, it should, it's pretty straight forward but I don't know why I think having 2 Prometheus sending the data to the Alertmanager can make this situation spicy.

2. If I have 2 Prometheus instances scraping the same targets, is it required to give them different external labels? 

[Yagyansh S. Kumar]

Making my second point a bit clear - 2 Prometheus instances scraping the same target and sending the data to the same VM instance, is it required to give them different external labels? 

[hage...@gmail.com]

> 1. If an alert is 3 days old and my Prometheus retention is 2 days, in this case, the alert will be generated again, as a new alert right?

If I get right, then - no. Alert entity is an object in Prometheus process memory which get updated on every rule evaluation. Moreover, every rule evaluation for firing alert will result into new notification request to the Alertmanager. If Prometheus stops to send these requests (alert is not firing anymore or Prometheus isn't running) - Alertmanager will automatically resolve alert after some period (5m by default afaik). So in two words - age of alert doesn't matter. The only fact that matter is if Prometheus retention is enough to cover alert's expression.

2. If you want to have deduplication in VM - then no, labels should be identical. Pls see https://github.com/VictoriaMetrics/VictoriaMetrics#deduplication

[Yagyansh S. Kumar]

Thanks a lot, for all your help.

Can VM help me managing by config files also across my Prometheus instance? Or will I manually have to update my config file every time in both the servers? If not possible via VM, can you suggest a solution?

[hage...@gmail.com]

> Can VM help me managing by config files also across my Prometheus instance?

Not sure that I understand the question. Could you pls elaborate a little bit more? Like describe a use case or scenario you have in mind?

[Yagyansh S. Kumar]

Like I add a new job in my prometheus.yml, now since I have 2 Prometheus instances will I have to make this change in both the instances manually? Or is there any way to keep the config files in sync between the 2 Prometheus instances.

[hage...@gmail.com]

I guess, this issue (delivering config changes) should be addressed by your deployment/orchestration system, not by Prometheus or VM.

Alternatively, you may use service-discovery system in Prometheus to automatically discover new targets - search for it here https://prometheus.io/docs/prometheus/latest/configuration/configuration/

[Yagyansh S. Kumar]

Thanks.

How are people generally managing the config delivery management system? Is there any active project for this?

Because I have a VMWare environment where I am using file discovery, hence, the targets have to be added to some file with a certain set of labels. I'm quite confused on how will I handle the config delivery with 2 instances in picture. One of the solutions I can think of is using a NFS or some sort of shared storage for storing the config files. What are your thoughts on this? I guess storing only the configs on shared storage shouldn't be an issue.

--
You received this message because you are subscribed to a topic in the Google Groups "victorametrics-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/victorametrics-users/BR4oHbrxXzc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to victorametrics-u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/victorametrics-users/71268c06-8e7f-485f-b111-dbad3abe8edbn%40googlegroups.com.

[Aliaksandr Valialkin]

On Sat, Oct 17, 2020 at 11:53 AM Yagyansh S. Kumar <yagyans...@gmail.com> wrote:

Thanks.

How are people generally managing the config delivery management system? Is there any active project for this?

People usually rely on various service discovery mechanisms provided by Prometheus and vmagent for automatic propagation of changes to scrape targets. For example, service discovery for consul, k8s, amazon ec2, google compute engine, dns, etc. See https://victoriametrics.github.io/vmagent.html#how-to-collect-metrics-in-prometheus-format for details.

 

Because I have a VMWare environment where I am using file discovery, hence, the targets have to be added to some file with a certain set of labels. I'm quite confused on how will I handle the config delivery with 2 instances in picture. One of the solutions I can think of is using a NFS or some sort of shared storage for storing the config files. What are your thoughts on this? I guess storing only the configs on shared storage shouldn't be an issue.

This looks like a reasonable solution.

--

Best Regards,

Aliaksandr Valialkin, CTO VictoriaMetrics

[Yagyansh S. Kumar]

Thanks a lot!