Behavior of replication in a cluster configuration.

254 views

Skip to first unread message

Seth Graham

unread,

Apr 8, 2021, 2:50:42 PM4/8/21

to victorametrics-users

I am in the process of evaluating victoria metrics for my environment, and have a three node cluster running with replicationFactor=2. It's working well but I have a question regarding the behavior when a vmstorage server is permanently lost that does not appear to be covered by the documentation.

When a vmstorage server is lost, does the data surviving on another server ever get replicated to a new server to match the replication factor? My tests suggest that this does not happen (a brand new vmstorage instance never grows to match the disk space/number of metrics usage of older systems) but I am new to this software so want to be sure I'm not missing something.

Is the intended design to be running snapshots and recover a lost server with vmrestore?

Aliaksandr Valialkin

unread,

Apr 10, 2021, 12:50:33 PM4/10/21

to Seth Graham, victorametrics-users

On Thu, Apr 8, 2021 at 9:50 PM Seth Graham <set...@tru7h.org> wrote:

I am in the process of evaluating victoria metrics for my environment, and have a three node cluster running with replicationFactor=2. It's working well but I have a question regarding the behavior when a vmstorage server is permanently lost that does not appear to be covered by the documentation.

When a vmstorage server is lost, does the data surviving on another server ever get replicated to a new server to match the replication factor? My tests suggest that this does not happen (a brand new vmstorage instance never grows to match the disk space/number of metrics usage of older systems) but I am new to this software so want to be sure I'm not missing something.

VictoriaMetrics doesn't automatically restore replication factor for lost replicas, because this process may go out of control, which can lead to performance, reliability and durability issues for extended durations. For instance, it may lead to a broken cluster with a hard-to-recover state if some vmstorage nodes become unavailable during replication factor restoration. There are plans to add an ability to manually restore replication factor via vmctl tool.

Is the intended design to be running snapshots and recover a lost server with vmrestore?

Yes. This is described at https://valyala.medium.com/speeding-up-backups-for-big-time-series-databases-533c1a927883

--
You received this message because you are subscribed to the Google Groups "victorametrics-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to victorametrics-u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/victorametrics-users/8d7e7270-8948-4486-a572-5ce723e08ad3n%40googlegroups.com.