Restoring backup in replica set from LVM snapshot - seeding secondaries with data from snapshot

429 views

Skip to first unread message

Andrzej Dębski

unread,

Jun 1, 2016, 7:39:11 PM6/1/16

to mongodb-user

Hello,

I have few questions about performing and restoring LVM backups in the replica set for which I can't find straight answers in the mongodb documentation.

Let us assume that LVM snapshot is performed on MongoDB 3.2 WiredTiger (or Mongo older then 3.2 with MMAP engine) with journal on the same volume as the database. Backups will be performed on one of the SECONDARY members (be it with or without locking the database) and I am backing up whole dbPath directory (so local database with information about current replica set status is also backed up). Also we can assume that replica set name is "replSet"

Now in a situation where I want to perform full disaster recovery - I want to have mongo replica set with 3 members with data straight from LVM snapshot) and I want to seed SECONDARY members with the data from the snapshots to avoid initial sync, question is what do I do with backed up local database.

I observed that in a situation where I start mongod instance over data from the snapshot (mongod --dbPath <someDirectory> --replSet replSet) I can observe that this new mongod instance reports being a member of replSet and also it already knowns about previous replica set members (probably based on the local database contents). What I tried is:

start mongod not as part of replica set
drop local database
restart mongod as part of replSet and initiate new replica set
shut it down
copy the contents of dbPath to other secondaries (with new version of local database that contains only single replica set member)
restart PRIMARY
start SECONDARIES over the data copied from new PRIMARY
add those SECONDARIES to replica set by invoking rs.add on primary

and I think by doing this SECONDARIES did not perform full initial sync (removing contents of dbPath and then downloading all data from primary) but only replicated changes that happened on PRIMARY but were not present on SECONDARY (I assume it was done based on last oplog operation in SECONDARY and PRIMARY).

I am wondering if those steps I outlined are correct (or maybe this process can be made even simpler) because section https://docs.mongodb.com/manual/tutorial/backup-with-filesystem-snapshots/#backup-and-restore-with-filesystem-snapshots does not say anything about actually restoring the data and next page https://docs.mongodb.com/manual/tutorial/restore-replica-set-from-backup/ instruct us to invoke rs.initiate() but it is not possible after performing LVM snapshot because local database is still present so the member is already part of old replica set.

Basically I would like to know

can I just drop local database from LVM snapshot data
when dbPath cleanup with full initial sync is performed and when mongod assumes that the data in dbPath is "good-enough" for the seed and only replicates missing data from the primary (after adding the SECONDARY with rs.add() on PRIMARY)

Kevin Adistambha

unread,

Jun 19, 2016, 11:17:08 PM6/19/16

to mongodb-user

Hi Andrzej,

I am wondering if those steps I outlined are correct (or maybe this process can be made even simpler) because section https://docs.mongodb.com/manual/tutorial/backup-with-filesystem-snapshots/#backup-and-restore-with-filesystem-snapshots does not say anything about actually restoring the data and next page https://docs.mongodb.com/manual/tutorial/restore-replica-set-from-backup/ instruct us to invoke rs.initiate() but it is not possible after performing LVM snapshot because local database is still present so the member is already part of old replica set.

Please note that the steps you outlined is not a supported nor tested method of restoring a replica set from filesystem snapshots. Specifically, the method outlined in the Backup and Restore with Filesystem Snapshots concerns a standalone deployment, and the page Restore a Replica Set from MongoDB Backups uses initial sync in the procedure.

I believe the page you are looking for is Sync by Copying Data Files from Another Member, where it describes a situation where you need to “seed” a member of a replica set.

Basically I would like to know

can I just drop local database from LVM snapshot data

As mentioned in the page Sync by Copying Data Files from Another Member, the local database must exist.

when dbPath cleanup with full initial sync is performed and when mongod assumes that the data in dbPath is “good-enough” for the seed and only replicates missing data from the primary (after adding the SECONDARY with rs.add() on PRIMARY)

In general, initial sync is performed when a node has no data, or when the member has data but is missing a history of the set’s replication (e.g. a missing local database). Please see the Initial Sync page for more details.