Mongo Cloud Manager restore a replica set is strangely complicated
Peng Wei
Stephen Steneker
On Wednesday, 23 September 2015 08:50:29 UTC+10, Peng Wei wrote:
I started using Mongo Cloud Manager a couple of days ago. I created a replica set through the manager, then I setup back up and tried to restore the replica set from a snapshot.
I thought it would be as simple as clicking a button and my data is restored. But instead I got a link to download a file (using the pull method).
Then I went to https://docs.cloud.mongodb.com/tutorial/restore-replica-set/ to see how to use this file to restore my db, and the page has like over 20 steps…
This seems kinda ridiculous; without the manager, I just have to run mongodump using a cron job to create daily backups and then use mongorestore to restore it. The restoring part is only 1 command. Why does the cloud manager ask me to do 20 steps?
Am I doing something wrong?
Hi Peng,
The backup provided by Cloud Manager provides “ready to use” data files similar to what you would get from a consistent filesystem backup of your dbPath. The backup includes prebuilt data files and indexes, and the restore point in Cloud Manager can be based on a selected snapshot or point-in-time in the last 24 hours.
The instructions you referenced describe how to rebuild your replica set using the backup data files for the new primary and then seeding the secondaries from the same backup to avoid the time taken by initial sync.
This is a more involved process in terms of administrative steps, but has a faster time to full recovery of a replica set than a mongorestore into an empty primary. A mongorestore would involve replicating the data to all members of the replica set via initial sync, and also include time to rebuild all indexes.
If you want to take an approach analogous to your cron’d backup and restore, you could always start your backup file on a standalone mongod and then mongodump and mongorestore --drop into your existing replica set.
Regards,
Stephen
A. Jalil @AJ
Speaking of bakckups, lets put AWS snapshots aside
for now, and would like to do this manually instead..
In Oracle world, we use Logical backup which is equivalent to mongodump. We also use cold backup where the database is
completely shutdown and all physical Oracle
data files on OS level are backed up accordingly. To do this in Mongo world, I am assuming to
stop mongod service, and backup/copy dbpath folder, that should be the equivalent
of a cold backup, right ?
We also use Hot Backups where physical data files are backed up while the
database is running. Is there an
equivalent of Hot Backup in Mongo ?
> Here are couple of scenarios
Scenario-1: I have 1 replica set of 3 members (1 primary
+ 2 secondary). To schedule a backup of
this replica set on cron, should I backup the primary instance only ? and when time to restore, I can mongorestore the same dump to each replica set, right ?
Scenario-2: I have a backup that runs every night at
11pm. The following day at 2pm my server crashes and I lost all 3
instances. Yes, that's a bad idea, I
should have these 3 instances on different servers, but hypothetically, I lost
everything. I only have the backup from
last night at 11pm. So I mongorestore the backup from last nigh 11pm, but I need to
apply transactions/journals from this morning at 8am up till 2pm to the point
where my server crashed. In Oracle
world, once I startup Oracle instance, it wakes up and it knows it is in
recovery mode, so it asks me to start applying the Archive logs. I was
wondering how mongo applies its journals in this type of situation.. I
mongorestore the dump file from last night, and then what’s next ?
Thank you.
@AJ
Stephen Steneker
On Thursday, 24 September 2015 13:21:37 UTC+10, A. Jalil @AJ wrote:
In Oracle world, we use Logical backup which is equivalent to mongodump. We also use cold backup where the database is completely shutdown and all physical Oracle data files on OS level are backed up accordingly. To do this in Mongo world, I am assuming to stop mongod service, and backup/copy dbpath folder, that should be the equivalent of a cold backup, right ?
We also use Hot Backups where physical data files are backed up while the database is running. Is there an equivalent of Hot Backup in Mongo ?
Hot backup approaches include:
- Filesystem/EBS snapshots (assuming all data + journal files are on the same volume)
- MongoDB Cloud/Ops Manager Backup
The MMAP storage engine also supports a db.fsyncLock() command you can use to quiesce and lock a running MongoDB instance in order to take a file copy backup (with a corresponding db.fsyncUnlock() to resume normal operation). The db.fsyncLock() command is not supported by the WiredTiger storage engine in MongoDB 3.0 (watch/upvote SERVER-18899 for WiredTiger support).
Here are couple of scenarios
Scenario-1: I have 1 replica set of 3 members (1 primary + 2 secondary). To schedule a backup of this replica set on cron, should I backup the primary instance only ? and when time to restore, I can mongorestore the same dump to each replica set, right ?
You only need to back up a single member of your replica set as all should have identical data (discounting any replication lag or intentional configuration variances). Backups are typically taken from a secondary in order to reduce impact on the primary and allow strategies like a db.fSyncLock() backup from a hidden secondary.
Scenario-2: I have a backup that runs every night at 11pm. The following day at 2pm my server crashes and I lost all 3 instances. Yes, that’s a bad idea, I should have these 3 instances on different servers, but hypothetically, I lost everything. I only have the backup from last night at 11pm.
So I mongorestore the backup from last nigh 11pm, but I need to apply transactions/journals from this morning at 8am up till 2pm to the point where my server crashed. In Oracle world, once I startup Oracle instance, it wakes up and it knows it is in recovery mode, so it asks me to start applying the Archive logs. I was wondering how mongo applies its journals in this type of situation.. I mongorestore the dump file from last night, and then what’s next ?
If you only have a backup from 11pm last night and have lost all your instances, that is the latest point you can recover to. A main goal of replica sets in MongoDB is data redundancy, so it does not make sense to have multiple replica set members on the same physical host. As long as there is a healthy surviving member of a replica set, this is usually the starting point for recovery.
If all the instances for a replica set have just crashed but you are able to restart them, they should recover automatically — there isn’t a separate transaction/journal log you can apply. A common backup strategy is to use an offsite backup tool like MongoDB Cloud Manager, which runs a continuous backup agent. You can also configure a replica set member as a delayed secondary which can mitigate against accidental deletion (for example adding a 1-hour delay before changes will be replicated to this secondary).
MongoDB’s journal records operations local to a node, and is only used for automatic local node recovery: http://docs.mongodb.org/manual/core/journaling/. The MongoDB journal is used for single node crash recovery and durability, and is not equivalent to an Oracle redo log.
Replicated operations are written to an oplog (operations log), which is a special capped collection in the local database for each node. This is conceptually similar to an Oracle redo log. If you had a bad operation (for example, accidental drop of a collection), it is possible to restore to a point in time using mongorestore (see: Modify and replay MongoDB oplog). It only makes sense to apply oplog changes if there is a point in common between the oplog backup and the oplog for a given node.
FYI, point-in-time restore is one of the homework scenarios covered in the M202 (Advanced Deployment and Operations) course ;-).
Regards,
Stephen