MongoDB oplogReplay not working

[Astro]

I am trying to perform the point in time recovery using mongorestore, oplogReplay and oploglimit. 

Following set of actions are performed:

• Dump

1.  I have a mongodump taken at 7:30 am in the morning. 
   
2.  Also, dumped oplog.rs collection from running mongodb.  The oplog.rs collection contains oplog since month of april.

•   Restore

1.  Restored dump taken in step 1
2.  Trying to restore oplogs taken in step 2 with:  
   mongorestore --oplogReplay --oplogLimit 1494229965:1 --port 27018

• Problem 

1. While restoring using oplogReplay, mongorestore exits with error:  
   Failed: restore error: error applying oplog: applyOps: Failed to apply insert due to missing collection
   
   
2. Note: There is a script which deletes unwanted collections periodically. 

• Question 

1. How to ensure point in time restore is such situation where the oplog contains old entries than the last mongodump
2. How to avoid above error when the collections may have been deleted?

Thanks.

[Astro]

 

Hi There!

Any help on this kind of restore?

[Kevin Adistambha]

Hi Astro,

How to ensure point in time restore is such situation where the oplog contains old entries than the last mongodump

I don’t think I fully understand your goal. Are you trying to perform a point-in-time restore?

The oplog is the log of all operations performed by MongoDB from a point in time in the past, to another point in time in the present. In other words, restoring with mongorestore --oplogReplay will replay all the states that the database went through between those two points in time. The only relevant option you can change is --oplogLimit, but this simply stops the replay at an earlier time than the present time. You cannot ignore part of the history that you don’t want, otherwise your database cannot be guaranteed to be consistent after the restore.

How to avoid above error when the collections may have been deleted?

Rather than dumping the oplog.rs collection manually, have you tried using mongodump with the --oplog option? This would dump the current state of the database, while at the same time record the operations going on in the database during the time mongodump is run. The mongorestore --oplogReplay option was meant to work with mongodump --oplog, and not a manual dump of the oplog.rs collection.

Best regards,
Kevin

​

[Astro]

Thanks, Kevin.

What are the other options to perform point-in-time restore apart from MongoDB Cloud Manager?

In this case, we have dump taken at 8:00 am, and the oplog backup runs every hour(the oplog.rs collection). Now that we have full backup taken at 8:00 am and oplog.rs collections backup, how can we perform point in time restore, lets say at 1:00 o'clock using the oplogs at hand?

Please guide.

Thanks.

[Kevin Adistambha]

Hi Astro

Sorry for the late reply. What you wanted to do is possible by instructing mongorestore to replay the oplog on top of your backup. Since the oplog is idempotent by design, applying the same oplog entry twice will not make a divergent version of the data.

First, I assume you have the dump of the oplog, e.g. the result of mongodump --host <host> -d local -c oplog.rs -o oplogDump.

Second, check the content of the oplog to determine the timestamp that you want to stop at by using e.g. bsondump oplogDump/local/oplog.rs.bson. For example, if you have a dropDatabase command and you want to stop the oplog replay just before the command was executed:

{"ts":{"$timestamp":{"t":1502172266,"i":1}},"t":{"$numberLong":"1"},"h":{"$numberLong":"7041819298365940282"},"v":2,"op":"c","ns":"test.$cmd","o":{"dropDatabase":1}}

Keep note of the t value in {"$timestamp":{"t":1502172266,"i":1}}

Third, restore using mongorestore --host <host> --oplogReplay --oplogLimit=1502172266 --oplogFile=oplogDump/local/oplog.rs.bson oplogDump

Note the parameter to oplogLimit, which is telling mongorestore to stop replaying the oplog once it hit that timestamp (which is the timestamp of the dropDatabase command).

The oplogFile parameter is new to MongoDB 3.4. For older versions, you would need to copy the oplogDump/local/oplog.rs.bson to the root of the dump directory to a file named oplog.bson, e.g. oplogDump/oplog.bson and remove the oplogFile parameter from the example command above.

Note that this only covers replaying the oplog to a standalone node (which could be extended to a replica set by restoring to a single node and doing initial sync). There is currently no easy/supported way to do a point-in-time restore to a sharded cluster. I would encourage you to have a recent backup and test the procedure before applying it to a production environment.

Best regards
Kevin

​