Incremental backups with oplog

[Ralf Kistner]

Background

I'm looking for an incremental backup solution for MongoDB. While point-in-time restore functionality is not a requirement, it would be nice to have. I'm running a replica set without sharding.

Some options I've tried include just taking a daily snapshot with mongodump, or storing incremental diffs of this with duplicity. However, with my data set being a few hundred GB, these approaches are too slow.

MMS Backups would be the perfect solution. Unfortunately I can't store the backups in the US because of privacy regulations, so I'm looking for something that runs on-premise. I am considering on-premise MMS as an option, but also want to look at other options.

Oplog approach

I'm looking into scripting a solution that backups up periodic dumps / snapshots, and then continuously saving the oplog. Would an approach like this work?

Backup:

1. Take a weekly snapshot with mongodump --oplog.

2. Every hour (assume the oplog always has more than an hour of changes), dump all new data from the oplog with something like this:

    mongodump -d local -c oplog.rs --query "{ts : { "$gt" : { "$timestamp" : { "t" : (timestamp here) } } }}"

    Some care will have to be taken to keep track of the last seen timestamp, and also that there are no gaps in the oplog.

3. Move these files to some offsite backup location.

Restore:

1. Take the latest full dump, and restore with mongorestore --oplogReplay.

2. For each oplog dump after that (in order):

       a. Place the oplog in an empty folder, say "oplog-n", as oplog.bson (instead of local/oplog.rs.bson)

       b. Restore the oplog with mongorestore --oplogReplay "oplog-n".

Notes:

* Point-in-time restore can be achieved by using the --oplogLimit option.

* The restore can be done with a single mongorestore command by concatenating the oplog.bson files (may need some care to make the format is valid).

Would this approach work? It feels like it's not the intended use case for the --oplogReplay command, but mongorestore doesn't complain when doing it.

I'm aware of the Tarja project, that does something similar. However, I'll be more comfortable using only standard MongoDB commands and some bash-like scripting.

Regards,
Ralf Kistner

[Asya Kamsky]

Yep, that looks good.

Key points - make sure that there are no gaps in the oplog!

If you want to concatenate, make sure there are no overlaps either -
technically replaying "should" be okay but there are cases where it
might not be (the same document being deleted and re-inserted
interacting with unique indexes or some such)...

But I don't see why this isn't the perfect use of oplogReplay :)

Asya

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user"
> group.
>
> For other MongoDB technical support options, see:
> http://www.mongodb.org/about/support/.
> ---
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mongodb-user...@googlegroups.com.
> To post to this group, send email to mongod...@googlegroups.com.
> Visit this group at http://groups.google.com/group/mongodb-user.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/mongodb-user/4a03253a-318f-4c85-9b29-4e23a1fd517a%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Ralf Kistner]

Thanks for the input!

I've adapted the process a little:

For an initial/full backup, instead of doing `mongodump --oplog`, I instead do the following:

1. Get the timestamp of the last oplog entry, and save it as the current position: db.oplog.rs.find().sort({'$natural': -1}).limit(1)[0]['ts']

2. Perform a normal mongodump.

3. Save the oplog from the saved position onwards, as described below.

I understand that this is essentially the same as what `mongodump --oplog` does, but integrates better with the rest of this backup process.

For an incremental (oplog) backup, I now do the following:

1. Get the current position.

2. Dump the oplog from the current position onwards: mongodump -d local -c oplog.js --query \"{ts : { \\$gte : { \\$timestamp : <position> } } }}\"

3. Save the new position as the last timestamp from the dumped oplog.

With this process there is always an overlap of exactly one entry between subsequent oplog backups. This makes it easy to verify the integrity of the backup (that there are no gaps in the dumped oplogs).

Restoring is done in two steps:

1. Concatenate all the dumped oplogs, removing duplicate entries (the overlap of 1), and checking to make sure they're in order and without gaps. This is the only data manipulation that the script performs without using the built-in tools directly.

2. Restore the dump with a single `mongorestore --oplogReplay`.

If a restore to a specific point-in-time is required, it can be done by filtering the oplogs in the concatenation step, or by using the --oplogLimit argument for mongorestore. 

I started building a proof-of-concept here:

https://github.com/journeyapps/mongo-oplog-backup

It still needs some work and a lot of testing before it will be ready for production use though.

-Ralf

[Styopa Semenukha]

Thanks for your findings and scripts! That's exactly what I was looking for.

One thought though: what if during initial full backup the data changes between steps 1 and 2? E.g. each number in the collection multiplied by 2, so mongodump data will be 2x, and when you replay the oplog, it will multiply it yet again, so the result will be 4x.

I assume mongodump without --oplog will not be aware of the "current" oplog position.

[Ralf Kistner]

This method depends on the oplog being idempotent, same as mongodump --oplog. If you for example have a record {count: 5} and multiply count by 2, MongoDB will store the operation as something like {$set: {count: 10}} in the oplog, instead of storing the multiplication operation. When the oplog is replayed, it will still have the same value. It's not well documented, but there are some more details here: http://docs.mongodb.org/manual/core/replica-set-oplog/

Without doing something like this, there is no reliable way (that I know of) to get a full dump on a live system, without blocking all writes for the duration of the dump or using filesystem snapshots.

-Ralf

[Styopa Semenukha]

Now it all makes sense, thank you Ralf!

On Monday, March 02, 2015 11:59:25 PM Ralf Kistner wrote:
> This method depends on the oplog being idempotent, same as mongodump
> --oplog. If you for example have a record {count: 5} and multiply count by
> 2, MongoDB will store the operation as something like {$set: {count: 10}}
> in the oplog, instead of storing the multiplication operation. When the
> oplog is replayed, it will still have the same value. It's not well
> documented, but there are some more details here:
> http://docs.mongodb.org/manual/core/replica-set-oplog/
>
> Without doing something like this, there is no reliable way (that I know
> of) to get a full dump on a live system, without blocking all writes for
> the duration of the dump or using filesystem snapshots.
>
> -Ralf

--
Best regards,
Styopa Semenukha.

[Anil Karamchandani]

thanks Ralf that helped.