Issue with MongoDB backup using EC2 Snapshot

[Avitash Purohit]

We are using MongoDB v3.2.11. To take a backup we are using the following process

1. db.fsyncLock() : The response of the command is : 

{

"info" : "now locked against writes, use db.fsyncUnlock() to unlock",

"seeAlso" : "http://dochub.mongodb.org/core/fsynccommand",

"ok" : 1

}

2. Create snapshot using EC2 snapshot command. The response of the command is: 

{

    "Description": "This is a test_5 snapshot",

    "Encrypted": true,

    "VolumeId": "vol-*",

    "State": "pending",

    "VolumeSize": 700,

    "Progress": "",

    "StartTime": "2017-05-08T09:55:30.000Z",

    "SnapshotId": "snap-*",

    "OwnerId": "id"

}

3. db.fsyncUnlock() : The response of the command is : 

{ "info" : "unlock completed", "ok" : 1 }

While trying to restore our backups we found that the backup is failing sometimes. On debugging deeper we found that sometimes a file "Wiredtiger.backup" is created in the snapshot with 0 bytes. This file is actually created when fsynclock is called. It gets removed once the fsyncUnlock is called. 

In the case when Wiredtiger.backup exists the following log is printed : 

"WiredTiger Both WiredTiger.turtle and WiredTiger.backup exist; recreating metadata from backup"

After this mongodb does not show any data. It behaves like an empty database. We are using the default config file only.

Please let us know if we are doing something wrong.

[Kunal Khattar]

Hi Avitash,

We also use mongoDB on AWS and have experienced this exact problem on MongoDB v3.2.16 and our workaround was to remove the backup file on the restored volume. After that we started the mongod process and it worked fine.

Did you have another resolution for this issue?

This message may contain confidential, proprietary or privileged information. If you are not the intended recipient, please notify the sender immediately and delete the message from your system. You should not copy or use it for any purpose, nor disclose its contents to any other person. Email transmission cannot be guaranteed to be secure or error-free. No guarantee is made that any attachments are virus free. Although the information is believed to be reliable, we do not guarantee its accuracy and it may be incomplete or condensed. All information is subject to change without notice.

[Kevin Adistambha]

Hi

For a detailed information on EC2 backup procedure using EBS snapshots, please see the EC2 Backup and Restore page.

Specifically for backup using filesystem snapshots, the page Back Up and Restore with Filesystem Snapshots contains instructions on how to do so.

For more general information regarding MongoDB backup methods, please see MongoDB Backup Methods

Please note that some backup methods may differ between deployment topologies, e.g. some backup methods are specifically for standalone nodes, and others for replica sets.

Best regards
Kevin

​

[David Resnick]

This looks like the problem I described (and still haven't solved) from here:

https://groups.google.com/d/msg/mongodb-user/3NlWVuheMT8/f7zddZ5_BwAJ

Avitash and Kunar, did you ever find a better workaround than deleting the WiredTiger.backup file before restoring the snapshot? I'll try it at your suggestion, but in my case the file is often quite large.

These are the files I have on an instance that fails to restore:

-rw-r--r-- 1 mongodb mongodb 156M Jul 16 11:05 /data/db/_mdb_catalog.wt

-rw-r--r-- 1 mongodb mongodb 7 Jul 16 09:37 /data/db/mongod.lock

-rw-r--r-- 1 mongodb mongodb 2.4G Jul 16 11:05 /data/db/sizeStorer.wt

-rw-r--r-- 1 mongodb mongodb 95 Feb 5 2017 /data/db/storage.bson

-rw-r--r-- 1 mongodb mongodb 46 Feb 5 2017 /data/db/WiredTiger

-rw-r--r-- 1 mongodb mongodb 3.2G Jul 16 11:05 /data/db/WiredTiger.backup

-rw-r--r-- 1 mongodb mongodb 4.0K Jul 16 09:03 /data/db/WiredTigerLAS.wt

-rw-r--r-- 1 mongodb mongodb 21 Feb 5 2017 /data/db/WiredTiger.lock

-rw-r--r-- 1 mongodb mongodb 976 Jul 16 11:05 /data/db/WiredTiger.turtle

-rw-r--r-- 1 mongodb mongodb 3.4G Jul 16 11:05 /data/db/WiredTiger.wt