[confluence] MongoDB > EC2 Backup & Restore
nor...@mongodb.onconfluence.com
EC2 Backup & Restore
Page edited by Sam Kleinman
Changes (1)
... |
| h3. Backup the database files |
| There are several ways to create a EBS Snapshot, for example with [Elastic Fox|http://aws.amazon.com/developertools/609] or the [AWS Command line|http://docs.amazonwebservices.com/AWSEC2/latest/CommandLineReference/index.html]. The following examples use the AWS command line tool. |
h4. Find the EBS volumes associated with the MongoDB |
... |
Full Content
Overview
This article describes how to backup, verify & restore a MongoDB running on EC2 using EBS Snapshots.
Backup
How you backup MongoDB will depend on whether you are using the --journal option in 1.8 (or above) or not.
Backup with --journal
The journal file allows for roll forward recovery. The journal files are located in the dbpath directory so will be snapshotted at the same time as the database files.
If the dbpath is mapped to a single EBS volume then proceed to the Backup the Database Files section.
If you dbpath is mapped to multiple EBS volumes, in order to guarantee the stability of the file-system then you will need to Flush and Lock the Database section.
 | Note that snapshotting with the journal is only possible if the journal resides on the same volume as the data files, so that one snapshot operation captures the journal state and data file state atomically. |
Backup without --journal
In order to correctly backup a MongoDB, you need to ensure that writes are suspended to the file-system before you backup the file-system. If writes are not suspended then the backup may contain partially written or data which may fail to restore correctly.
Backing up MongoDB is simple to achieve using the fsync + lock command. If the file-system is being used only by the database, then you can then use the snapshot facility of EBS volumes to create a backup. If you are using the volume for any other application then you will need to ensure that the file-system is frozen as well (e.g. on XFS file-system use xfs_freeze) before you initiate the EBS snapshot. The overall process looks like:
Flush and Lock the database
Writes have to be suspended to the file-system in order to make a stable copy of the database files. This can be achieved through the MongoDB shell using the fsync + lock command.
mongo shell> use admin
mongo shell> db.runCommand({fsync:1,lock:1});
{
"info" : "now locked against writes, use db.$cmd.sys.unlock.findOne() to unlock",
"ok" : 1
}
During the time the database is locked, any write requests that this database receives will be rejected. Any application code will need to deal with these errors appropriately.
Backup the database files
There are several ways to create a EBS Snapshot, for example the AWS Command line. The following examples use the AWS command line tool.
Find the EBS volumes associated with the MongoDB
If the mapping of EBS Block devices to the MongoDB data volumes is already known, then this step can be skipped. The example below shows how to determine the mapping for an LVM volume, please confirm with your System Administrator how the original system was setup if you are unclear.
Find the EBS block devices associated with the running instance
shell> ec2-describe-instances RESERVATION r-eb09aa81 289727918005 tokyo,default INSTANCE i-78803e15 ami-4b4ba522 ec2-50-16-30-250.compute-1.amazonaws.com ip-10-204-215-62.ec2.internal running scaleout 0 m1.large 2010-11-04T02:15:34+0000 us-east-1a aki-0b4aa462 monitoring-disabled 50.16.30.250 10.204.215.62 ebs paravirtual BLOCKDEVICE /dev/sda1 vol-6ce9f105 2010-11-04T02:15:43.000Z BLOCKDEVICE /dev/sdf vol-96e8f0ff 2010-11-04T02:15:43.000Z BLOCKDEVICE /dev/sdh vol-90e8f0f9 2010-11-04T02:15:43.000Z BLOCKDEVICE /dev/sdg vol-68e9f101 2010-11-04T02:15:43.000Z BLOCKDEVICE /dev/sdi vol-94e8f0fd 2010-11-04T02:15:43.000Z
As can be seen in this example, there are a number of block devices associated with this instance. We have to determine which volumes make up the file-system we need to snapshot.
Determining how the dbpath is mapped to the file-system
Log onto the running MongoDB instance in EC2. To determine where the database file are located, either look at the startup parameters for the mongod process or if mongod is running, then you can examine the running process.
root> ps -ef | grep mongo ubuntu 10542 1 0 02:17 ? 00:00:00 /var/opt/mongodb/current/bin/mongod --port 27000 --shardsvr --dbpath /var/lib/mongodb/tokyo0 --fork --logpath /var/opt/mongodb/log/server.log --logappend --rest
dbpath is set to /var/lib/mongodb/tokyo0 in this example.
Mapping the dbpath to the physical devices
Using the df command, determine what the --dbpath directory is mapped to
root> df /var/lib/mongodb/tokyo0 Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/data_vg-data_vol 104802308 4320 104797988 1% /var/lib/mongodb
Next determine the logical volume associated with this device, in the example above /dev/mapper/data_vg-data_vol
root> lvdisplay /dev/mapper/data_vg-data_vol --- Logical volume --- LV Name /dev/data_vg/data_vol VG Name data_vg LV UUID fixOyX-6Aiw-PnBA-i2bp-ovUc-u9uu-TGvjxl LV Write Access read/write LV Status available # open 1 LV Size 100.00 GiB ...
This output indicates the volume group associated with this logical volume, in this example data_vg. Next determine how this maps to the physical volume.
root> pvscan PV /dev/md0 VG data_vg lvm2 [100.00 GiB / 0 free] Total: 1 [100.00 GiB] / in use: 1 [100.00 GiB] / in no VG: 0 [0 ]
From the physical volume, determine the associated physical devices, in this example /dev/md0.
root> mdadm --detail /dev/md0
/dev/md0:
Version : 00.90
Creation Time : Thu Nov 4 02:17:11 2010
Raid Level : raid10
Array Size : 104857472 (100.00 GiB 107.37 GB)
Used Dev Size : 52428736 (50.00 GiB 53.69 GB)
Raid Devices : 4
...
UUID : 07552c4d:6c11c875:e5a1de64:a9c2f2fc (local to host ip-10-204-215-62)
Events : 0.19
Number Major Minor RaidDevice State
0 8 80 0 active sync /dev/sdf
1 8 96 1 active sync /dev/sdg
2 8 112 2 active sync /dev/sdh
3 8 128 3 active sync /dev/sdi
We can see that block devices /dev/sdf through /dev/sdi make up this physical devices. Each of these volumes will need to be snapped in order to complete the backup of the file-system.
Create the EBS Snapshot
Create the snapshot for each devices. Using the ec2-create-snapshot command, use the Volume Id for the device listed by the ec2-describe-instances command.
shell> ec2-create-snapshot -d backup-20101103 vol-96e8f0ff SNAPSHOT snap-417af82b vol-96e8f0ff pending 2010-11-04T05:57:29+0000 289727918005 50 backup-20101103 shell> ec2-create-snapshot -d backup-20101103 vol-90e8f0f9 SNAPSHOT snap-5b7af831 vol-90e8f0f9 pending 2010-11-04T05:57:35+0000 289727918005 50 backup-20101103 shell> ec2-create-snapshot -d backup-20101103 vol-68e9f101 SNAPSHOT snap-577af83d vol-68e9f101 pending 2010-11-04T05:57:42+0000 289727918005 50 backup-20101103 shell> ec2-create-snapshot -d backup-20101103 vol-94e8f0fd SNAPSHOT snap-2d7af847 vol-94e8f0fd pending 2010-11-04T05:57:49+0000 289727918005 50 backup-20101103
Unlock the database
After the snapshots have been created, the database can be unlocked. After this command has been executed the database will be available to process write requests.
mongo shell> db.$cmd.sys.unlock.findOne();
{ "ok" : 1, "info" : "unlock requested" }
Verifying a backup
In order to verify the backup, the following steps need to be completes
- Check the status of the snapshot to ensure that they are "completed"
- Create new volumes based on the snapshots and mount the new volumes
- Run mongod and verify the collections
Typically, the verification will be performed on another machine so that you do not burden your production systems with the additional CPU and I/O load of the verification processing.
Describe the snapshots
Using the ec2-describe-snapshots command, find the snapshots that make up the backup. Using a filter on the description field, snapshots associated with the given backup are easily found. The search text used should match the text used in the -d flag passed to ec2-create-snapshot command when the backup was made.
backup shell> ec2-describe-snapshots --filter "description=backup-20101103"
SNAPSHOT snap-2d7af847 vol-94e8f0fd completed 2010-11-04T05:57:49+0000 100% 289727918005 50 backup-20101103
SNAPSHOT snap-417af82b vol-96e8f0ff completed 2010-11-04T05:57:29+0000 100% 289727918005 50 backup-20101103
SNAPSHOT snap-577af83d vol-68e9f101 completed 2010-11-04T05:57:42+0000 100% 289727918005 50 backup-20101103
SNAPSHOT snap-5b7af831 vol-90e8f0f9 completed 2010-11-04T05:57:35+0000 100% 289727918005 50 backup-20101103
Create new volumes based on the snapshots
Using the ec2-create-volume command, create a new volumes based on each of the snapshots that make up the backup.
backup shell> ec2-create-volume --availability-zone us-east-1a --snapshot snap-2d7af847 VOLUME vol-06aab26f 50 snap-2d7af847 us-east-1a creating 2010-11-04T06:44:27+0000 backup shell> ec2-create-volume --availability-zone us-east-1a --snapshot snap-417af82b VOLUME vol-1caab275 50 snap-417af82b us-east-1a creating 2010-11-04T06:44:38+0000 backup shell> ec2-create-volume --availability-zone us-east-1a --snapshot snap-577af83d VOLUME vol-12aab27b 50 snap-577af83d us-east-1a creating 2010-11-04T06:44:52+0000 backup shell> ec2-create-volume --availability-zone us-east-1a --snapshot snap-5b7af831 VOLUME vol-caaab2a3 50 snap-5b7af831 us-east-1a creating 2010-11-04T06:45:18+0000
Attach the new volumes to the instance
Using the ec2-attach-volume command, attach each volume to the instance where the backup will be verified.
backup shell> ec2-attach-volume --instance i-cad26ba7 --device /dev/sdp vol-06aab26f ATTACHMENT vol-06aab26f i-cad26ba7 /dev/sdp attaching 2010-11-04T06:49:32+0000 backup shell> ec2-attach-volume --instance i-cad26ba7 --device /dev/sdq vol-1caab275 ATTACHMENT vol-1caab275 i-cad26ba7 /dev/sdq attaching 2010-11-04T06:49:58+0000 backup shell> ec2-attach-volume --instance i-cad26ba7 --device /dev/sdr vol-12aab27b ATTACHMENT vol-12aab27b i-cad26ba7 /dev/sdr attaching 2010-11-04T06:50:13+0000 backup shell> ec2-attach-volume --instance i-cad26ba7 --device /dev/sds vol-caaab2a3 ATTACHMENT vol-caaab2a3 i-cad26ba7 /dev/sds attaching 2010-11-04T06:50:25+0000
Mount the volumes groups etc.
Make the file-system visible on the host O/S. This will vary by the Logical Volume Manager, file-system etc. that you are using. The example below shows how to perform this for LVM, please confirm with your System Administrator on how the original system system was setup if you are unclear.
Assemble the device from the physical devices. The UUID for the device will be the same as the original UUID that the backup was made from, and can be obtained using the mdadm command.
backup shell> mdadm --assemble --auto-update-homehost -u 07552c4d:6c11c875:e5a1de64:a9c2f2fc --no-degraded /dev/md0 mdadm: /dev/md0 has been started with 4 drives.
You can confirm that the physical volumes and volume groups appear correctly to the O/S by executing the following:
backup shell> pvscan PV /dev/md0 VG data_vg lvm2 [100.00 GiB / 0 free] Total: 1 [100.00 GiB] / in use: 1 [100.00 GiB] / in no VG: 0 [0 ] backup shell> vgscan Reading all physical volumes. This may take a while... Found volume group "data_vg" using metadata type lvm2
Create the mount point and mount the file-system:
backup shell> mkdir -p /var/lib/mongodb backup shell> cat >> /etc/fstab << EOF /dev/mapper/data_vg-data_vol /var/lib/mongodb xfs noatime,noexec,nodiratime 0 0 EOF backup shell> mount /var/lib/mongodb
Startup the database
After the file-system has been mounted, MongoDB can be started. Ensure that the owner of the files is set to the correct user & group. Since the backup was made with the database running, the lock file will need to be removed in order to start the database.
backup shell> chown -R mongodb /var/lib/mongodb/toyko0 backup shell> rm /var/lib/mongodb/tokyo0/mongod.lock backup shell> mongod --dbpath /var/lib/mongodb/tokyo0
Verify the collections
Each collection can be verified in order to ensure that it valid and does not contain any invalid BSON objects.
mongo shell> db.blogs.validate({full:true})
Restore
Restore uses the same basic steps as the verification process. 
After the file-system is mounted you can decide to
- Copy the database files from the backup into the current database directory
- Startup mongod from the new mount point, specifying the new mount point in the --dbpath argument
After the database is started, it will be ready to transact. It will be at the specific point in time from the backup, so if it is part of a master/slave or replica set relationship, then the instance will need to synchronize itself to get itself back up to date.