I am following the guide for EC2 backup and restore (http://goo.gl/mzKUK). I have created an EC2 image from the 10gen provided AMI on the AWS Marketplace (http://goo.gl/v1PzT) which uses a Raid 10 EBS for storage. I am able to create backup snapshots of all 4 drives. My real problem is with restoring them! My full steps are:
1. Create a new RAID 10 Mongo instance using this: http://goo.gl/v1PzT 2. Start up mongo, put it some test data to see if it gets backed up 3. Create snapshots of all 4 raid drives as described http://goo.gl/mzKUK 4. Create a new instance, using the same base AMI from the AWS Marketplace link above in step 1 5. Create volumes from the snapshots made and attach them to the instance 6. *Where things break* - Mounting the volumes groups section on the 10gen howto page (http://goo.gl/mzKUK) 1. mdadm has no *--auto-update-homehost* option 2. The following is the commands and results that are causing the issues
mdadm --assemb -u ec655893:c8ed37fa:b05bfb20:fd44dbd4 --no-degraded /dev/md0 mdadm: /dev/md0 has been started with 4 drives.
pvscan Found duplicate PV 5aZHqMeG2jHqmK1c1d1jellaYZIYA6U2: using /dev/md127 not /dev/md0 PV /dev/md127 VG vg0 lvm2 [200.00 GiB / 8.00 MiB free] Total: 1 [200.00 GiB] / in use: 1 [200.00 GiB] / in no VG: 0 [0 ]
vgscan Reading all physical volumes. This may take a while... Found duplicate PV 5aZHqMeG2jHqmK1c1d1jellaYZIYA6U2: using /dev/md127 not /dev/md0 Found volume group "vg0" using metadata type lvm2
mount /var/lib/mongodb mount: /dev/mapper/vg0-data already mounted or /var/lib/mongodb busy mount: according to mtab, /dev/mapper/vg0-data is mounted on /data
I believe that what is happening is that since both volumes are coming from the same root AMI, that they are registered the same (ie same UUID?). But I have no idea on how to actual restore these volumes.
It does seem though, that since this is a 10gen provided AMI, that there should be a documented way to restore data.
If anyone needs access to this box to test, I have keys and addresses I can post.
Chase,
How did you conduct the backup? Was the database running or did you fsync+lock and then take the snapshots?
If you launched another instance of the same AMI with EBS RAID there could be issues trying to restore the backup onto that box because of device name collisions, etc.
Please send me the keys/addresses and I will definitely take a look (off list is fine).
On Saturday, October 6, 2012 at 12:50 PM, Chase wrote:
> I am following the guide for EC2 backup and restore (http://goo.gl/mzKUK). I have created an EC2 image from the 10gen provided AMI on the AWS Marketplace (http://goo.gl/v1PzT) which uses a Raid 10 EBS for storage. I am able to create backup snapshots of all 4 drives. My real problem is with restoring them! My full steps are:
> Create a new RAID 10 Mongo instance using this: http://goo.gl/v1PzT > Start up mongo, put it some test data to see if it gets backed up
> Create snapshots of all 4 raid drives as described http://goo.gl/mzKUK > Create a new instance, using the same base AMI from the AWS Marketplace link above in step 1
> Create volumes from the snapshots made and attach them to the instance
> Where things break - Mounting the volumes groups section on the 10gen howto page (http://goo.gl/mzKUK)
> mdadm has no --auto-update-homehost option
> The following is the commands and results that are causing the issues
> > mount /var/lib/mongodb
> > mount: /dev/mapper/vg0-data already mounted or /var/lib/mongodb busy
> > mount: according to mtab, /dev/mapper/vg0-data is mounted on /data
> I believe that what is happening is that since both volumes are coming from the same root AMI, that they are registered the same (ie same UUID?). But I have no idea on how to actual restore these volumes.
> It does seem though, that since this is a 10gen provided AMI, that there should be a documented way to restore data.
> If anyone needs access to this box to test, I have keys and addresses I can post.
> Thanks for any help!
> -- > You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongodb-user@googlegroups.com (mailto:mongodb-user@googlegroups.com)
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe@googlegroups.com (mailto:mongodb-user+unsubscribe@googlegroups.com)
> See also the IRC channel -- freenode.net (http://freenode.net)#mongodb
Thank you for responding! I was banging my head on this one.
- For how I did the backup: Did a mongo lock via `db.runCommand({fsync:
1, lock: 1});` and then took snapshots, just like how this guide recommends
http://goo.gl/mzKUK - Yes, I did launch another instance from the same AMI (the 10gen
created one). It seems like you would want to restore the backup to the
same AMI to avoid changing configurations of the database and operating
system right?
- I will email to you keys and addresses now offline
On Mon, Oct 8, 2012 at 4:16 PM, Sandeep Parikh <s...@10gen.com> wrote:
> Chase,
> - How did you conduct the backup? Was the database running or did you
> fsync+lock and then take the snapshots?
> - If you launched another instance of the same AMI with EBS RAID there
> could be issues trying to restore the backup onto that box because of
> device name collisions, etc.
> Please send me the keys/addresses and I will definitely take a look (off
> list is fine).
> -Sandeep
> On Saturday, October 6, 2012 at 12:50 PM, Chase wrote:
> I am following the guide for EC2 backup and restore (http://goo.gl/mzKUK).
> I have created an EC2 image from the 10gen provided AMI on the AWS
> Marketplace (http://goo.gl/v1PzT) which uses a Raid 10 EBS for storage. I
> am able to create backup snapshots of all 4 drives. My real problem is with
> restoring them! My full steps are:
> 1. Create a new RAID 10 Mongo instance using this: http://goo.gl/v1PzT > 2. Start up mongo, put it some test data to see if it gets backed up
> 3. Create snapshots of all 4 raid drives as described
> http://goo.gl/mzKUK > 4. Create a new instance, using the same base AMI from the AWS
> Marketplace link above in step 1
> 5. Create volumes from the snapshots made and attach them to the
> instance
> 6. *Where things break* - Mounting the volumes groups section on the
> 10gen howto page (http://goo.gl/mzKUK)
> 1. mdadm has no *--auto-update-homehost* option
> 2. The following is the commands and results that are causing the
> issues
> mdadm --assemb -u ec655893:c8ed37fa:b05bfb20:fd44dbd4 --no-degraded
> /dev/md0
> mdadm: /dev/md0 has been started with 4 drives.
> pvscan
> Found duplicate PV 5aZHqMeG2jHqmK1c1d1jellaYZIYA6U2: using /dev/md127 not
> /dev/md0
> PV /dev/md127 VG vg0 lvm2 [200.00 GiB / 8.00 MiB free]
> Total: 1 [200.00 GiB] / in use: 1 [200.00 GiB] / in no VG: 0 [0 ]
> vgscan
> Reading all physical volumes. This may take a while...
> Found duplicate PV 5aZHqMeG2jHqmK1c1d1jellaYZIYA6U2: using /dev/md127 not
> /dev/md0
> Found volume group "vg0" using metadata type lvm2
> mount /var/lib/mongodb
> mount: /dev/mapper/vg0-data already mounted or /var/lib/mongodb busy
> mount: according to mtab, /dev/mapper/vg0-data is mounted on /data
> I believe that what is happening is that since both volumes are coming
> from the same root AMI, that they are registered the same (ie same UUID?).
> But I have no idea on how to actual restore these volumes.
> It does seem though, that since this is a 10gen provided AMI, that there
> should be a documented way to restore data.
> If anyone needs access to this box to test, I have keys and addresses I
> can post.
> Thanks for any help!
> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongodb-user@googlegroups.com
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe@googlegroups.com
> See also the IRC channel -- freenode.net#mongodb
> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongodb-user@googlegroups.com
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe@googlegroups.com
> See also the IRC channel -- freenode.net#mongodb
Just out of interest: would the mongodb-maintenance<https://github.com/rudionrails/mongodb-maintenance>script compilation be of interest to you when backing up your stuff? I could add restoration scripts if there are people in the need for that feature :-)
Am Samstag, 6. Oktober 2012 19:50:16 UTC+2 schrieb Chase:
> I am following the guide for EC2 backup and restore (http://goo.gl/mzKUK). > I have created an EC2 image from the 10gen provided AMI on the AWS > Marketplace (http://goo.gl/v1PzT) which uses a Raid 10 EBS for storage. I > am able to create backup snapshots of all 4 drives. My real problem is with > restoring them! My full steps are:
> 1. Create a new RAID 10 Mongo instance using this: http://goo.gl/v1PzT > 2. Start up mongo, put it some test data to see if it gets backed up > 3. Create snapshots of all 4 raid drives as described > http://goo.gl/mzKUK > 4. Create a new instance, using the same base AMI from the AWS > Marketplace link above in step 1 > 5. Create volumes from the snapshots made and attach them to the > instance > 6. *Where things break* - Mounting the volumes groups section on the > 10gen howto page (http://goo.gl/mzKUK) > 1. mdadm has no *--auto-update-homehost* option > 2. The following is the commands and results that are causing the > issues
> mdadm --assemb -u ec655893:c8ed37fa:b05bfb20:fd44dbd4 --no-degraded > /dev/md0 > mdadm: /dev/md0 has been started with 4 drives.
> pvscan > Found duplicate PV 5aZHqMeG2jHqmK1c1d1jellaYZIYA6U2: using /dev/md127 not > /dev/md0 > PV /dev/md127 VG vg0 lvm2 [200.00 GiB / 8.00 MiB free] > Total: 1 [200.00 GiB] / in use: 1 [200.00 GiB] / in no VG: 0 [0 ]
> vgscan > Reading all physical volumes. This may take a while... > Found duplicate PV 5aZHqMeG2jHqmK1c1d1jellaYZIYA6U2: using /dev/md127 not > /dev/md0 > Found volume group "vg0" using metadata type lvm2
> mount /var/lib/mongodb > mount: /dev/mapper/vg0-data already mounted or /var/lib/mongodb busy > mount: according to mtab, /dev/mapper/vg0-data is mounted on /data
> I believe that what is happening is that since both volumes are coming > from the same root AMI, that they are registered the same (ie same UUID?). > But I have no idea on how to actual restore these volumes.
> It does seem though, that since this is a 10gen provided AMI, that there > should be a documented way to restore data.
> If anyone needs access to this box to test, I have keys and addresses I > can post.
Sandeep emailed me, and it worked! Thanks very very much. This was seriously very very helpful. I have documented the whole recovery process and the commands I used below for reference for others.
- Create snapshots of current volume using the guide - Start and connect to a new mongo instance - View logical volumes: lvdisplay -v - Unmount data volume: umount /dev/vg0/data - Unmount log volume: umount /dev/vg0/log - Unmount journal volume: umount /dev/vg0/journal - Remove the volume group: vgremove vg0 - Show physical volumes: pvs -a - Remove the physical volume: pvremove /dev/md127 - Stop the RAID: mdadm --stop /dev/md127 - Detach volumes through AWS, wait for confirmation that they are dettached - Restart instance - Reattach all volumes, wait for confirmation that they are attached - View logical volumes, notice that they probably say NOT Available: lvdisplay -v - Fix the data volume: lvchange -a y /dev/vg0/data - Fix the log volume: lvchange -a y /dev/vg0/log - Fix the journal volume: lvchange -a y /dev/vg0/journal - View logical volumes, notice that they now say available: lvdisplay -v - Mount all the volumes!: mount /data, /log, /journal - Remove the mongo lock file (since the snapshot happened with the lock): rm -rf /data/mongod.lock - Start mongo up: /etc/init.d/mongod start - Wait for mongo to finish allocating and starting up - Check data - Success! -
On Saturday, October 6, 2012 11:50:16 AM UTC-6, Chase wrote:
> I am following the guide for EC2 backup and restore (http://goo.gl/mzKUK). > I have created an EC2 image from the 10gen provided AMI on the AWS > Marketplace (http://goo.gl/v1PzT) which uses a Raid 10 EBS for storage. I > am able to create backup snapshots of all 4 drives. My real problem is with > restoring them! My full steps are:
> 1. Create a new RAID 10 Mongo instance using this: http://goo.gl/v1PzT > 2. Start up mongo, put it some test data to see if it gets backed up > 3. Create snapshots of all 4 raid drives as described > http://goo.gl/mzKUK > 4. Create a new instance, using the same base AMI from the AWS > Marketplace link above in step 1 > 5. Create volumes from the snapshots made and attach them to the > instance > 6. *Where things break* - Mounting the volumes groups section on the > 10gen howto page (http://goo.gl/mzKUK) > 1. mdadm has no *--auto-update-homehost* option > 2. The following is the commands and results that are causing the > issues
> mdadm --assemb -u ec655893:c8ed37fa:b05bfb20:fd44dbd4 --no-degraded > /dev/md0 > mdadm: /dev/md0 has been started with 4 drives.
> pvscan > Found duplicate PV 5aZHqMeG2jHqmK1c1d1jellaYZIYA6U2: using /dev/md127 not > /dev/md0 > PV /dev/md127 VG vg0 lvm2 [200.00 GiB / 8.00 MiB free] > Total: 1 [200.00 GiB] / in use: 1 [200.00 GiB] / in no VG: 0 [0 ]
> vgscan > Reading all physical volumes. This may take a while... > Found duplicate PV 5aZHqMeG2jHqmK1c1d1jellaYZIYA6U2: using /dev/md127 not > /dev/md0 > Found volume group "vg0" using metadata type lvm2
> mount /var/lib/mongodb > mount: /dev/mapper/vg0-data already mounted or /var/lib/mongodb busy > mount: according to mtab, /dev/mapper/vg0-data is mounted on /data
> I believe that what is happening is that since both volumes are coming > from the same root AMI, that they are registered the same (ie same UUID?). > But I have no idea on how to actual restore these volumes.
> It does seem though, that since this is a 10gen provided AMI, that there > should be a documented way to restore data.
> If anyone needs access to this box to test, I have keys and addresses I > can post.
