Ready-to-roll System Backup
AlanM
I'm trying to come up with a backup solution that will allow me to
incrementally clone the entire system (Box A) each night to another
machine on the network (Box B) so that if Box A goes down, Box B will
be an exact replica of Box A, ready to boot up and run.
What's the best way to achieve this? I've looked at tar solutions,
rsync solutions, dd solutions...I'm just not sure which method would
be best for this situation.
Thanks in advance!
Alan
Douglas Mayne
I guess it depends on how exact the backup needs to be. Most backup
techniques are like taking photographs, and photography is a
discrete process. That is, if you were using photographs to rank the
finishers at a horse race, then one photograph might establish the
winner, but that may not be enough to fix "place" and "show." The
horses keep moving; likewise, your filesystem is in continuous motion, too.
The first step is to establish you requirements in finer detail. Here are
some items which might be included for consideration:
possible failure modes,
risks associated with downtime,
expense of hardware,
discrete or continuous backup strategy (is losing transactions permitted),
the chosen solution's complexity.
Here are some possibilities, rated from lower to higher complexity:
1. LVMv2 (device mapper) snapshots. I wrote about this here:
http://groups.google.com/group/comp.os.linux.misc/msg/6c9e22af15298610
A discrete snapshot can be restored to an earlier state. Some downtime is
to be expected.
2. Two boxes, which are hosts for a virtual machine which is stored as a
virtual machine image on a SAN.
SAN
VM Host 1 ----------+-------------VM Host 2
If hardware fails one VM Host, then the other can take over while it is
repaired with very little downtime. This solution assumes no failure in
the SAN and the both VM host boxes are not disabled during the same time
interval or maintenance window.
3. Live, "on the wire" clustering. Each box in the cluster does
its own calculations. If the primary cluster box fails, then another
can take over using some failover mechanism. This is beyond my experience.
I know it exists, though.
--
Douglas Mayne
AlanM
the second. The backup would only be run once each night. Basically, I
just need the backup server to be a duplicate of the main server so
that if the main server goes down, we can just call networking and
have the DNS pointed to the backup until the main server is fixed (or
replaced).
I suppose rsync would do the trick but i was hoping to do an entire
system clone in one fell swoop. I guess one way of doing it might be
to clone the basic system, then do an rsync each night just for all
the data that needs to be backed up. (It's a web server so just
rsyncing the web docs would suffice, don't you think?)
Alan
Douglas Mayne
> Thanks for the response. It really doesn't have to be exact down to
> the second. The backup would only be run once each night. Basically, I
> just need the backup server to be a duplicate of the main server so
> that if the main server goes down, we can just call networking and
> have the DNS pointed to the backup until the main server is fixed (or
> replaced).
>
> I suppose rsync would do the trick but i was hoping to do an entire
> system clone in one fell swoop. I guess one way of doing it might be
> to clone the basic system, then do an rsync each night just for all
> the data that needs to be backed up. (It's a web server so just
> rsyncing the web docs would suffice, don't you think?)
>
Idea: rsync based solution using a device mapper snapshot as its source
Steps (in general):
0. Have a box which you would like to mirror (call it box 1).
1. Build a box (box 2) with nearly identical hardware to box 1 (disc
capacity, controller, etc.)
2. Plan your backup strategy, with attention to box 2's
bootstrap. Box 2 probably needs to be either a dual boot system (with a
separte disk), or bootable from a live CD. The goal is a platform where
networking and rsync is available to perform the backup of box 1. An
independant boot allows the backup to run without interferring with the
running system. A live CD based solution may be preferable because it is
less intrusive. With a little planning, both boxes can be used as either
the primary (live) or secondary (on backup) system.
3. Install an initial image on box 2. Probably, restore a backup of
box 1 to box 2 and fix bootloader, etc.
4. On box 1: schedule a task to create a device mapper snapshot at a
certain time of day. Mount the "fixed" image (device mapper device) into
the filespace.
5. On box 2: schedule a task to update the image on box 2 using rsync,
using the mount point reference above. Send a signal to box 1 when
complete.
6. On box 1: when the "complete" signal is noticed/arrives from box 2,
then "tear down" the fixed image (umount, dmsetup remove, etc.).
7. Rinse and repeat.
If these instructions don't help you, then maybe they'll help me because I
think this strategy has some potential for simple backups. I am impressed
with the capabilities of the device mapper facility.
--
Douglas Mayne
AlanM
back up one of the system partitions.../usr, for example, couldn't I
just tar the partition that it's on? For example,
# mount /dev/sda5 /backup
# tar -cvjpf sda5.tar.bz2 backup
Would I be able to do this without booting from a live CD and still
get an exact duplicate of that partition?
Thanks!
Alan
Douglas Mayne
<snip>
Also, a live CD image can be installed to a hard disk. This eliminates
the need to boot from a separate disk. The box's boot menu would include
options for the mode the box would work in, primary (live), or secondary
(on backup.)
This post shows how to setup Slax on a HD:
http://groups.google.com/group/comp.os.linux.misc/msg/62ab4e99488378dd
>>
>> 3. Install an initial image on box 2. Probably, restore a backup of box
>> 1 to box 2 and fix bootloader, etc.
>>
>> 4. On box 1: schedule a task to create a device mapper snapshot at a
>> certain time of day. Mount the "fixed" image (device mapper device)
>> into the filespace.
>>
>> 5. On box 2: schedule a task to update the image on box 2 using rsync,
>> using the mount point reference above. Send a signal to box 1 when
>> complete.
>>
>> 6. On box 1: when the "complete" signal is noticed/arrives from box 2,
>> then "tear down" the fixed image (umount, dmsetup remove, etc.).
>>
>> 7. Rinse and repeat.
>>
>> If these instructions don't help you, then maybe they'll help me
>> because I think this strategy has some potential for simple backups. I
>> am impressed with the capabilities of the device mapper facility.
>>
<paste>
> That sounds like a feasible plan. One question though. If I wanted to
> back up one of the system partitions.../usr, for example, couldn't I
> just tar the partition that it's on? For example,
>
> # mount /dev/sda5 /backup
What is the context of this mount? The previous posts discussed networking
between two boxes.
> # tar -cvjpf sda5.tar.bz2 backup
>
> Would I be able to do this without booting from a live CD and still get
> an exact duplicate of that partition?
>
> Thanks!
>
Note: Some comments inline.
Please, do not top post. I should have said that earlier. I have fixed the
last reply to follow my response.
Back to the subject. I am a little confused about the context (how will
you mount your target partition?) in this folloup question, especially
given the previous discussion. In any case, the key phrase is "exact
duplicate." If you are attempting to backup the system which is booted
(i.e, not a live CD), then you have to worry about open files and files
subject to change during the time it takes to backup. Using a live CD, or
a dual boot system, are easy ways to achieve a coherent state for the
backup. Device mapper is a way to "fix" a snapshot. I suppose my
previous instructions could be adapted to use device mapper on both boxes,
but this seems somewhat less straight-forward. For one thing, it would be
clear which mode a given box was working (either as the backup, or as the
online server.) To change modes, a reboot is required, and the user
selects the box's mode at reboot.) Again, both boxes could be setup to
boot in either mode. Other approaches could be used to eliminate the need
for a reboot, but I discussed the approach I thought most simple. You can
adapt liberally to meet your specific needs.
If you have questions at a more fundamental level, I recommend studying
the hard disk upgrade howto:
http://tldp.org/HOWTO/Hard-Disk-Upgrade/index.html
It is definitely worth studying and practicing with that howto.
Understanding the basics of filesystems and bootloaders at this level will
definitely help you in your GNU/Linux career.
--
Douglas Mayne
gray
there is anyway to create this effect with commands such as rsync.
Maybe.
After having a motherboard fail recently with a 2 month repair/
warranty cycle I decided that this was no way to run a multimedia
business. With multimedia linux work I find that getting my system
installed, configured and tweaked is very time-consuming. And once
the system is running well I become timid of playing with the
installation and running the risk of corrupting it (managed to corrupt
my FC4_64 multimedia setup Glibs trying to build ardour2 recently).
All this means that having a backup of a home partition is far from
adequate. The root partition and the hardware/software interactions
are critical too. So I need clone systems. Here is my plan ; I
haven't fully implemented this yet although I have built my cloned
systems and worked most of the neccessary commands out ; My strategy
doesn't involve networking. Feedback welcome.
1. 2 machines with identical hardware - thus a supply of swaps for
all parts. The second machine to be used by my wife unless needed in
my backup strategy.
2. One extra backup drive - a hotplugged sata backup that is actually
foms the redundant half of a raid 1 array. You plug it in and let it
sync up with the on-board disk then unplug and put it on the bookshelf
safe away from lightning strikes and surges. While it is unplugged
you are running the computer as a degraded raid-1 array.
The advantage of the raid syncing process is that the drives will
slowly come into sync but you still have processing power to keep
working during the 'backup'. Providing you wait until the background
raid re-syncing catches up you will have a bootable snapshot of the
entire system at the point you decided to unplug. Very useful. Just
plug it into your second hardware setup and go. You don't even have
to trouble-shoot a hardware failure until you have time. Also if you
corrupt your installation somehow and want to revert to the last
system-snapshot you just sync the array in reverse (ie boot off the
backup and then hotplug and sync the onboard disk.).
I would really like this all to work without any manual intervention
but don't quite have the knowledge. Also hotplugging is very new in
the linux kernel. The idea would be just hotplug the drives in and
out of the raid 1 array without needing to issue commands - the newly
plugged drive would always sync to the other drive.
Graham E.
Boaz
@a75g2000cwd.googlegroups.com:
>
> 1. 2 machines with identical hardware - thus a supply of swaps for
> all parts. The second machine to be used by my wife unless needed in
> my backup strategy.
>
> 2. One extra backup drive - a hotplugged sata backup that is actually
> foms the redundant half of a raid 1 array. You plug it in and let it
> sync up with the on-board disk then unplug and put it on the bookshelf
> safe away from lightning strikes and surges. While it is unplugged
> you are running the computer as a degraded raid-1 array.
>
>
I've just been playing with something similar on a Proliant box so taht
I have a backup of the system when I do experimentation and something
breaks. I want to be able to just boot from the image on another drive.
The problem I'm having is that if I do a binary copy (using 'dd') then
the system sometimes finds and mounts the partition of the backup drive
instead of the boot drive.
I can't solve this by constantly removing and replacing the backup hard
drive as this entails shutting down before and after each backup.
(Doing otherwise might break the backup somehow and replacing the drive
seems to do the same.) And having to do so would be som cumbersome that
I will end up not doing it regularly.
This is a problem on a somewhat older SCSI RAID setup, but it might be
the same problem you'll experience with SATA hotplug setup.
You'd better test it out first.
Boaz
Alan Adams
"gray" <gr...@karricountry.com.au> wrote:
> I have some backup ideas related to using raid arrays. Not sure if
> there is anyway to create this effect with commands such as rsync.
> Maybe.
>
<snip>
> 2. One extra backup drive - a hotplugged sata backup that is actually
> foms the redundant half of a raid 1 array. You plug it in and let it
> sync up with the on-board disk then unplug and put it on the bookshelf
> safe away from lightning strikes and surges. While it is unplugged
> you are running the computer as a degraded raid-1 array.
>
> The advantage of the raid syncing process is that the drives will
> slowly come into sync but you still have processing power to keep
> working during the 'backup'. Providing you wait until the background
> raid re-syncing catches up you will have a bootable snapshot of the
> entire system at the point you decided to unplug. Very useful. Just
Have you considered / is it possible to / have a 3-disk raid-1 set,
and remove the member which you store as backup, leaving a fully
redundant running system?
--
Alan Adams, from Northamptonshire
alan....@orchard-way.freeserve.co.uk
http://www.nckc.org.uk/
gray
> the system sometimes finds and mounts the partition of the backup drive
> instead of the boot drive.
you bios should allow you to select which device to boot off but - if
this is a problem there should be another solution...
>
> I can't solve this by constantly removing and replacing the backup hard
> drive as this entails shutting down before and after each backup.
> (Doing otherwise might break the backup somehow and replacing the drive
> seems to do the same.) And having to do so would be som cumbersome that
> I will end up not doing it regularly.
If the system insists on booting off your backup drive it doesn't
really matter - as long as the fstab entries point to the right drive
then the right partitions will be mounted. Your problem is probably
those new fangled volume labels like ROOT, HOME etc in your fstab
device line instead of the old fashioned /dev/hda2, /dev/hda4 and the
like. I have already run into problems with this and reverted the
fstab file to the old fashioned device pointers.
interested in hearing any other details of your system. I'm also
interested in knowing if rsync is really capable of creating this sort
of functionality over a network and if so how to implement it.
Graham
>
> This is a problem on a somewhat older SCSI RAID setup, but it might be
> the same problem you'll experience with SATA hotplug setup.
>
> You'd better test it out first.
heh heh. Just one more 400MB drive and I'm ready to blast off...
>
> Boaz
gray
>and remove the member which you store as backup, leaving a fully
>redundant running system?
Having a backup drive on the shelf and connecting it into the array
once a day is enough redundancy for me. I don't regularly have disks
go down and when they do it would be no trouble to restore the array
from the backup with a new drive. The only advantage I can see with
your set up is protection against a simultaneous failure of the
unplugged disk and the plugged in disk. Unlikely. And that always-on
redundancy will cost cpu cycles (we are talking about software RAID
here).
Graham