RAID Setup with Offline Spare
Douglas Mayne
knows about the mdadm options, --add vs. --re-add. Let's say a partition
on a physical device is setup as RAID 1 mirror, then is removed from the
array- it will actually be "placed in a drawer" for a few days. Then, at
a later time, it will be rejoined to the array. Can I use the "re-add"
option to do a quick block level update of the device? Or if too much
time has passed, do I have to use the "add" option which rewrites the
entire device?
The re-add option is potentially much faster. IIRC, an initial add on a
TB size partition will require about 8 hours on my hardware. I am
guessing that the re-add option performs updates based on some list of
changed blocks, correct? If the lists on each device do not "lap" over
one another, then too many changes have occurred, then the re-add will
fail, correct?
TIA
--
Douglas Mayne
TIY
marrgol
> I am starting to use RAID devices more and more. I am curious if anyone
> knows about the mdadm options, --add vs. --re-add. Let's say a partition
> on a physical device is setup as RAID 1 mirror, then is removed from the
> array- it will actually be "placed in a drawer" for a few days. Then, at
> a later time, it will be rejoined to the array. Can I use the "re-add"
> option to do a quick block level update of the device? Or if too much
> time has passed, do I have to use the "add" option which rewrites the
> entire device?
--re-add is only needed if your MD device does not have a superblock,
i.e. it is part of a legacy array created with --build option. If you
created yours with --create option it does not really matter if you
use --add instead of --re-add - the superblock of the device being
added will be examined by mdadm and the device re-added if possible.
> The re-add option is potentially much faster. IIRC, an initial add on a
> TB size partition will require about 8 hours on my hardware. I am
> guessing that the re-add option performs updates based on some list of
> changed blocks, correct?
Yes, but only if you created your array with write-intent bitmap
(or added it later) - it is not there by default. And when it's not
there full resynchronization has to be performed after any failure,
not only after removal and (re-)addition of a device.
--
mrg
Douglas Mayne
Thanks for that information about the bitmap. The manual page for mdadm
is a bit daunting. I think I need to do some more experiments on a non-
critical system to verify what happens when swapping devices in and out.
BTW, I found this book from 2002,
http://oreilly.com/catalog/9781565927308
Do you think it is worth reading, or is it dated at this point?
Thanks!
--
Douglas Mayne
David Brown
It's worth looking around at a few how-tos as well as the mdadm manual
page. And try it out too - just remember to use spare disks, and to
make very small partitions so you don't have to wait for hours for
re-builds while testing.
> BTW, I found this book from 2002,
> http://oreilly.com/catalog/9781565927308
>
> Do you think it is worth reading, or is it dated at this point?
>
Around ten years old is a /long/ time in this world! Even though many
parts will still be relevant, md raid has come far since then, and the
tools used have changed completely (I guess that in 2002 the standard
tools were "raidtools" - now it is "mdadm" together with the kernel proc
interface).
A good source of information and advice is the
linux...@vger.kernel.org mailing list. If you don't want to join the
mailing list itself, you can access it through the gmane gateway at:
news://news.gmane.org/gmane.linux.raid
mvh.,
David
marrgol
>>> The re-add option is potentially much faster. IIRC, an initial add on a
>>> TB size partition will require about 8 hours on my hardware. I am
>>> guessing that the re-add option performs updates based on some list of
>>> changed blocks, correct?
>>
>> Yes, but only if you created your array with write-intent bitmap (or
>> added it later) - it is not there by default. And when it's not there
>> full resynchronization has to be performed after any failure, not only
>> after removal and (re-)addition of a device.
>>
> Thanks for that information about the bitmap. The manual page for mdadm
> is a bit daunting. I think I need to do some more experiments on a non-
> critical system to verify what happens when swapping devices in and out.
>
> BTW, I found this book from 2002,
> http://oreilly.com/catalog/9781565927308
>
> Do you think it is worth reading, or is it dated at this point?
I heaven't read it so I can't really tell but I'd say it's dated.
If all you've read is mdadm's man page, try also md(4)'s and
http://tldp.org/HOWTO/Software-RAID-HOWTO.html
--
mrg
Grant
No idea, but I think (mis)using RAID1 for a backup has its own dangers, just
doesn't sound like the right thing to do, to me.
I'd prefer an rsync backup to an external drive for the same scenario.
And you're the one talking about snapshots recently!
One doesn't need a backup of the OS, just user data.
OTOH read about and deploy RAID6, fun stuff, and with two redundant drives
and checksum syndromes seems to take care of that oft-complained about
double drive failure and dataloss on RAID5?
With RAID6 I found it easier to work with partitions on top of the RAID,
rather than triggering resyncs at the RAID level. Of course I'm a beginner
here, no idea if I've missed something.
Cheers,
Grant.
Douglas Mayne
> On Thu, 20 Jan 2011 15:20:55 +0000 (UTC), Douglas Mayne
> <no...@invalid.com> wrote:
>
>>I am starting to use RAID devices more and more. I am curious if anyone
>>knows about the mdadm options, --add vs. --re-add. Let's say a partition
>>on a physical device is setup as RAID 1 mirror, then is removed from the
>>array- it will actually be "placed in a drawer" for a few days. Then, at
>>a later time, it will be rejoined to the array. Can I use the "re-add"
>>option to do a quick block level update of the device? Or if too much
>>time has passed, do I have to use the "add" option which rewrites the
>>entire device?
>>
>>The re-add option is potentially much faster. IIRC, an initial add on a
>>TB size partition will require about 8 hours on my hardware. I am
>>guessing that the re-add option performs updates based on some list of
>>changed blocks, correct? If the lists on each device do not "lap" over
>>one another, then too many changes have occurred, then the re-add will
>>fail, correct?
>
> No idea, but I think (mis)using RAID1 for a backup has its own dangers,
> just doesn't sound like the right thing to do, to me.
>
I hear what you are saying that RAID is not a backup system. (RAID is not
a backup system because it doesn't have the ability to roll back to an
earlier point in time.) In my case, I am actually looking for the best
way to store many point-in-time snapshots. Up until now, I have used rsync
quite extensively to do that. I have used multiple physical devices
(external USB devices) because multiple devices are good for peace of
mind, and I plan to keep that concept. However, management has dictated a
new policy that appears to be more well suited to a block-level solution
(RAID), rather than the file level solution (rsync) that I had been
using. The new backup solution that I have been experimenting with is a
rolling point-in-time snapshot system that makes extensive use of hard
links. To make matters a bit worse, the snapshots are stored in an
encrypted container. Those two facts (hardlinks, encryption) make
utilizing RAID 1 with multiple devices very attractive. I am in the early
testing phase with RAID's "write-intent" bitmap. It seems ideally suited
to give it a rsync-like incremental update ability, without having to
read/rewrite the entire container (currently 1+ TB). A benefit with the
RAID 1 solution appears to be that data will be managed at the block
level. That's good too because the data need not even be decrypted befire
being "resynced." Also, resync operations with RAID generally has very
good performance, IME. YMMV.
>
> I'd prefer an rsync backup to an external drive for the same scenario.
>
> And you're the one talking about snapshots recently!
>
Right, I wrote this:
http://groups.google.com/group/alt.os.linux.slackware/msg/d6c5b053c3ac38c4
>
> One doesn't need a backup of the OS, just user data.
>
> OTOH read about and deploy RAID6, fun stuff, and with two redundant
> drives and checksum syndromes seems to take care of that oft-complained
> about double drive failure and dataloss on RAID5?
>
> With RAID6 I found it easier to work with partitions on top of the RAID,
> rather than triggering resyncs at the RAID level. Of course I'm a
> beginner here, no idea if I've missed something.
>
> Cheers,
> Grant.
>
Note: comments inline.
This is going somewhat off-topic, but here are some tentative details of
the backup method that I plan to start using to make point in time
backups. This method makes heavy use of hardlinks and that is a very good
thing. Now that Moore's Law has delivered disk capacities of >1 TB, even
the use of compression in backups is much less important, but as
consolation, hardlinks offer a very good form of "compression" ;)
I may write more documentation later for this simple backup system that
uses mirrored snapshots. First, consider the following key command
sequences (and adjust as necessary or to suit your taste):
$ SRC=/mnt/some_data
$ PIT_DATE_1=/mnt/tmp/snapshot.2011-01-14
$ PIT_DATE_2=/mnt/tmp/snapshot.2011-01-21
$ mkdir $PIT_DATE_2
$ cp -anl ${PIT_DATE_1}/. ${PIT_DATE_2}
$ rsync -lptrv --dry-run --delete ${SRC} ${PIT_DATE_2}
Removing "dry run" option will actually perform the operation, and will
give two point in time snapshots. Thes snapshots will have maximum space
savings due to files in common being hardlinked. The next backup can
simply "roll forward" in time using the same incremental procedure. Old
backups can be "expired" off of the "old" end as needed or required by
simply removing the directory in its entirety. Of course, more variations
of this technique are possible and probably desirable because "one size"
does not "fit all" in the backup realm.
As mentioned above, RAID only enters into the picture because I need
multiple devices for offsite/offline storage.
--
Douglas Mayne
David Brown
As I was reading through your post and writing the comments below, it
has occurred to me that I'm a bit mixed up about what you are trying to
do, and how you are trying to do it. I'm not sure if this is my
misunderstanding of what you wrote, or that you were not clear in
describing your problem and suggested solution. So it might turn out
that the comments and recommendations I made below don't actually apply
to you. But since I believe them to be sound general advice, I'm
leaving them as is. When you have read them and replied, I expect I'll
be a bit clearer about what you are doing, and perhaps be able to make
other useful comments.
On 22/01/11 21:26, Douglas Mayne wrote:
> On Sun, 23 Jan 2011 04:59:47 +1100, Grant wrote:
>
>> On Thu, 20 Jan 2011 15:20:55 +0000 (UTC), Douglas Mayne
>> <no...@invalid.com> wrote:
>>
>>> I am starting to use RAID devices more and more. I am curious if anyone
>>> knows about the mdadm options, --add vs. --re-add. Let's say a partition
>>> on a physical device is setup as RAID 1 mirror, then is removed from the
>>> array- it will actually be "placed in a drawer" for a few days. Then, at
>>> a later time, it will be rejoined to the array. Can I use the "re-add"
>>> option to do a quick block level update of the device? Or if too much
>>> time has passed, do I have to use the "add" option which rewrites the
>>> entire device?
>>>
>>> The re-add option is potentially much faster. IIRC, an initial add on a
>>> TB size partition will require about 8 hours on my hardware. I am
>>> guessing that the re-add option performs updates based on some list of
>>> changed blocks, correct? If the lists on each device do not "lap" over
>>> one another, then too many changes have occurred, then the re-add will
>>> fail, correct?
>>
>> No idea, but I think (mis)using RAID1 for a backup has its own dangers,
>> just doesn't sound like the right thing to do, to me.
>>
> I hear what you are saying that RAID is not a backup system. (RAID is not
> a backup system because it doesn't have the ability to roll back to an
> earlier point in time.)
No - that is totally, and dangerously, wrong. Raid is not a backup
system because it has /nothing/ to do with backup. Raid is about
minimizing down-time in the event of hard disk failures. It has a
side-benefit of avoiding the loss of data that has changed between
backup runs, but that's a minor effect.
There are many ways to lose data - hard disk failure is only one
situation. It is a low probability failure, though generally you will
lose all the data on the disk at the time. But you are far more likely
to lose data due to human error - deleting the wrong files, or
accidentally overwriting files.
Backup is about preserving your data against /any/ failure. That
includes fire or theft in your main location - backups should always be
off-site.
So if you are thinking about raid1 to prevent data loss, you would be
/much/ better off using that spare disk as an independent disk and
making copies of your data onto it.
> In my case, I am actually looking for the best
> way to store many point-in-time snapshots. Up until now, I have used rsync
> quite extensively to do that. I have used multiple physical devices
> (external USB devices) because multiple devices are good for peace of
> mind, and I plan to keep that concept. However, management has dictated a
> new policy that appears to be more well suited to a block-level solution
> (RAID), rather than the file level solution (rsync) that I had been
> using. The new backup solution that I have been experimenting with is a
> rolling point-in-time snapshot system that makes extensive use of hard
> links.
I'm slightly confused here - when I think of a point-in-time snapshot
system that makes extensive use of hard links, rsync is the tool of
choice. Look at something like rsnapshot as an example, though it is
not hard to make your own scripts that use rsync directly.
RAID can't do point-in-time snapshots or backups. It can do duplication
of block data, but that has nothing to do with backup. When someone
deletes the wrong file, your RAID system will happily mirror that
deletion to all copies. With a snapshot system, you would go back to a
previous snapshot to retrieve the old data.
There /are/ ways to do block-level snapshots, with some advantages and
disadvantages. A simple way is to use LVM and snapshot logical
partitions. This makes future use of the disk steadily more
inefficient, especially if you have more than one snapshot, but is quite
useful to make a temporary snapshot which you can then back up. More
advanced file systems, such as zfs and btrfs, support efficient
snapshots. But zfs can't work well on Linux, and btrfs is not yet
mainstream. And even then, your snapshots are on the same system as
your main data - backups should always be separate (failure of your
server should not compromise your backup).
File-level backups with rsync are much more efficient to make, much more
efficient to store, and /very/ much more efficient for searching and
restoring individual files or directories, or for getting out additional
information (such as "when was this file changed?"). Block level
backups are a bit faster for rebuilding a complete system - but that is
a rare event.
I am not saying you shouldn't use raid - raid means that if you lose a
hard disk, you will not have much down-time, and is thus definitely a
good thing for most servers. But you need to educate your management
that raid is not backup.
> To make matters a bit worse, the snapshots are stored in an
> encrypted container.
I would /never/ recommend storing backups in encrypted format if it is
possible to avoid it. Use physical security (put them in a locked box
or cupboard). Encryption makes it so much harder to test or to restore
data, and introduces a range of new potential problems (forgotten
password, incorrect password, errors when saving, old employees
remembering the password, etc.). Encryption is a good idea if it is
better that no one has the data than the wrong person seeing the data.
Douglas Mayne
> As I was reading through your post and writing the comments below, it
> has occurred to me that I'm a bit mixed up about what you are trying to
> do, and how you are trying to do it. I'm not sure if this is my
> misunderstanding of what you wrote, or that you were not clear in
> describing your problem and suggested solution. So it might turn out
> that the comments and recommendations I made below don't actually apply
> to you. But since I believe them to be sound general advice, I'm
> leaving them as is. When you have read them and replied, I expect I'll
> be a bit clearer about what you are doing, and perhaps be able to make
> other useful comments.
>
Thanks for your input. I have created this webpage to more fully document
the complete process that I am in the process of implementing:
http://www.xmission.com/~ddmayne/misc/raid1.html
My intent is to use RAID 1 to be an efficient disk cloner.
Maybe so, but can RAID be used a simple disk cloner?
>
> There are many ways to lose data - hard disk failure is only one
> situation. It is a low probability failure, though generally you will
> lose all the data on the disk at the time. But you are far more likely
> to lose data due to human error - deleting the wrong files, or
> accidentally overwriting files.
>
Exactly right. There is a need for many "point in time" backups. That
provides the ability to rollback a complete system or individual files to
some state in the past.
>
> Backup is about preserving your data against /any/ failure. That
> includes fire or theft in your main location - backups should always be
> off-site.
>
Ok.
>
> So if you are thinking about raid1 to prevent data loss, you would be
> /much/ better off using that spare disk as an independent disk and
> making copies of your data onto it.
>
Ok. I am using RAID 1 as to create disk clones.
AIUI, RAID's write-intent bitmap can be used to mark the changed blocks.
I don't know for sure, but I am in the process of testing it (as
described in the linked documentation.)
>
> File-level backups with rsync are much more efficient to make, much more
> efficient to store, and /very/ much more efficient for searching and
> restoring individual files or directories, or for getting out additional
> information (such as "when was this file changed?"). Block level
> backups are a bit faster for rebuilding a complete system - but that is
> a rare event.
>
> I am not saying you shouldn't use raid - raid means that if you lose a
> hard disk, you will not have much down-time, and is thus definitely a
> good thing for most servers. But you need to educate your management
> that raid is not backup.
>
>
>> To make matters a bit worse, the snapshots are stored in an encrypted
>> container.
>
> I would /never/ recommend storing backups in encrypted format if it is
> possible to avoid it.
This data set requires encryption as a business requirement.
> Use physical security (put them in a locked box
> or cupboard).
That's a good idea, as far as it physical security can be maintained.
>Encryption makes it so much harder to test or to restore
> data, and introduces a range of new potential problems (forgotten
> password, incorrect password, errors when saving, old employees
> remembering the password, etc.). Encryption is a good idea if it is
> better that no one has the data than the wrong person seeing the data.
Ok, no disagreement. Still, it is required.
>
<snip>
>> As mentioned above, RAID only enters into the picture because I need
>> multiple devices for offsite/offline storage.
>>
Note: comments inline.
--
Douglas Mayne
Douglas Mayne
I have created this webpage to document my initial experiments with
multiple RAID 1 mirrors:
http://www.xmission.com/~ddmayne/misc/raid1.html
--
Douglas Mayne
David Brown
> On Sun, 23 Jan 2011 17:45:00 +0100, David Brown wrote:
>
>> As I was reading through your post and writing the comments below, it
>> has occurred to me that I'm a bit mixed up about what you are trying to
>> do, and how you are trying to do it. I'm not sure if this is my
>> misunderstanding of what you wrote, or that you were not clear in
>> describing your problem and suggested solution. So it might turn out
>> that the comments and recommendations I made below don't actually apply
>> to you. But since I believe them to be sound general advice, I'm
>> leaving them as is. When you have read them and replied, I expect I'll
>> be a bit clearer about what you are doing, and perhaps be able to make
>> other useful comments.
>>
> Thanks for your input. I have created this webpage to more fully document
> the complete process that I am in the process of implementing:
> http://www.xmission.com/~ddmayne/misc/raid1.html
>
> My intent is to use RAID 1 to be an efficient disk cloner.
I'm snipping the rest of the post, because we agree on the main points,
and I find the order of the parts confusing.
First off, yes - RAID 1 can be a reasonably efficient disk cloner using
the write-intent bitmap. It will get a lot less efficient if you try to
clone it to two disks at a time.
However, I still fail to see /why/ you want to do disk cloning at all.
Rsync gives you all you need, as far as I can see, and it does so more
efficiently. You already have the basis of a rolling snapshot system.
It could be made slightly more efficient - with reasonably modern
versions of rsync, use the "--dest-link" (or is it "--link-dest"? The
man page will know) to make your hard links rather than having a
separate "cp -al" pass.
Where rsync scores particularly highly is when you are doing the backup
to a different machine - which you should be, except perhaps for an
initial run to save time. Backup to another disk on the same machine is
a poor choice - any disaster hitting the main machine (fire, theft,
power-supply chaos, kernel bugs, drunken user, etc.) could easily take
out the backup too.
Over the network, rsync will only transfer the files that have changed,
and for larger files it will only transfer the changes to the file. It
can also do compression on the fly. Overall, it will probably involve
significantly less network traffic than a block copy with raid.
I am still not totally sure how you plan to put the parts together, but
I think roughly you are thinking that your server has a main drive, and
a backup drive that is basically a third of a three-way raid1. When you
want to do a backup, you plug in two disks and add them to the raid1.
Then you do your rsync from the main drive to the backup trio, remove
the two external disks and put them somewhere else. Is that about right?
The time saved by doing this, rather than three rsyncs (one to the
"master" backup disk, then two from there to the "slave" backup disks),
is going to be small. And it will be heavily outweighed by the
inconvenience of a system that involves manually adding and removing the
disks.
I would recommend just putting the two other disks on two different
computers (preferably at different sites), and doing rsync over the
network. If the PHB's want encryption, use encryption on the backup
drives, and use a VPN or SSH tunnel for the rsync traffic. The system
will be safer, give better protection of your data, and run with less
manual intervention.
When setting up a system like that for use at my company, I considered
using RAID1 with a remote drive - it's easy to put the second copy on an
iSCSI drive mapped over a network, and mdadm lets you make such a remote
device be basically write-only so that it is not a bottleneck. The idea
was the remote rsync backup would come from the remote raid1 copy (or
more precisely, from an LVM snapshot of that copy). But after more
thought, such an arrangement would be a good deal slower than a simple
rsync over the network (though it would be fun!).
mvh.,
David
Grant
>On Sun, 23 Jan 2011 17:45:00 +0100, David Brown wrote:
>
>> As I was reading through your post and writing the comments below, it
>> has occurred to me that I'm a bit mixed up about what you are trying to
>> do, and how you are trying to do it. I'm not sure if this is my
>> misunderstanding of what you wrote, or that you were not clear in
>> describing your problem and suggested solution. So it might turn out
>> that the comments and recommendations I made below don't actually apply
>> to you. But since I believe them to be sound general advice, I'm
>> leaving them as is. When you have read them and replied, I expect I'll
>> be a bit clearer about what you are doing, and perhaps be able to make
>> other useful comments.
>>
>Thanks for your input. I have created this webpage to more fully document
>the complete process that I am in the process of implementing:
>http://www.xmission.com/~ddmayne/misc/raid1.html
>
>My intent is to use RAID 1 to be an efficient disk cloner.
Problem there is that RAID1 needs a boot block or some info on disk to
identify the RAID, therefore not such a good cloning tool? One is forced
to use degraded RAID1 in your scenario? Smells 'off' to me, but I've not
looked at your site yet -- my mind is elsewhere.
Grant.
Stan Bischof
disk enclosure, a stack of disk drives, and a simple
rsync script. Mine's on a remote server but you could
do the same on your source machine ( hot swap
disks are readily available ). Or if you really care
about your data get a proper NAS which has snapshot
capability ( one I access has daily snapshots so end user
can put themselves back to anything in the last month
or so ).
In any case, this does _not_ sound like a good application
of a RAID.
Stan
PS- as to whethere RAID can be set up to do what you want, most
likely if you really want- but then why use a hammer to drive
in a screw?