backups using rsync

[Ronald F. Guilmette]

As a result of this past Black Friday weekend, I now enjoy a true abundance
of disk space, for the first time in my life.

I wanna make a full backup, on a weekly basis, of my main system's shiny
new 1TB drive onto another 1TB drive that I also picked up cheap back on
Black Friday.

I've been planning to set this up for some long time now, but I've only
gotten 'round to working on it now.

Now, unfortunately, I have just been bitten by the evil... and apparently
widely known (except to me)... ``You can't use dump(8) to dump a journaled
filesystem with soft updates'' bug-a-boo.

Sigh. The best laid plans of mice and men...

I _had_ planned on using dump/restore and making backups from live mounted
filesystems while the system was running. But I really don't want to have
to take the system down to single-user mode every week for a few hours while
I'm making my disk-to-disk backup. So now I'm looking at doing the backups
using rsync.

I see that rsync can nowadays properly cope with all sorts of oddities,
like fer instance device files, hard-linked files, ACLs, file attributes,
and all sorts of other unusual but important filesystem thingies. That's
good news, but I still have to ask the obvious question:

If I use all of the following rsync options... -a,-H,-A, -X, and -S ....
when trying to make my backups, and if I do whatever additional fiddling
is necessary to insure that I separately copy over the MBR and boot loader
also to my backup drive, then is there any reason that, in the event of
a sudden meteor shower that takes out my primary disk drive while leaving
my backup drive intact, I can't just unplug my old primary drive, plug in
my (rsync-created) backup drive, reboot and be back in the sadddle again,
almost immediately, and with -zero- problems?


Regards,
rfg


P.S. My apologies if I've already asked this exact same question here
before. I'm getting a sense of deja vu... or else a feeling that I am
often running around in circles, chasing my own tail.

P.P.S. Before anyone asks, no I really _do not_ want to just use RAID
as my one and only backup strategy. RAID is swell if your only problem
is hardware failures. As far as I know however it will not save your
bacon in the event of a fumble fingers "rm -rf *" moment. Only frequent
and routine actual backups can do that.
_______________________________________________
freebsd-...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questi...@freebsd.org"

[Polytropon]

On Mon, 04 Mar 2013 03:35:30 -0800, Ronald F. Guilmette wrote:
> Now, unfortunately, I have just been bitten by the evil... and apparently
> widely known (except to me)... ``You can't use dump(8) to dump a journaled
> filesystem with soft updates'' bug-a-boo.

There are other tools you can use, for example tar or cpdup
or rsync, as you've mentioned in the subject.

> I _had_ planned on using dump/restore and making backups from live mounted
> filesystems while the system was running. But I really don't want to have
> to take the system down to single-user mode every week for a few hours while
> I'm making my disk-to-disk backup. So now I'm looking at doing the backups
> using rsync.

The same problems that apply when dumping live systems can
bite you using rsync, but support for this "on file system
level" seems to be better in rsync than what dump does "on
block level".

> If I use all of the following rsync options... -a,-H,-A, -X, and -S ....
> when trying to make my backups, and if I do whatever additional fiddling
> is necessary to insure that I separately copy over the MBR and boot loader
> also to my backup drive, then is there any reason that, in the event of
> a sudden meteor shower that takes out my primary disk drive while leaving
> my backup drive intact, I can't just unplug my old primary drive, plug in
> my (rsync-created) backup drive, reboot and be back in the sadddle again,
> almost immediately, and with -zero- problems?

You would have to make sure _many_ things are consistent
on the backup disk.

Regarding terminology, that would make the disk a failover
disk, even if the act of making it the actual "work disk"
is something you do manually.

The disk would need to have an initialized file system and
a working boot mechanism, both things rsync does not deal
with, if I remember correctly. But as soon as you have
initialized the disk for the first time and made sure (by
testing your first result of a rsync run), it should work
with any subsequent change of data you transfer to that
disk.

> P.P.S. Before anyone asks, no I really _do not_ want to just use RAID
> as my one and only backup strategy.

RAID _is_ **NO** backup. It's for dedundancy and performance.
If something is erased or corrupted, it's on all disks. And
all the disks permanently run. A backup disk only runs twice:
when backing something up, or when restoring. In your case,
restoring means that the disk is put into operation in its
role as a failover disk.

> RAID is swell if your only problem
> is hardware failures.

Still hardware failures can corrupt data on all participating
disks.

> As far as I know however it will not save your
> bacon in the event of a fumble fingers "rm -rf *" moment. Only frequent
> and routine actual backups can do that.

Correct. It's important to learn that lesson _before_ it is
actually needed. :-)




--
Polytropon
Magdeburg, Germany
Happy FreeBSD user since 4.0
Andra moi ennepe, Mousa, ...

[Warren Block]

On Mon, 4 Mar 2013, Ronald F. Guilmette wrote:

> Now, unfortunately, I have just been bitten by the evil... and apparently
> widely known (except to me)... ``You can't use dump(8) to dump a journaled
> filesystem with soft updates'' bug-a-boo.

Until SUJ has been deemed 100%, I avoid it and suggest others do also.
It can be disabled on an existing filesystem from single user mode.

> If I use all of the following rsync options... -a,-H,-A, -X, and -S ....
> when trying to make my backups, and if I do whatever additional fiddling
> is necessary to insure that I separately copy over the MBR and boot loader
> also to my backup drive, then is there any reason that, in the event of
> a sudden meteor shower that takes out my primary disk drive while leaving
> my backup drive intact, I can't just unplug my old primary drive, plug in
> my (rsync-created) backup drive, reboot and be back in the sadddle again,
> almost immediately, and with -zero- problems?

It works. I use this to "slow mirror" SSDs to a hard disk, avoiding the
speed penalty of combining an SSD with a hard disk in RAID1.

Use the latest net/rsync port, and enable the FLAGS option. I use these
options, copying each filesystem individually:

-axHAXS --delete --fileflags --force-change

--delete removes files present on the copy that are not on the original.
Some people may want to leave those.

--exclude= is used on certain filesystems to skip directories that are
full of easily recreated data that changes often, like /usr/obj.

Yes, the partitions and bootcode must be set up beforehand. After that,
it works. Like any disk redundancy scheme, test it before an emergency.

> P.P.S. Before anyone asks, no I really _do not_ want to just use RAID

> as my one and only backup strategy. RAID is swell if your only problem
> is hardware failures. As far as I know however it will not save your

> bacon in the event of a fumble fingers "rm -rf *" moment. Only frequent
> and routine actual backups can do that.

Yes, RAID is not a backup. Another suggestion I've been making often:
use sysutils/rsnapshot to make an accessible history of files. The
archive go on another partition on the mirror drive, which likely has
more space than the original. rsnapshot uses rsync with hard links to
make an archive that lets you easily get to old versions of files that
have changed in the last few hours/days/weeks/months.

[Ronald F. Guilmette]

In message <20130304125634....@edvax.de>,
Polytropon <fre...@edvax.de> wrote:

>On Mon, 04 Mar 2013 03:35:30 -0800, Ronald F. Guilmette wrote:
>> Now, unfortunately, I have just been bitten by the evil... and apparently
>> widely known (except to me)... ``You can't use dump(8) to dump a journaled
>> filesystem with soft updates'' bug-a-boo.
>

>There are other tools you can use, for example tar or cpdup
>or rsync, as you've mentioned in the subject.

tar I already knew about, but I think you will agree that it has lots of
limitations that make it entirely inappropriate for mirroring an entire
system.

This cpdup thing is entirely new to me. Thanks for mentioning it! I really
never heard of it before, but I just now installed it from ports, and I'm
perusing the man page. It looks very promising. Too bad it doesn't
properly handle sparse files, but oh well. That's just a very minor nit.
(Does it properly handle everything else that rsync claims to be able to
properly handle, e.g. ACLs, file attributes, etc., etc.?)

>The same problems that apply when dumping live systems can
>bite you using rsync,

What problems are we talking about, in particular?

I am guessing that if I use rsync, then I *won't* encounter this rather
annoying issue/problem relating to UFS filesystems that have both soft
updates and journaling enabled, correct?

>but support for this "on file system
>level" seems to be better in rsync than what dump does "on
>block level".

What exactly did you mean by "this" ?

>> If I use all of the following rsync options... -a,-H,-A, -X, and -S ....
>> when trying to make my backups, and if I do whatever additional fiddling
>> is necessary to insure that I separately copy over the MBR and boot loader
>> also to my backup drive, then is there any reason that, in the event of
>> a sudden meteor shower that takes out my primary disk drive while leaving
>> my backup drive intact, I can't just unplug my old primary drive, plug in
>> my (rsync-created) backup drive, reboot and be back in the sadddle again,
>> almost immediately, and with -zero- problems?
>

>You would have to make sure _many_ things are consistent
>on the backup disk.

Well, this is what I am getting at. This is/was the whole point of my post
and my question. I want to know: What is that set of things, exactly?

>Regarding terminology, that would make the disk a failover disk

OK. Thank you. I will henceforth use that terminology.

>The disk would need to have an initialized file system and
>a working boot mechanism, both things rsync does not deal with

Check and check. I implicitly understood the former, and I explicitly
mentioned the latter in my original post in this thread.

But is there anything else, other than those two things (which, just as
you say, are both clearly outside of the scope of what rsync does)?
Anything else I need to do or worry about in order to be able to use
rsync to create & maintain a full-blown fully-working system failover
drive?

If so, I'd much rather learn about it now... you know... as opposed
to learning about it if and when I actually have to _use_ my failover
drive.


Regards,
rfg

[Ronald F. Guilmette]

In message <alpine.BSF.2.00.1...@wonkity.com>,

Warren Block <wbl...@wonkity.com> wrote:

>Until SUJ has been deemed 100%, I avoid it and suggest others do also.
>It can be disabled on an existing filesystem from single user mode.

hehe

Silly me! What do *I* know? I just go about my business and try not to
create too much trouble for myself. To be honest and truthful I have to
say that this journaling stuff entirely snuck up on me. I confess...
I wasn't paying attention (to the world of FreeBSD innovations) and thus,
when I moved myself recently to 9.x (from 8.3) I did so without even
having been aware that the new filesystems that I was creating during
my clean/fresh install of 9.1 had journaling turned on by default.
(As the saying goes, I didn't get the memo.) Not that I mind, really.
It sounds like a great concept and a great feature and I was happy to
have it right up until the moment that "dump -L" told me to go pound
sand. :-(

So, um, I was reading about this last night, but I was sleepy and my eyes
glazed over... Please remind me, what is the exact procedire for turning
off the journaling? I boot to single user mode (from a live cd?) and
then what? Is it tunefs with some special option?

>> If I use all of the following rsync options... -a,-H,-A, -X, and -S ....
>> when trying to make my backups, and if I do whatever additional fiddling
>> is necessary to insure that I separately copy over the MBR and boot loader
>> also to my backup drive, then is there any reason that, in the event of
>> a sudden meteor shower that takes out my primary disk drive while leaving
>> my backup drive intact, I can't just unplug my old primary drive, plug in
>> my (rsync-created) backup drive, reboot and be back in the sadddle again,
>> almost immediately, and with -zero- problems?
>

>It works. I use this to "slow mirror" SSDs to a hard disk, avoiding the
>speed penalty of combining an SSD with a hard disk in RAID1.

Great! Thanks Warren.

>Use the latest net/rsync port, and enable the FLAGS option. I use these
>options, copying each filesystem individually:
>
>-axHAXS --delete --fileflags --force-change

Hummm... I guess that I have some non-current rsync installed. In the man
page I have there is no mention of any "--force-change" option. What does
it do?

>Yes, the partitions and bootcode must be set up beforehand. After that,
>it works.

Good to know. Thanks again Warren.

>Like any disk redundancy scheme, test it before an emergency.

Naw. I like to live dangerously. :-)

[CyberLeo Kitsana]

On 03/04/2013 05:35 AM, Ronald F. Guilmette wrote:
> As a result of this past Black Friday weekend, I now enjoy a true abundance
> of disk space, for the first time in my life.
>
> I wanna make a full backup, on a weekly basis, of my main system's shiny
> new 1TB drive onto another 1TB drive that I also picked up cheap back on
> Black Friday.
>
> I've been planning to set this up for some long time now, but I've only
> gotten 'round to working on it now.
>

> Now, unfortunately, I have just been bitten by the evil... and apparently
> widely known (except to me)... ``You can't use dump(8) to dump a journaled
> filesystem with soft updates'' bug-a-boo.

You can use dump(8) to dump a SU-journaled filesystem; you just cannot
create a snapshot. This implies that dump(8) will be run against the
live and possibly changing filesystem, which can lead to issues with the
consistency of the contents of files thus dumped; but not necessarily
with the consistency of the dump itself. Any tool that backs up a live
filesystem, such as rsync or tar, will have these issues.

> Sigh. The best laid plans of mice and men...
>

> I _had_ planned on using dump/restore and making backups from live mounted
> filesystems while the system was running. But I really don't want to have
> to take the system down to single-user mode every week for a few hours while
> I'm making my disk-to-disk backup. So now I'm looking at doing the backups
> using rsync.

I've used rsync to back up Linux and FreeBSD machines daily for years,
and I've never had a problem with the backups nor subsequent
restorations. Especially for restorations of the laptop that ate SSDs.

Having a decent snapshot capability on the backup target filesystem can
help a lot if you want to maintain multiple sparse backup revisions;
otherwise, you're stuck using creative scripting around rsync's
--link-dest option.

> I see that rsync can nowadays properly cope with all sorts of oddities,
> like fer instance device files, hard-linked files, ACLs, file attributes,
> and all sorts of other unusual but important filesystem thingies. That's
> good news, but I still have to ask the obvious question:
>

> If I use all of the following rsync options... -a,-H,-A, -X, and -S ....
> when trying to make my backups, and if I do whatever additional fiddling
> is necessary to insure that I separately copy over the MBR and boot loader
> also to my backup drive, then is there any reason that, in the event of
> a sudden meteor shower that takes out my primary disk drive while leaving
> my backup drive intact, I can't just unplug my old primary drive, plug in
> my (rsync-created) backup drive, reboot and be back in the sadddle again,
> almost immediately, and with -zero- problems?

There will /always/ be problems. The best you can do is become familiar
with the tools and procedures so you can tackle them when they happen.

My suggestion for something that you can use as a warm standby is to
create it as a warm standby: go through the entire installation
procedure again for the backup drive, and then use rsync or suchlike to
periodically synchronize the second filesystem with the first. When you
update the boot code on one, do so on the other.

Be extremely careful if you decide to do this with both disks attached
to the same machine: if you use geom labels (gpt, ufs, glabel, et alia)
or dynamically numbered storage devices, you can easily run into a
situation where a reboot with both devices attached suddenly starts
using your backup instead without you realizing it, or flips back and forth.

> P.S. My apologies if I've already asked this exact same question here
> before. I'm getting a sense of deja vu... or else a feeling that I am
> often running around in circles, chasing my own tail.
>

> P.P.S. Before anyone asks, no I really _do not_ want to just use RAID
> as my one and only backup strategy. RAID is swell if your only problem
> is hardware failures. As far as I know however it will not save your
> bacon in the event of a fumble fingers "rm -rf *" moment. Only frequent
> and routine actual backups can do that.

--
Fuzzy love,
-CyberLeo
Furry Peace! - http://www.fur.com/peace/

[Lawrence K. Chen, P.Eng.]

----- Original Message -----
> On Mon, 04 Mar 2013 03:35:30 -0800, Ronald F. Guilmette wrote:

> > Now, unfortunately, I have just been bitten by the evil... and
> > apparently
> > widely known (except to me)... ``You can't use dump(8) to dump a
> > journaled
> > filesystem with soft updates'' bug-a-boo.
>

> There are other tools you can use, for example tar or cpdup
> or rsync, as you've mentioned in the subject.
>
>

Or if you want to be ambitious you could install something like 'sysutils/backuppc' (where one of its methods is rsync, its what I use for all the systems I back up with it. - Windows, Linux, Mac OSX.)

And, then could get more than just the weekly rsync to it....though it could probably be made to only do fulls every week. But, you could potentially then restore from an older full.

I do system fulls of my other systems to it...can't do a baremetal restore, but it can get me back up and running faster. IE: I recently had harddrive failures in a couple of FreeBSD systems. I did a fresh install and at first I restored /home and /usr/local (and some other dirs, like /var/db/pkg & /var/db/ports)...and then other dirs and files as I found things missing. Had to rebuild a handful of ports after that and then things were good.

The second system didn't go as well, because it had been silently corrupting things for a long time before....but I still did the same kind of restore at first, but ended up rebuilding all the ports to get things good again.

Not sure if losing the system disk, if I could recover from a local backuppc... but I have my old backuppc system, getting most of my current system (mainly omit the backuppc pool, think my backup storage requirements would grow exponentially if I didn't....my main backuppc pool is currently 6300G out of 7300G zpool.) But, I've suffered bit rot on the old backuppc pool in the past..when it was a RAID 1+0 array....probably worse now that its a 2.7TB volume without raid (the only volume on that system that isn't mirrored.) Though wonder if I want to try zfs on linux again, or replace it with FreeBSD.

I was faced with something like this on my Windows box....where eventually, I ended up writing off restoring from the local backup (a commercial time machine like product)...the mistake was using a windows fake raid5 external array as my backup drive. And, losing the system due to problems in the fake raid. I did briefly put together a CentOS live CD that could access the array, but the drives I copied the data to promptly failed on me shortly after I had broken the array and turned them into a raidz pool. Someday I need to get back to going through the disk image of the failed system drive and recover as much as possible from that. The box that was my Windows desktop is now my FreeBSD desktop....

Lawrence

[Warren Block]

On Mon, 4 Mar 2013, Ronald F. Guilmette wrote:

> So, um, I was reading about this last night, but I was sleepy and my eyes
> glazed over... Please remind me, what is the exact procedire for turning
> off the journaling? I boot to single user mode (from a live cd?) and
> then what? Is it tunefs with some special option?

Just boot in single user mode so all the filesystems are unmounted or
mounted readonly. Then use 'tunefs -j disable /dev/...'. It will also
mention the name of the journal file, which can be deleted.

>> Use the latest net/rsync port, and enable the FLAGS option. I use these
>> options, copying each filesystem individually:
>>
>> -axHAXS --delete --fileflags --force-change
>
> Hummm... I guess that I have some non-current rsync installed. In the man
> page I have there is no mention of any "--force-change" option. What does
> it do?

"affect user/system immutable files/dirs". Probably only included in
the man page when the port is built with the FLAGS option set.

An additional note: the script that runs my rsync backup also modifies
the mirrored /etc/fstab to use the appropriate labels for the backup
filesystems.

[Polytropon]

On Mon, 04 Mar 2013 12:19:09 -0800, Ronald F. Guilmette wrote:
>
> In message <20130304125634....@edvax.de>,
> Polytropon <fre...@edvax.de> wrote:
>

> >On Mon, 04 Mar 2013 03:35:30 -0800, Ronald F. Guilmette wrote:
> >> Now, unfortunately, I have just been bitten by the evil... and apparently
> >> widely known (except to me)... ``You can't use dump(8) to dump a journaled
> >> filesystem with soft updates'' bug-a-boo.
> >
> >There are other tools you can use, for example tar or cpdup
> >or rsync, as you've mentioned in the subject.
>

> tar I already knew about, but I think you will agree that it has lots of
> limitations that make it entirely inappropriate for mirroring an entire
> system.

That's true. If your purpose is "backup of data files",
tar is a good tool, especially for cross-platform use.
But if you need to deal with "exceptional" things like
extended permissions, ACL, sparse files and such, you
will quickly see its limits. On the other hand, it can
be used for multi-volume savesets, but this is not your
intention.

> This cpdup thing is entirely new to me. Thanks for mentioning it! I really
> never heard of it before, but I just now installed it from ports, and I'm
> perusing the man page.

It's a little bit comparable to rsync and can also do
things like "only add" (so you won't lose any files:
if they are removed in source, they will be kept in
backup). It also has limitations that rsync will not.

> It looks very promising. Too bad it doesn't
> properly handle sparse files, but oh well. That's just a very minor nit.
> (Does it properly handle everything else that rsync claims to be able to
> properly handle, e.g. ACLs, file attributes, etc., etc.?)

That's something you should check with an "example
dataset" you back up, restore, and compare. I've been
using it for "normal files" successfully.

> >The same problems that apply when dumping live systems can
> >bite you using rsync,
>
> What problems are we talking about, in particular?

The problems I'm refering to is the kind of _possible_
trouble you can get into when backing up files that
keep changing. The ability to make a snapshot prior
to starting the backup is a great help here (if you
don't have the chance to unmount the partitions to
backup). I can't imagine _how_ programs will react
if they start reading a file, prepare sufficient space
in some kind of TOC, then continue reading while the
file grows... or if a file is being read which is
removed during read... If you minimize the writing
activity to the (still) _live_ data you're dealing
with, that could be a benefit.

> I am guessing that if I use rsync, then I *won't* encounter this rather
> annoying issue/problem relating to UFS filesystems that have both soft
> updates and journaling enabled, correct?
>
> >but support for this "on file system
> >level" seems to be better in rsync than what dump does "on
> >block level".
>
> What exactly did you mean by "this" ?

As mentioned above: Unexpected and unpredictable results,
strange kinds of inconsistency, may they appear during
backup or later on restore.

> >> If I use all of the following rsync options... -a,-H,-A, -X, and -S ....
> >> when trying to make my backups, and if I do whatever additional fiddling
> >> is necessary to insure that I separately copy over the MBR and boot loader
> >> also to my backup drive, then is there any reason that, in the event of
> >> a sudden meteor shower that takes out my primary disk drive while leaving
> >> my backup drive intact, I can't just unplug my old primary drive, plug in
> >> my (rsync-created) backup drive, reboot and be back in the sadddle again,
> >> almost immediately, and with -zero- problems?
> >

> >You would have to make sure _many_ things are consistent
> >on the backup disk.
>
> Well, this is what I am getting at. This is/was the whole point of my post
> and my question. I want to know: What is that set of things, exactly?

The backup disk (or failover disk, as I said) needs to be
initialized properly prior to the first backup run: Make
sure it's bootable. Depending on how you handle identification
of the disk (by device name, by label or UFSID) and how
you're going to boot from it (by selecting the failover
disk in some post-BIOS/POST dialog or by swapping cables
or bays), you should check it actually starts booting.

> >Regarding terminology, that would make the disk a failover disk
>
> OK. Thank you. I will henceforth use that terminology.

Just a suggestion from how you described you will be
using the disk, which isn't what commonly (or mostly)
is expressed by the term "backup" (also cf. "archive"
which is something entirely different).

> >The disk would need to have an initialized file system and
> >a working boot mechanism, both things rsync does not deal with
>
> Check and check. I implicitly understood the former, and I explicitly
> mentioned the latter in my original post in this thread.
>
> But is there anything else, other than those two things (which, just as
> you say, are both clearly outside of the scope of what rsync does)?
> Anything else I need to do or worry about in order to be able to use
> rsync to create & maintain a full-blown fully-working system failover
> drive?

I don't think so, the provided summary seems to be complete
from my understanding.

> If so, I'd much rather learn about it now... you know... as opposed
> to learning about it if and when I actually have to _use_ my failover
> drive.

No. That's the case when you learn about forensic analysis
and disaster recovery. ;-)

--
Polytropon
Magdeburg, Germany
Happy FreeBSD user since 4.0
Andra moi ennepe, Mousa, ...

[Giorgos Keramidas]

On 2013-03-04 03:35, "Ronald F. Guilmette" <r...@tristatelogic.com> wrote:
> As a result of this past Black Friday weekend, I now enjoy a true
> abundance of disk space, for the first time in my life.
>
> I wanna make a full backup, on a weekly basis, of my main system's
> shiny new 1TB drive onto another 1TB drive that I also picked up cheap
> back on Black Friday.
>
> I've been planning to set this up for some long time now, but I've
> only gotten 'round to working on it now.
>

> Now, unfortunately, I have just been bitten by the evil... and
> apparently widely known (except to me)... ``You can't use dump(8) to
> dump a journaled filesystem with soft updates'' bug-a-boo.
>

> Sigh. The best laid plans of mice and men...
>
> I _had_ planned on using dump/restore and making backups from live mounted
> filesystems while the system was running. But I really don't want to have
> to take the system down to single-user mode every week for a few hours while
> I'm making my disk-to-disk backup. So now I'm looking at doing the backups
> using rsync.

Yes, this should be possible...

One thing that can bite you when using rsync to traverse & copy large
filesystems is that the filesystem may still be changing beneath rsync
*as it's doing* the copy.

If this is a UFS2 filesystem, it may be a good idea to snapshot the
filesystem, and then rsync-backup the snapshot instead.

[Matthias Petermann]

Hello,

Zitat von Giorgos Keramidas <kera...@ceid.upatras.gr>:

> If this is a UFS2 filesystem, it may be a good idea to snapshot the
> filesystem, and then rsync-backup the snapshot instead.

Last time I tried UFS2 snapshots I found out two serious limitations.
The first is it doesn't work when UFS Journaling is used. The second is
that taking a snapshop on a large filesystem can cause parts of the
system to freeze for many minutes up to hours when accessing files
part of the snapshot, depending on the size of the filesystem.
That's why I could not use it on my server with > 1TB UFS2.

Did this improve in the last year? (I guess my experience is from the
time around 9.0 release).

Kind regards,
Matthias


--
Matthias Petermann <matt...@d2ux.org>

[David Brodbeck]

On Mon, Mar 4, 2013 at 1:37 PM, CyberLeo Kitsana <cybe...@cyberleo.net>wrote:

> You can use dump(8) to dump a SU-journaled filesystem; you just cannot
> create a snapshot. This implies that dump(8) will be run against the
> live and possibly changing filesystem, which can lead to issues with the
> consistency of the contents of files thus dumped; but not necessarily
> with the consistency of the dump itself. Any tool that backs up a live
> filesystem, such as rsync or tar, will have these issues.
>

Note that this is mainly a problem for things like databases, where the
contents of multiple files, or different portions of the same file, have to
be in sync. For example, take your typical MySQL database table. You have
the actual data, in a .MYD file, and the indexes, in a .MYI file. If your
rsync backup hits while a table is being updated, it might get the .MYD
file before an update, and the .MYI file after, leaving the table and index
inconsistent. Or it might catch the .MYD file *partway* through an update,
giving a file that's internally inconsistent. This is likely to give very
unexpected results if you load the backup back into the database.

Note that even if you take a filesystem snapshot, if you don't halt
database updates while you take it, you can still end up with inconsistent
files. Snapshots are mostly useful for limiting the downtime in these
kinds of scenarios -- instead of taking the DB offline for the whole backup
window, you just down it long enough to take the snapshot.

In the absence of snapshots, the easiest way is to use whatever backup
tools the database offers to make sure a consistent copy exists to be
backed up. For example, before you run the backup, run mysqlhotcopy or
mysqldump to write-lock the database, make consistent backup copies of all
the files, then unlock it. That way, even if the backup of the active
database is inconsistent, the copies that were backed up along with it are
guaranteed to be consistent.

Anything database-like can have this problem; another common example is a
Subversion FSFS repository. Backing it up without running "svnadmin
hotcopy" first is asking for corrupt commits when you do a restore.