ZFS backups: retrieving a few files?
Andrew Reilly
Being a cutting-edge hipster, my new personal server is using a
ZFS RAIDZ array, where my older system used UFS on a GMIRROR.
This is, by and large, a joy: ZFS rocks.
One issue that I'm having difficulty coming to grips with,
though, is that of backup. I used to do a weekly cycle (with a
two week retention) of incremental backups to an external hard
drive. Dump/restore allowed this to be useful both for disaster
(hardware failure) recovery and for clumsiness (file destruction)
recovery.
Dump/restore doesn't work for ZFS. I *think* that I'm running
backups in the appropriate equivalent fashion: I take file
system snapshots (both absolute == level 0) and relative
(incremental), and zfs send those to files on the backup disk.
I haven't tried the recovery procedure yet, but it seems that
one would zfs receive the same files to snapshots, and then
either grab the files scrunged or perform the snapshot->working
file system upgrade process described in the manual, depending
on the situation.
The troubling aspect of this plan is that it would seem that
grabbing just a few files would require space in the working
zpool equivalent to the whole backed-up file system, for the
zfs receive of the snapshot. Contrast this with restore, which
had a nice interactive (or command line) mode of operation that
could retrieve nominated files or directories, requiring only
space for the results. Is there any similar tool that can
operate on a ZFS "send" serialization? It would seem that the
information must be "in" there somewhere, but I've not heard of
it.
Clues? How does everyone else manage this?
Cheers,
--
Andrew
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stabl...@freebsd.org"
Andriy Gapon
> The troubling aspect of this plan is that it would seem that
> grabbing just a few files would require space in the working
> zpool equivalent to the whole backed-up file system, for the
> zfs receive of the snapshot. Contrast this with restore, which
> had a nice interactive (or command line) mode of operation that
> could retrieve nominated files or directories, requiring only
> space for the results. Is there any similar tool that can
> operate on a ZFS "send" serialization? It would seem that the
> information must be "in" there somewhere, but I've not heard of
> it.
>
> Clues? How does everyone else manage this?
What I do is keep the snapshots for long enough and just retrieve the files from
there.
That is, snapshots - for the "clumsiness case", backups - for the disaster case.
--
Andriy Gapon
Jonathan Stewart
> Dump/restore doesn't work for ZFS. I *think* that I'm running
> backups in the appropriate equivalent fashion: I take file
> system snapshots (both absolute == level 0) and relative
> (incremental), and zfs send those to files on the backup disk.
This is actively discouraged, there is no recovery ability when
receiving zfs streams so 1 bad bit would invalidate your entire backup.
The currently accepted practice is to create a ZFS file system on the
backup drive and just keep sending incremental snapshots to it. As long
as the backup drive and host system have a snapshot in common you can do
incremental transfers. This way you only have to keep the most recent
snapshot on the main system and can keep as many as you have space for
on the backup drive. You also have direct access to any backed up
version of every file.
HTH,
Jonathan
Andrew Reilly
> On 11/22/2010 6:35 AM, Andrew Reilly wrote:
> >Dump/restore doesn't work for ZFS. I *think* that I'm running
> >backups in the appropriate equivalent fashion: I take file
> >system snapshots (both absolute == level 0) and relative
> >(incremental), and zfs send those to files on the backup disk.
>
> This is actively discouraged, there is no recovery ability when
> receiving zfs streams so 1 bad bit would invalidate your entire backup.
Hmm. Isn't that a problem that also affects the "sending
snapshots" scheme that you describe, below?
> The currently accepted practice is to create a ZFS file system on the
> backup drive and just keep sending incremental snapshots to it. As long
> as the backup drive and host system have a snapshot in common you can do
> incremental transfers. This way you only have to keep the most recent
> snapshot on the main system and can keep as many as you have space for
> on the backup drive. You also have direct access to any backed up
> version of every file.
That sounds like a very cool notion. Not unlike the
time-machine scheme. Interesting how different capabilities
require going back and re-thinking the problem, rather than just
trying to implement the old solution with the new tools.
I'll see how I go with it...
Cheers,
--
Andrew
Jonathan
> On Mon, Nov 22, 2010 at 10:26:30AM -0500, Jonathan Stewart wrote:
>> On 11/22/2010 6:35 AM, Andrew Reilly wrote:
>>> Dump/restore doesn't work for ZFS. I *think* that I'm running
>>> backups in the appropriate equivalent fashion: I take file
>>> system snapshots (both absolute == level 0) and relative
>>> (incremental), and zfs send those to files on the backup disk.
>>
>> This is actively discouraged, there is no recovery ability when
>> receiving zfs streams so 1 bad bit would invalidate your entire backup.
>
> Hmm. Isn't that a problem that also affects the "sending
> snapshots" scheme that you describe, below?
When sending the snapshots to a new file system you know right away
whether they are bad or not. Otherwise you find out when you are trying
to restore and your backups are bad... A send could fail but you can
always retry that, you can't retry anything when your trying to restore.
Jonathan
David Magda
>> The currently accepted practice is to create a ZFS file system on the
>> backup drive and just keep sending incremental snapshots to it. As
>> long
>> as the backup drive and host system have a snapshot in common you
>> can do
>> incremental transfers. This way you only have to keep the most
>> recent
>> snapshot on the main system and can keep as many as you have space
>> for
>> on the backup drive. You also have direct access to any backed up
>> version of every file.
>
> That sounds like a very cool notion. Not unlike the
> time-machine scheme. Interesting how different capabilities
> require going back and re-thinking the problem, rather than just
> trying to implement the old solution with the new tools.
As noted, saving the output of "zfs send" isn't very useful and
generally not recommended as a backup mechanism. It's come up quite
often on Sun/Oracle's zfs-discuss list:
http://www.google.com/search?q=zfs+send/receive+as+backup+tool
In addition to regular snapshots, also make sure to have an offline
backup of some kind (tar, Networker, NetBackup, Amanada, etc.). Errors
can propagate to online copies / backups, and if an intruder can
penetrate your primary system, there's a good chance they can get to
the secondary copy of your data; penetrating a tape on a shelf over
the network would be much more challenging. :)
Newer versions of ZFS also support a "zfs diff" command, but I'm not
sure if that functionality has made it into FreeBSD yet:
http://www.google.com/search?q=zfs+diff
Combine "diff" with some snapshots and scripting, and you can speed up
things like tar and rsync a lot since you don't have to traverse the
entire file system to find changes.
And at the end of the day remember it's not about about backups, but
about restores. If you can't restore coherent data in a timely manner
it's pointless.
Andrew Reilly
> On 11/22/2010 6:35 AM, Andrew Reilly wrote:
> >Dump/restore doesn't work for ZFS. I *think* that I'm running
> >backups in the appropriate equivalent fashion: I take file
> >system snapshots (both absolute == level 0) and relative
> >(incremental), and zfs send those to files on the backup disk.
>
> This is actively discouraged, there is no recovery ability when
> receiving zfs streams so 1 bad bit would invalidate your entire backup.
>
> backup drive and just keep sending incremental snapshots to it. As long
> as the backup drive and host system have a snapshot in common you can do
> incremental transfers. This way you only have to keep the most recent
> snapshot on the main system and can keep as many as you have space for
> on the backup drive. You also have direct access to any backed up
> version of every file.
For those playing along at home, I'll issue a small warning,
based on today's frolics:
Say, for example, one had done a:
zfs send -vR tank/home@0 | zfs receive -d /backup/snapshots
in order to experiment with this strategy.
One would then become alarmed when one discovered that the
receive mechanism also invoked the mountpoint= parameter of the
source filesystem, and the zfs propensity for just doing stuff,
and boom: you have a read-only version of your home directory
mounted *on top of* your actual home directory...
Required a reboot to single user mode, to go in and reset the
mountpoint setting for the newly created file system (by way of
hitting the power switch, after using zfs unmount -f to royally
screw things up, preventing subsequent network logins.) Left
wondering how to manage that change as part of an automated
backup schedule.
I think that this backup strategy has a few sharp edges...
No, I don't like tar, rsync and friends for backups: they don't
deal well with hard links, special files or sparse files.
Cheers,
--
Andrew
Thomas Ronner
> No, I don't like tar, rsync and friends for backups: they don't
> deal well with hard links, special files or sparse files.
>
rsync -avHxS --delete --numeric-ids /src/. /dst/.
Handles sparse files (S) and hard links (H). Never had any trouble with
special files. What sort of special files are not handled correctly by
rsync? I'd like to know because I'm relying on rsync for backups for
years on my home network.
Thomas
Freddie Cash
> On 11/23/10 1:45 PM, Andrew Reilly wrote:
>> No, I don't like tar, rsync and friends for backups: they don't
>> deal well with hard links, special files or sparse files.
>
> rsync -avHxS --delete --numeric-ids /src/. /dst/.
>
> Handles sparse files (S) and hard links (H). Never had any trouble with
> special files. What sort of special files are not handled correctly by
> rsync? I'd like to know because I'm relying on rsync for backups for years
> on my home network.
One problem with using rsync when dealing with hard-linked files: it
doesn't like it when the source switches from hard-linked to
non-hard-linked files. You end up with a mix of hard-linked and
non-hard-linked files in the destination, with the contents of the
non-hard-linked files all mixed around.
We just discovered this when we upgraded our Debian 4.0 (Etch) boxes
to Debian 5.0 (Lenny). On Etch, all the gzip tools are hard-links
(zcat, zless, gzip, gunzip, etc). On Lenny, they are all separate
files, and most are just shell scripts that use gzip. Doing an rsync
of a Lenny box onto a directory from an Etch box, you end up with some
hard-linked files, some regular files, and the contents of all the
files are mixed-up based on which source file (script or binary) was
read first.
We've had to resort to clearing out the backups directory when doing a
Debian upgrade, in order to guarantee that we get a clean backup via
rsync.
--
Freddie Cash
fjw...@gmail.com
Thomas Ronner
> On Tue, Nov 23, 2010 at 4:56 AM, Thomas Ronner<tho...@ronner.org> wrote:
>> rsync -avHxS --delete --numeric-ids /src/. /dst/.
> One problem with using rsync when dealing with hard-linked files: it
> doesn't like it when the source switches from hard-linked to
> non-hard-linked files. You end up with a mix of hard-linked and
> non-hard-linked files in the destination, with the contents of the
> non-hard-linked files all mixed around.
>
> We just discovered this when we upgraded our Debian 4.0 (Etch) boxes
> to Debian 5.0 (Lenny). On Etch, all the gzip tools are hard-links
> (zcat, zless, gzip, gunzip, etc). On Lenny, they are all separate
> files, and most are just shell scripts that use gzip. Doing an rsync
> of a Lenny box onto a directory from an Etch box, you end up with some
> hard-linked files, some regular files, and the contents of all the
> files are mixed-up based on which source file (script or binary) was
> read first.
>
> We've had to resort to clearing out the backups directory when doing a
> Debian upgrade, in order to guarantee that we get a clean backup via
> rsync.
Thanks for this info! I'm going to try to reproduce this.
Thomas.
Malcolm Waltz
"mount -v" to find the fsid for each double-mounted filesystem and then
umount them using the fsid.
To avoid this, always use the "-R" option (temporary, alternate root
mount point) with zpool if you are working with a root pool. This also
applies if you boot off of a CD or DVD and want to examine or modify a
root pool.
FWIW, I have successfully tested a bare metal restore from ZFS "full"
and "incremental" send/receive backups (from an NFS mount off of a
DataDomain). So, risks aside, it does work. I believe gzipped tar
files would also fail if a bit flipped.
No, I don't like tar, rsync and friends for backups: they don't
deal well with hard links, special files or sparse files.
Cheers,
--
Andrew
Mike Tancsa
> On Nov 22, 2010, at 17:13, Andrew Reilly wrote:
>
>>> backup drive and just keep sending incremental snapshots to it. As long
>>> as the backup drive and host system have a snapshot in common you can do
>>> incremental transfers. This way you only have to keep the most recent
>>> snapshot on the main system and can keep as many as you have space for
>>> on the backup drive. You also have direct access to any backed up
>>> version of every file.
>>
>> time-machine scheme. Interesting how different capabilities
>> require going back and re-thinking the problem, rather than just
>> trying to implement the old solution with the new tools.
>
> As noted, saving the output of "zfs send" isn't very useful and
> generally not recommended as a backup mechanism. It's come up quite
> often on Sun/Oracle's zfs-discuss list:
>
> http://www.google.com/search?q=zfs+send/receive+as+backup+tool
>
> In addition to regular snapshots, also make sure to have an offline
> backup of some kind (tar, Networker, NetBackup, Amanada, etc.). Errors
> can propagate to online copies / backups, and if an intruder can
> penetrate your primary system, there's a good chance they can get to the
> secondary copy of your data; penetrating a tape on a shelf over the
> network would be much more challenging. :)
I am still trying to figure out the best way to do zfs backups locally
here for rollbacks as well as DR. I was looking at some of the
techniques at
http://www.cuddletech.com/blog/pivot/entry.php?id=984
But thats outdated ? WRT errors in the file, perhaps PAR* tools can
overcome some of these issues if you are dumping to a file on tape
*/usr/ports/archivers/par2cmdline
---Mike
David Magda
> I am still trying to figure out the best way to do zfs backups locally
> here for rollbacks as well as DR. I was looking at some of the
> techniques at
>
> http://www.cuddletech.com/blog/pivot/entry.php?id=984
>
> But thats outdated ? WRT errors in the file, perhaps PAR* tools can
> overcome some of these issues if you are dumping to a file on tape
>
> */usr/ports/archivers/par2cmdline
The above is still quite valid, and snapshots are one of base principles
of ZFS. From the official ZFS Admin Guide:
> The following solutions for saving ZFS data are provided:
> * Saving ZFS snapshots and rolling back snapshots, if necessary.
> * Saving full and incremental copies of ZFS snapshots and restoring
> the snapshots and file systems, if necessary.
> * Remotely replicating ZFS file systems by saving and restoring ZFS
> snapshots and file systems.
> * Saving ZFS data with archive utilities such as tar and cpio or
> third-party backup products.
http://dlc.sun.com/osol/docs/content/ZFSADMIN/gbchx.html
You'll notice that 3/4 mention snapshots. I think them and zfs send/recv
are the starting points for getting consistent images of disks. There's no
equivalent to dump(8)/restore(8), and so tar and cpio are the main
utilities if you want offline stuff:
http://www.freebsd.org/doc/handbook/backup-basics.html
Until very recently the output format of "zfs send" was not stabilized, so
there was no guarantee that it was readable from one version to the next.
I believe that has been fixed in [Open]Solaris, but I haven't been
tracking pjd's commits that closely to know about FreeBSD.
Hopefully "zfs diff" will make it into FreeBSD soon-ish, so that it's
easier to do incremental backups to previous snapshots/check points.
Traversing large file systems is getting really old in this day-and-age,
and that one little thing can certainly remove a lot of I/O seeks if you
only want to grab the files that have changed recently.
See also the best practice guide, which should be generic enough to cover
most operating systems:
http://tinyurl.com/2gehn4#Recommendations_for_Saving_ZFS_Data
http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#Recommendations_for_Saving_ZFS_Data
--
David Magda
Toronto, Canada
Peter Jeremy
>zfs send -vR tank/home@0 | zfs receive -d /backup/snapshots
>
>in order to experiment with this strategy.
>
>One would then become alarmed when one discovered that the
>receive mechanism also invoked the mountpoint= parameter of the
>source filesystem, and the zfs propensity for just doing stuff,
>and boom: you have a read-only version of your home directory
>mounted *on top of* your actual home directory...
Been there, done that. The undocumented '-u' option to receive will
prevent the receive side performing mounts. The poorly documented
'-R' option on import allows you to specify an alternative root
mountpoint. Once you have done the initial transfer, you can also set
'canmount=noauto' on each fileset (it isn't inherited) to prevent ZFS
automounting them on import.
BTW, the entire export is performed at the current compression level -
recompressing existing data.
--
Peter Jeremy
Andrew Reilly
> On 11/23/10 1:45 PM, Andrew Reilly wrote:
> >deal well with hard links, special files or sparse files.
>
>
> special files. What sort of special files are not handled correctly by
> rsync? I'd like to know because I'm relying on rsync for backups for
> years on my home network.
I remember having problems with hard links, but it's possible
that I wasn't using rsync correctly. Of special files, I don't
remember non-dump backups doing well with the unix-domain
sockets that were liberally used and tricky to set up right for
djb's daemon-tools. Most uses of sockets put them in /tmp,
don't get backed up, and they don't need to persist. Dan puts
them in /var, and they're expected to persist across reboots,
which means they need to be backed-up. Maybe rsync and modern
tar handle these OK, but I remember at least one of them getting
wedged just trying to read one as a file.
The special files in /dev used to be a problem too, but that's
gone away, now that we have devfs.
I really like dump/restore. I expect that I will miss them.
Cheers,
--
Andrew
Alexander Leidinger
06:32:07 +1100):
> BTW, the entire export is performed at the current compression level -
> recompressing existing data.
Are you sure the compression is done on the sending side, and not at
the receiving side? I would expect the later (as I can specify a
different compression level on an existing destination, if I remember
correctly).
Bye,
Alexander.
--
Kansas state law requires pedestrians crossing the highways at night to
wear tail lights.
http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7
http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137
Peter Jeremy
>Quoting Peter Jeremy <peter...@acm.org> (from Wed, 24 Nov 2010
>06:32:07 +1100):
>
>> BTW, the entire export is performed at the current compression level -
>> recompressing existing data.
>
>Are you sure the compression is done on the sending side, and not at
>the receiving side? I would expect the later (as I can specify a
>different compression level on an existing destination, if I remember
>correctly).
Sorry, that was poorly worded. The actual send stream is not
compressed but the entire filesystem stream will be re-compressed on
the receive side as specified by the "compression" parameter on the
sending filesystem.
--
Peter Jeremy