zfs send/receive... and tape... and stuff...
Gary R. Schmidt
ZFS array, and as a result doubling the size of the zpool and once again
not being able to back it up onto a single chunk of whatever without
doing things bit-by-bit, did anyone ever do anything about solving the
"zfs send to /dev/rmt/0 doesn't work" problem?
Because if they didn't, I may be forced to actually write some real code
for a change and solve it...
What I am thinking of is basically appropriating the tape-handling code
from tar/cpio/whatever and using that in a wrapper around zfs send and
receive, like this:
magic_program -r "zfs send -R pool@snapshot" -f /dev/rmt/0
to put a zfs stream onto a tape, and:
magic_program -w "zfs receive -Fud filesystem" -f /dev/rmt/0
to restore a zpool from a stream.
So my magic program would chunk up the data from "zfs send" and put
headers/footers/checksums and stuff like that on it, write/read the
resulting blocks to/from the device, and, when a write/read fails, ask
for a new "tape" and carry on from there.
Does this make sense? Or am I suffering from Friday-itis?
Cheers,
Gary B-)
cindy
Hi Gary,
Its only Thursday but a good topic.
Someone on zfs-discuss posted a zfsdump feature that you give some
more
ideas:
http://www.quantmodels.co.uk/zfsdump/
Or, look at the Amanda or Bacula links on the ZFS BP site:
http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide
If you have additional disks and are running a current OpenSolaris
build like 131,
you might consider the zpool split feature, where you split a pool and
move
it to another system. I'm not saying this would replace snapshots or
backing
up data, but provides a pool replication method.
Thanks,
Cindy
John D Groenveld
Gary R. Schmidt <grsc...@acm.org> wrote:
>So my magic program would chunk up the data from "zfs send" and put
>headers/footers/checksums and stuff like that on it, write/read the
>resulting blocks to/from the device, and, when a write/read fails, ask
>for a new "tape" and carry on from there.
Just beware that a corrupted ZFS stream cannot be received.
I don't know the reliability of your tape streamer, but I suspect
you're better off using conventional archive tools to backup your
snapshots to it and only sending your ZFS streams to media that
has error correction.
John
groe...@acm.org
Thommy M.
> Its only Thursday but a good topic.
Don't be to sure about that. Gary might live elsewhere from you... ;)
Gary R. Schmidt
> Its only Thursday but a good topic.
> Someone on zfs-discuss posted a zfsdump feature that you give some
> more
> ideas:
>
> http://www.quantmodels.co.uk/zfsdump/
Interesting - but it looks like you need n storage locations, rather
than a single output.
> Or, look at the Amanda or Bacula links on the ZFS BP site:
I use Bacula at work, it needs a custom file daemon to be written to use
snapshots, unless I've missed something. (I mean to use snapshots to
backup filesystems, it can backup ".zfs" directory trees quite happily,
but that's not quite what I want[1].)
I've also noticed there is/was an NDMP project on the OpenSOlarius
things, I may look into that, does anyone know of a free NDMP backup tool?
Cheers,
Gary B-)
1 - This needs some more explanation.
If a file system looks like:
/a/b/c
/a/b/.zfs/backup
then it is trivial to backup /a/b/.zfs/backup, but there is no easy
mechanism to restore /a/b/c from that backup. It is doable, but it is a
manual task.
Gary R. Schmidt
Line, it was Thursday when I posted!
Cheers,
Gary B-)
Gary R. Schmidt
> In article <vsmuf7-...@paranoia.mcleod-schmidt.id.au>,
> Gary R. Schmidt <grsc...@acm.org> wrote:
>> So my magic program would chunk up the data from "zfs send" and put
>> headers/footers/checksums and stuff like that on it, write/read the
>> resulting blocks to/from the device, and, when a write/read fails, ask
>> for a new "tape" and carry on from there.
>
> Just beware that a corrupted ZFS stream cannot be received.
> I don't know the reliability of your tape streamer, but I suspect
> you're better off using conventional archive tools to backup your
> snapshots to it and only sending your ZFS streams to media that
> has error correction.
This lead me to think about using eSATA/USB disks - create a ZFS file
system on them and that problem goes away!
Cheers,
Gary B-)
Thomas Maier-Komor
mbuffer can do exactly what you want. Additionally, mbuffer speeds up
the whole process. Some people have reported that it works for them on
the zfs-discuss mailing list.
You can get it either as source code from the main site [1] or as a
binary package [2].
[1]: http://www.maier-komor.de/mbuffer.thml
[2]: http://www.opencsw.org/packages/CSWmbuffer/
HTH,
Thomas
Gary R. Schmidt
> On 01.07.10 16:55, Gary R. Schmidt wrote:
>> I was just wondering, since I'm about to start rolling new disks into my
>
> mbuffer can do exactly what you want. Additionally, mbuffer speeds up
> the whole process. Some people have reported that it works for them on
> the zfs-discuss mailing list.
>
but I couldn't think in relation to what.
I now recall seeing it when I was looking for sysstat some time ago.
Thank you, Thomas.
Cheers,
Gary B-)
Tristram Scott
> cindy wrote:
>> On Jul 1, 8:55 am, "Gary R. Schmidt" <grschm...@acm.org> wrote:
> [SNIP]
>> Its only Thursday but a good topic.
> Ta.
>
>> Someone on zfs-discuss posted a zfsdump feature that you give some
>> more ideas:
>>
>> http://www.quantmodels.co.uk/zfsdump/
> Interesting - but it looks like you need n storage locations, rather
> than a single output.
>
Yes, I haven't implemented end of media detection yet. Instead, you
declare the tape size on the command line. It will write the first tape,
then return. You change tapes, and call zfsdump again to write the second
tape, and so on until the dump is complete.
I have it reporting checksums, per tape and for the stream as a whole, but
don't incorporate any error correction code. My experience with tape is
that errors are very infrequent, and almost always seem to show up on
write.
If you are after an archive solution, i.e. put your filesystem on tape and
keep it there for many years, then I would suggest tar or similar, because
a single error won't corrupt the entire archive, just the file it relates
to.
For a disaster recovery backup, I am happy with zfs send / receive, which
is what I have built zfsdump around. I like to backup to tape so it is
physically removed from the system, not prone to lightening strikes etc.
Chances of a catastrophic hardware failure (or operator error) are small,
but it does happen. Chances of tape corruption are also small, but this
also does happen. The key thing here, though, is the independance of the
two failures. What circumstances contribute to higher probability of
disk failure? Do these circumstances also mean higher probability of tape
error?
For partial disasters, e.g. I just deleted the wrong file or directory,
then restore from zfs snapshot without any need to refer to tape.
The zfs send method allows dumping of the complete zfs filesystem,
including ACLS and snapshots. After disaster recovery, I still have the
ability to roll back to last week or last month. An archive using tar
won't give you that. I understand that using star instead of tar you can
get close to this, so maybe that is something to look into in more detail.
Anyway, if your problem is that you want to split a zfs send stream across
several tapes, in sequence (not in parallel) then zfsdump will do this for
you. If you are keen to write something more robust, please let me know of
your progress.
My opinion is the same as it was when Sun first announced zfs. If they want
people to take zfs seriously, then they should provide a backup solution to
go with it. This is something that Sun should be implementing themselves.
I buy their servers, with their disk arrays and their tape drives. I run
their operating system, and I backup to tape media which they sell me. I
even buy their support contracts. The bit that is missing is something
from them to allow me to dump and restore zfs to tape(s). It just leaves
zfs with an unfinished feel to it.
--
Dr Tristram J. Scott
Energy Consultant
John D Groenveld
Tristram Scott <tristra...@ntlworld.com> wrote:
>won't give you that. I understand that using star instead of tar you can
>get close to this, so maybe that is something to look into in more detail.
I've been using the incremental backup feature of Joerg Schilling's
star for a few years now. See INCREMENTAL BACKUPS in star(1).
Works fine, though I'm no longer archiving to tape, just local disks
and NFS over the LAN and WAN.
I'm doing the same with my ZFS sends.
Where the archive/backup media is RAIDZ, I'm not stressing
corruption.
Not sure I'll buy another tape drive again.
Happy hacking!
John
groe...@acm.org
David Combs
>ZFS array, and as a result doubling the size of the zpool and once again
>not being able to back it up onto a single chunk of whatever without
>doing things bit-by-bit, did anyone ever do anything about solving the
>"zfs send to /dev/rmt/0 doesn't work" problem?
>
>Because if they didn't, I may be forced to actually write some real code
>for a change and solve it...
>
>What I am thinking of is basically appropriating the tape-handling code
>from tar/cpio/whatever and using that in a wrapper around zfs send and
Some years ago on this newsgroup there was a long thread on the "tar" version
named STAR. I believe that the concensus was that that its code was
cleaner and bug-freeer than any of the other tars (sun, gnu, ...).
Plus a whole lot more features, needed features, I think.
If you do some of your own coding, why not use STAR as your base?
(I mean, from that long thread, both sun and gnu tar had bugs -- and
now with no more ufsrestore and ufsdump, givcn zfs, those bugs might
now turn out to not be minor AT ALL -- not when doing the most
critical tape-writing at one's site, making backups. So STAR?)
PLEASE, some follow-ups from those who know (and use) STAR --
so we can all learn more about it.
(One very much needed addition to its man-page would be a
whole set of examples, ANNOTATED, using those very, very many
options that program provides.)
THANKS!
David
David Combs
John D Groenveld <groe...@cse.psu.edu> wrote:
>In article <zGe%n.231249$k15.144024@hurricane>,
>Tristram Scott <tristra...@ntlworld.com> wrote:
>>won't give you that. I understand that using star instead of tar you can
>>get close to this, so maybe that is something to look into in more detail.
>
>I've been using the incremental backup feature of Joerg Schilling's
>star for a few years now. See INCREMENTAL BACKUPS in star(1).
Does that mean that when doing that under zfs that
your backup-ing is NOT making use of zfs "snapshots" -- of their
own way of doing "incrementals"?
What about the equivalent of a "level 0" dump?
. do you let star do all the work, traverse the file-tree,
. or do you grab a snapshot, and save that?
(Star, of course(?), wouldn't know what to do
with a "level 0" (or any other level) zfs-snapshot,
so I guess star WOULD have to do its own "level 0"?
>
>Works fine, though I'm no longer archiving to tape, just local disks
>and NFS over the LAN and WAN.
Well, there are some few of us who still ARE making tape backups,
and then storing them off-premises in eg bank safe-deposit boxes --
THEREFORE, please do at least TALK about the way you WOULD
do tape backups, as advice and education for us tape-types.
THANKS!
>
>I'm doing the same with my ZFS sends.
PLEASE say more about this. By "doing the same", does
that mean star, for writing out snapshots?
>Where the archive/backup media is RAIDZ,
This means you have SEVERAL identical disks that you have zfs
write (in parallel) your snapshots to?
I'm not stressing
>corruption.
What are you saying? That zfs catches and fixes so many of
the possible disk-problems, that when using zfs, you're
just not going to worry about it?
>
>Not sure I'll buy another tape drive again.
Again, when posting on backups, please do include
the case of writing it to a tape, as benefit
to those who do still use tapes.
THANK YOU!
David
David Combs
Don't modern tape drives (ie for the last 10 or 15 years or so)
already have a lot of error-correction facilities?
EG the Exabyte VXA series, for one?
Or are you talking about something else?
Thanks!
David
David Combs
You're saying that disks (eSATA/USB, as opposed to scsi?) are a
more robust means of storing critical data?
What about off-site? Wouldn't tapes be easier there?
For LONG-TERM storage (years), I'd heard that these newish
"metal based" tapes simply wouldn't lose data (maybe that's
just the sales pitch).
Especially if you mistakenly drop it -- not much would happen
to a tape (in the plastic box it originally came in), but
a disk, being mechanical, with moving parts, etc, seems
like it might "break"?
Thanks,
David
Ian Collins
> In article<i0j2bn$gc6i$1...@tr22n12.aset.psu.edu>,
> John D Groenveld<groe...@cse.psu.edu> wrote:
>> In article<vsmuf7-...@paranoia.mcleod-schmidt.id.au>,
>> Gary R. Schmidt<grsc...@acm.org> wrote:
>>> So my magic program would chunk up the data from "zfs send" and put
>>> headers/footers/checksums and stuff like that on it, write/read the
>>> resulting blocks to/from the device, and, when a write/read fails, ask
>>> for a new "tape" and carry on from there.
>>
>> Just beware that a corrupted ZFS stream cannot be received.
>>
>> I don't know the reliability of your tape streamer, but I suspect
>> you're better off using conventional archive tools to backup your
>> snapshots to it and only sending your ZFS streams to media that
>> has error correction.
>>
> already have a lot of error-correction facilities?
>
All it takes is one bad bit in a zfs stream file and the whole thing is
a goner. Writing zfs stream files to tape is a bad idea, use a
conventional backup tool.
One suggestion that crops up on the zfs lists for those who do want to
write directly to tape is to create a redundant pool out of files and
write those files to tape. Then you have a better chance of recovery if
a file is corrupt.
--
Ian Collins
Richard B. Gilbert
The author of star used to hang out here but I don't recall that he has
posted here in the last year or two. His name is Georg Schilling
(subject to lapses in my recollection of the correct spelling). You
should be able to find some of his posts in the archives of this news group.
As for the man page, it's traditional to make them terse and cryptic
just like the rest of Unix. After all, the source code IS available. .
. . ;-)
John D Groenveld
David Combs <dkc...@panix.com> wrote:
>Does that mean that when doing that under zfs that
>your backup-ing is NOT making use of zfs "snapshots" -- of their
>own way of doing "incrementals"?
I would do a full backup of your ZFS snapshots either via
zfs send or star(1).
>Well, there are some few of us who still ARE making tape backups,
>and then storing them off-premises in eg bank safe-deposit boxes --
>THEREFORE, please do at least TALK about the way you WOULD
>do tape backups, as advice and education for us tape-types.
I recommend you stop writing to tape and move your backups
to your off-site location via the wire instead of via your feet.
But if you insist on writing to tape, the star(1) manpage
shows you how.
Let us know what problems you have implementing your
particular backup solution.
John
groe...@acm.org