zfs snapshots to optical disc

[Joe Reid]

I've never been one for backups but after nearly losing 20 years of data a while back I guess it's time to consider it. I've got a blu-ray writer in my new machine and I'd like to send zfs snapshots from a couple different filesystems to it in such a manner that I could mount the disc and recover individual files. Anyone scripted this up yet?

--
joe

[YTC#1]

Why use the blueray ?

Surely more economical and easier to have a large USB drive mounted (I
have a 500Gb one) and just snapshot then zfs send the files there ?

You can send them to a "dump" file for retrieval or a new FS.

Also you can step down the .zfs structure on the current machine if it
is a quick recover from a recent delete.


--
Bruce Porter
"The internet is a huge and diverse community but mainly friendly"
http://ytc1.blogspot.co.uk/
There *is* an alternative! http://www.openoffice.org/

[Joe Reid]

On Friday, December 5, 2014 11:00:36 AM UTC-6, YTC#1 wrote:
> Why use the blueray ?
>
> Surely more economical and easier to have a large USB drive mounted (I
> have a 500Gb one) and just snapshot then zfs send the files there ?

Single layer 25Gb BR-RE (blu-ray rewritable) discs are ~$2.50 each (bought a 5 pack for $12 plus tax).

--
joe

[Doug McIntyre]

YTC#1 <b...@ytc1-spambin.co.uk> writes:
>Why use the blueray ?

>Surely more economical and easier to have a large USB drive mounted (I
>have a 500Gb one) and just snapshot then zfs send the files there ?

Easier maybe. Economical and more reliable is questionable. I don't
trust any drive to last at most 10 years, even sitting dead, and maybe
not even more than 5.

OOTH, they do have burnable optical media that is supposed to have 100
year lifespans.
--
Doug McIntyre
do...@themcintyres.us

[Casper H.S. Dik]

YTC#1 <b...@ytc1-spambin.co.uk> writes:

>Surely more economical and easier to have a large USB drive mounted (I
>have a 500Gb one) and just snapshot then zfs send the files there ?

That is what I am using (I have a 4TB one, USB 3.0)

>Also you can step down the .zfs structure on the current machine if it
>is a quick recover from a recent delete.

Indeed.

The USB is for off-site storage.

Casper

[Casper H.S. Dik]

Doug McIntyre <mer...@dork.geeks.org> writes:

>YTC#1 <b...@ytc1-spambin.co.uk> writes:
>>Why use the blueray ?

>>Surely more economical and easier to have a large USB drive mounted (I
>>have a 500Gb one) and just snapshot then zfs send the files there ?

>Easier maybe. Economical and more reliable is questionable. I don't
>trust any drive to last at most 10 years, even sitting dead, and maybe
>not even more than 5.

Ah, but you'd need to keep the data on spinning rust and upgrade
the spinning rust every few years.

Casper

[John D Groenveld]

In article <54822c9c$0$2854$e4fe...@news2.news.xs4all.nl>,

Casper H.S. Dik <Caspe...@OrSPaMcle.COM> wrote:
>That is what I am using (I have a 4TB one, USB 3.0)

A mirrored pair duct taped together wins your snapshots some
protection assuming the twins don't die together, at the cost
of a second drive and USB port.

John
groe...@acm.org

[Joe Reid]

On Friday, December 5, 2014 4:07:27 PM UTC-6, Casper H. S. Dik wrote:

>
> The USB is for off-site storage.

My goal for this process was to have a pair of alternating discs so I could recover a file I deleted 2 weeks ago. For offsite, or archival storage I will use different solutions. This was just for immediate recovery and so I don't have to keep 2 weeks of snapshots, just 1 and still have 2 weeks of data.

My plan was to shapshot hourly, then every Sunday at ~4am move that snapshot to disc and alternate discs. This was I'm only storing 168 snapshots but still have 2 weeks of accessible recovery.

For offsite, yeah, probably 2 external disks, swapped quarterly. For long term archival storage I'd go with non-LTH blu-ray discs. Panasonic seems to be the leader at the moment, but I've also read that JVC is actively making/marketing a blu-ray disc as archival storage.

--
joe

[ITguy]

> I've never been one for backups but after nearly losing 20 years of data a while back I guess it's time to consider it. I've got a blu-ray writer in my new machine and I'd like to send zfs snapshots from a couple different filesystems to it in such a manner that I could mount the disc and recover individual files. Anyone scripted this up yet?

The fact that you're considering blu-ray leads me to believe we're not talking about huge volumes of data. For my personal data I simply tar-zip the bits I care about, encrypt the resulting file(s), and upload to Google drive for free.

[YTC#1]

And then you need to store and catalogue them, where as the USB device
would be just one item, very portable, and if you want replaceable.

I think there are ven dual drive ones out there, so you can have a ZFS
formatted one and gain some more protection :-)

But at least start taking some backups ...

[YTC#1]

On 05/12/2014 23:11, Joe Reid wrote:
> On Friday, December 5, 2014 4:07:27 PM UTC-6, Casper H. S. Dik
> wrote:
>
>>
>> The USB is for off-site storage.
>
> My goal for this process was to have a pair of alternating discs so I
> could recover a file I deleted 2 weeks ago. For offsite, or archival
> storage I will use different solutions. This was just for immediate
> recovery and so I don't have to keep 2 weeks of snapshots, just 1 and
> still have 2 weeks of data.

daily/weekly ZFS snapshot (I use daily) giving me 7 day rotation with
daily zfs send (1 week rotation) to my 500G USB device (USB3 but in a
USB 2 port)

>
> My plan was to shapshot hourly, then every Sunday at ~4am move that
> snapshot to disc and alternate discs. This was I'm only storing 168
> snapshots but still have 2 weeks of accessible recovery.

Activate time-slider ?

>
> For offsite, yeah, probably 2 external disks, swapped quarterly. For
> long term archival storage I'd go with non-LTH blu-ray discs.
> Panasonic seems to be the leader at the moment, but I've also read
> that JVC is actively making/marketing a blu-ray disc as archival
> storage.

Too much like hard work (IMO) for a home system (I assume this is a home
system)

[Joe Reid]

On Saturday, December 6, 2014 8:32:00 AM UTC-6, YTC#1 wrote:
> On 05/12/2014 23:11, Joe Reid wrote:
> > On Friday, December 5, 2014 4:07:27 PM UTC-6, Casper H. S. Dik
> >

> > For offsite, yeah, probably 2 external disks, swapped quarterly. For
> > long term archival storage I'd go with non-LTH blu-ray discs.
> > Panasonic seems to be the leader at the moment, but I've also read
> > that JVC is actively making/marketing a blu-ray disc as archival
> > storage.
>
> Too much like hard work (IMO) for a home system (I assume this is a home
> system)

I'm not sure where you're coming from here...when I was a professional Solaris admin there is no data at work that was worth anywhere near as much *to me* as something like my family photographs or my home directory with 20 years of personal source code and 20 years of email. Seriously, work could kiss off for the loss of family photographs, they were just investment bankers their data was truly irrelevant in the grand scheme of things. So 2 disks (or 2 mirrored pair), taken quarterly, rotated offsite seems pretty simple. As for lower volatility data, like my near 1Tb music collection; yes, those CDs are all in a couple of boxes over there (looks over shoulder at closet) but there is an enormous amount of work involved in recreating that data set, so a couple of bucks for archival quality discs to backup that data yearly seems pretty insignificant.

Again, this whole process was to produce something that would allow me to restore a file I deleted over 13 days ago (2 BD-sized disks rotated containing home directories). All of the data I'm talking about is either on a mirrored pair or a raidz2 volume, so disk failure isn't the issue. There would be no cataloging of discs, there would be 2 discs, this week's and last week's. With hourly snapshots for 7 days and 2 discs swapped every week, that would give me 14 days of recovery from being an idiot.

I don't see any of this as "too much work" once setup.

FWIW, I tinkered with mkisofs, but that doesn't like directories deeper than 7 levels. (Seriously? 7 levels?) Then I started playing with xorriso which writes straight to disc (including BD-REs). I need to tinker with the options to make it faster (thought BD-REs are only 2x) but it appears to be working in a manner that would allow me to recover a single file from a mounted disc.

--
joe

[YTC#1]

On 08/12/2014 08:36, Joe Reid wrote:
> On Saturday, December 6, 2014 8:32:00 AM UTC-6, YTC#1 wrote:
>> On 05/12/2014 23:11, Joe Reid wrote:
>>> On Friday, December 5, 2014 4:07:27 PM UTC-6, Casper H. S. Dik
>>>
>>> For offsite, yeah, probably 2 external disks, swapped quarterly.
>>> For long term archival storage I'd go with non-LTH blu-ray
>>> discs. Panasonic seems to be the leader at the moment, but I've
>>> also read that JVC is actively making/marketing a blu-ray disc as
>>> archival storage.
>>
>> Too much like hard work (IMO) for a home system (I assume this is a
>> home system)
>
> I'm not sure where you're coming from here...when I was a
> professional Solaris admin there is no data at work that was worth
> anywhere near as much *to me* as something like my family photographs
> or my home directory with 20 years of personal source code and 20

For someone who has not backed up his home data before that is a strange
statement.

I agree home/personal data is more important, and that is why I have
backed up (in various forms) for more than 20 years.

Ease of use is always a preference.

When backups got to big for floppy I then went to using a 2nd hard disk.

Twice over the years this has saved me from disaster (usually of my own
doing).

> years of email. Seriously, work could kiss off for the loss of
> family photographs, they were just investment bankers their data was
> truly irrelevant in the grand scheme of things. So 2 disks (or 2
> mirrored pair), taken quarterly, rotated offsite seems pretty simple.
> As for lower volatility data, like my near 1Tb music collection; yes,
> those CDs are all in a couple of boxes over there (looks over
> shoulder at closet) but there is an enormous amount of work involved
> in recreating that data set, so a couple of bucks for archival
> quality discs to backup that data yearly seems pretty insignificant.
>
> Again, this whole process was to produce something that would allow
> me to restore a file I deleted over 13 days ago (2 BD-sized disks

ZFS snapshot, rotated (as already suggested) over a 2 week period. Easy
bit of scripting.

> rotated containing home directories). All of the data I'm talking
> about is either on a mirrored pair or a raidz2 volume, so disk
> failure isn't the issue. There would be no cataloging of discs,
> there would be 2 discs, this week's and last week's. With hourly
> snapshots for 7 days and 2 discs swapped every week, that would give
> me 14 days of recovery from being an idiot.
>
> I don't see any of this as "too much work" once setup.

You have to insert the disk, take the disk out and then take it off site.
What happens when you are on holiday ?

>
> FWIW, I tinkered with mkisofs, but that doesn't like directories
> deeper than 7 levels. (Seriously? 7 levels?) Then I started
> playing with xorriso which writes straight to disc (including
> BD-REs). I need to tinker with the options to make it faster
> (thought BD-REs are only 2x) but it appears to be working in a manner
> that would allow me to recover a single file from a mounted disc.
>

Just use a USB attached HD. As I said, less work :-)

[Joe Reid]

On Monday, December 8, 2014 4:09:35 AM UTC-6, YTC#1 wrote:
> For someone who has not backed up his home data before that is a strange
> statement.

you're right, I'm trying to improve my system - it's going to be more work than whatever I had before.

> When backups got to big for floppy I then went to using a 2nd hard disk.
>
> Twice over the years this has saved me from disaster (usually of my own
> doing).

I've been using ZFS to cover the disasters that weren't of my own making for as long as it has been around, and that has saved me from disaster more than once - I've lost a couple of HDDs, just enough to make me paranoid about "spinning rust". But I lost a controller card a few months ago and for about 24 hours every bit of data I'd generated in 20 years was inaccessible. I finally coaxed the system to mount half the zfs mirror again and got it back.

> You have to insert the disk, take the disk out and then take it off site.
> What happens when you are on holiday ?

the optical discs aren't for offsite, the 2 HDD (pair) built quarterly, using zfs send, are for offsite.

--
joe