Archive 20GB?
Stan Bischof
Have need of archiving around 20GB with a million
or so files to DVD. DVD is _not_
local so need to create files that can be later moved
to machine with DVD writer and then written. Network
mount is not feasible in this case.
In old days this would be a split tarball ( tar piped into split )
with small enough chunks to write to DVD.
RHEL 5.6 system
Is there a better way to write the archives? Ideally one
in which I don't have to recreate the entire tarball
to dig out one archived file, but also one which will be
readable in 5 or 10 years?
thanks
Stan
Dave U. Random
SB> Have need of archiving around 20GB with a million or so files to
SB> DVD. DVD is _not_ local so need to create files that can be later
SB> moved to machine with DVD writer and then written.
SB>
SB> Is there a better way to write the archives? Ideally one in which
SB> I don't have to recreate the entire tarball to dig out one archived
SB> files
rar a -v4500k archive /path/to/the/million/files
will create archive.part001.rar, archive.part002.rar, ... which
can be used individually for file recovery.
rar l -v archive.part00N.rar
will print a list of all files contained in parts N, N+1, ...
SB> but also one which will be readable in 5 or 10 years?
rar packages are available in source and binary form for most linux
distributions. rar is also available on many other operating systems.
Keith Keller
>
> rar packages are available in source and binary form for most linux
> distributions. rar is also available on many other operating systems.
I thought creating rar archives required proprietary and/or nonfree
software. (I know extracting them is possible using the free unrar
utility.)
--keith
--
kkeller...@wombat.san-francisco.ca.us
(try just my userid to email me)
AOLSFAQ=http://www.therockgarden.ca/aolsfaq.txt
see X- headers for PGP signature information
Kevin the Drummer
If it were me, then I'd put it onto a hard drive and lock away
the hard drive. 40-80GB hard drives are dirt cheap. Buy 2 or 3.
Write to all of them. Keep them in different time zones in case
of disaster. Maybe even write to a 32GB SD card.
Good luck....
--
PLEASE post a SUMMARY of the answer(s) to your question(s)!
Unless otherwise noted, the statements herein reflect my personal
opinions and not those of any organization with which I may be affiliated.
David Brown
Presumably you've already thought of this, is the tarball still too big
when you've bzip'ed it?
And can you use double-layer DVD's that are something like 7.5 GB?
The Natural Philosopher
> Stan Bischof <st...@worldbadminton.com> wrote:
>> Have need of archiving around 20GB with a million
>> or so files to DVD. DVD is _not_
>> local so need to create files that can be later moved
>> to machine with DVD writer and then written. Network
>> mount is not feasible in this case.
>>
>> In old days this would be a split tarball ( tar piped into split )
>> with small enough chunks to write to DVD.
>>
>> RHEL 5.6 system
>>
>> Is there a better way to write the archives? Ideally one
>> in which I don't have to recreate the entire tarball
>> to dig out one archived file, but also one which will be
>> readable in 5 or 10 years?
>
> If it were me, then I'd put it onto a hard drive and lock away
> the hard drive. 40-80GB hard drives are dirt cheap. Buy 2 or 3.
> Write to all of them. Keep them in different time zones in case
> of disaster. Maybe even write to a 32GB SD card.
>
> Good luck....
>
up and archived to another disk.
Dave U. Random
KK> I thought creating rar archives required proprietary and/or
KK> nonfree software.
No, the archiver is not proprietary. It's shareware. That's why it's
available from the nonfree distribution archives.
http://packages.debian.org/sid/rar says:
This is the RAR archiver from Eugene Roshal. It supports multiple
volume archives and damage protection. It can also create
SFX-archives. There are versions which run on DOS, Windows
(3.1x,95,NT), FreeBSD, BSDI.
This program is shareware and you must register it after 40 days of use.
David Brown
Shareware is proprietary, closed-source, and illegal to use (except for
demonstration and testing purposes) unless you pay for it.
unrar is free and open source, as far as I know, so you won't lose
access to your data.
Keith Keller
> Keith Keller <kkeller-use...@wombat.san-francisco.ca.us> [KK]:
> KK> I thought creating rar archives required proprietary and/or
> KK> nonfree software.
>
> No, the archiver is not proprietary. It's shareware. That's why it's
> available from the nonfree distribution archives.
By Debian's definition, if it's in nonfree then it is nonfree and/or
proprietary.
> http://packages.debian.org/sid/rar says:
>
> This is the RAR archiver from Eugene Roshal. It supports multiple
> volume archives and damage protection. It can also create
> SFX-archives. There are versions which run on DOS, Windows
> (3.1x,95,NT), FreeBSD, BSDI.
> This program is shareware and you must register it after 40 days of use.
Well, if the OP wants a program that can create archives for many years
then this rar version is not free-as-in-beer for him. That may be perfectly
acceptable to him, but you should be clear that creating rar archives is
not free-as-in-speech, and is gratis for a limited time only. (I also
note that your Debian page did not have links to source code, another
indicator that the code is not actually freely available.)
Please note that I am not complaining about rar's licensing terms, but
about inaccurately calling it "free" when it isn't.
Aragorn
as Dave U. Random wrote...
> No, the archiver is not proprietary. It's shareware. That's why it's
> available from the nonfree distribution archives.
>
> http://packages.debian.org/sid/rar says:
>
> This is the RAR archiver from Eugene Roshal. It supports multiple
> volume archives and damage protection. It can also create
> SFX-archives. There are versions which run on DOS, Windows
> (3.1x,95,NT), FreeBSD, BSDI.
> This program is shareware and you must register it after 40 days of
> use.
Both "shareware" and "freeware" *are* proprietary software. Proprietary
means that it does not have a Free/Libre Software license, and that the
source code is not open.
The word "Free" in "Free Software" pertaints to "freedom", not
to "gratis".
http://www.gnu.org/philosophy/free-sw.html
--
*Aragorn*
(registered GNU/Linux user #223157)
Stan Bischof
with ttried and true: tar and DVD since the odds of
reading those in 10 years are much higher than
most other solutions.
Stan
Mark
DVDR & RW are very poor as an archive medium as they don't last very
long.
With backing up or achiving quantity is the key. i.e. Make /many/
different copies on different media and in different places.
--
(\__/) M.
(='.'=) Due to the amount of spam posted via googlegroups and
(")_(") their inaction to the problem. I am blocking some articles
posted from there. If you wish your postings to be seen by
everyone you will need use a different method of posting.
Mark
wrote:
I would estimate the odds of being able to read a DVDR or RW after 10
years to be very small.
Tony
> On 24 May 2011 18:58:15 GMT, Stan Bischof<st...@worldbadminton.com>
> wrote:
>
>> Thanks for the various hints. I ended up sticking
>> with ttried and true: tar and DVD since the odds of
>> reading those in 10 years are much higher than
>> most other solutions.
>
> I would estimate the odds of being able to read a DVDR or RW after 10
> years to be very small.
Estimate based on any evidence? Or a gut feel?
--
Tony Evans
I'm trying to revive uk.media.films - why not join me there?
[ anything below this line wasn't written by me ]
Keith Keller
>
> I would estimate the odds of being able to read a DVDR or RW after 10
> years to be very small.
Why do you say that? We can still read 3.5" floppies, after all. If
you've still got the hardware you could probably still read a 5.25"
floppy even.
In 10 years we won't be able to read floppy disks, but it's hard to
imagine not being able to read DVD media in that time.
Rui Maciel
> Estimate based on any evidence? Or a gut feel?
I've had a couple of boxes of CD-Rs fail after being stored for about 6
years. These weren't cheap chinese store knock-off CD-Rs. They were
stored in air-tight containers and away from sunlight, which is supposed
to be the right way of storing optical media. Granted, not all 50 CD-Rs
failed, but a considerable number of them failed, and I was left without
any way to recover that data after 6 years.
This case was a personal anecdote which can't possibly be generalized to
every case where someone stored information on CD-Rs, let alone DVDs.
Yet, this is also a bit beyond the "gut feeling" territory.
Rui Maciel
Jean-David Beyer
> On 2011-05-25, Mark <i...@dontgetlotsofspamanymore.invalid> wrote:
>> I would estimate the odds of being able to read a DVDR or RW after 10
>> years to be very small.
>
> Why do you say that? We can still read 3.5" floppies, after all. If
> you've still got the hardware you could probably still read a 5.25"
> floppy even.
I do not know about that. After replacing several floppy drives, I just
took them out of my computers. When I get brand new floppy discs,
perhaps 25% of them cannot be formatted. Others can be formatted, but if
I try to compare what I wrote on them with the originals, I got failures
right away. In the old days 5 1/4 and 3 1/2 floppies worked quite well,
but by 2005 or so, they were useless.
>
> In 10 years we won't be able to read floppy disks, but it's hard to
> imagine not being able to read DVD media in that time.
I have been able to read CD-R disks 5 years later. But I do not know if
I have written enough of them to have statistically significant results.
I guess I do know: my experience is probably not statistically significant.
I do my backups on VXA digital tapes using VXA-2 tape drives. I can
certainly read them easily a year later, and probably much longer. I
have been using magnetic tape since the IBM 7-track tapes and they have
all worked perfectly many years after writing them. Of course, I doubt
you could get an IBM 7-track compatible tape drive these days. I have
used the 4mm DAT tapes, but they were hopeless. I used some cartridge
tapes that hooked up to floppy controllers (QIC?) and they worked for
ears too, until I could not get drive belts (o-rings) for the drive. I
switched to a VXA-1 tape drive in about 2002 after giving up on the DAT
drive, and upgraded to VXA-2 drives somewhat later.
>
> --keith
>
--
.~. Jean-David Beyer Registered Linux User 85642.
/V\ PGP-Key: 9A2FC99A Registered Machine 241939.
/( )\ Shrewsbury, New Jersey http://counter.li.org
^^-^^ 11:15:01 up 39 days, 14:33, 3 users, load average: 4.96, 4.95, 4.98
Mark
<kkeller...@wombat.san-francisco.ca.us> wrote:
>On 2011-05-25, Mark <i...@dontgetlotsofspamanymore.invalid> wrote:
>>
>> I would estimate the odds of being able to read a DVDR or RW after 10
>> years to be very small.
>
>Why do you say that? We can still read 3.5" floppies, after all. If
>you've still got the hardware you could probably still read a 5.25"
>floppy even.
Totally different technology. Floppy disks are magnetic & DVDs are
optical.
>In 10 years we won't be able to read floppy disks, but it's hard to
>imagine not being able to read DVD media in that time.
It's easy to imagine. The dyes used in CD & DVD media decay over
time. In a good environment they can be made to last longer but I
wouldn't rely on any dye or film based optical media lasting. I have
had several fail after a very short time myself and this is not
uncommon.
Jean-David Beyer
> On Wed, 25 May 2011 07:25:14 -0700, Keith Keller
> <kkeller...@wombat.san-francisco.ca.us> wrote:
>
>> On 2011-05-25, Mark <i...@dontgetlotsofspamanymore.invalid> wrote:
>>> I would estimate the odds of being able to read a DVDR or RW after 10
>>> years to be very small.
>> Why do you say that? We can still read 3.5" floppies, after all. If
>> you've still got the hardware you could probably still read a 5.25"
>> floppy even.
>
> Totally different technology. Floppy disks are magnetic & DVDs are
> optical.
>
>> In 10 years we won't be able to read floppy disks, but it's hard to
>> imagine not being able to read DVD media in that time.
>
> It's easy to imagine. The dyes used in CD & DVD media decay over
> time. In a good environment they can be made to last longer but I
> wouldn't rely on any dye or film based optical media lasting. I have
> had several fail after a very short time myself and this is not
> uncommon.
Decades ago, someone well known in the computer business, but I forget
just who, remarked
Data should be stored magnetically, as God intended.
--
.~. Jean-David Beyer Registered Linux User 85642.
/V\ PGP-Key: 9A2FC99A Registered Machine 241939.
/( )\ Shrewsbury, New Jersey http://counter.li.org
^^-^^ 11:55:01 up 39 days, 15:13, 3 users, load average: 4.63, 4.77, 4.95
The Natural Philosopher
> On 2011-05-25, Mark <i...@dontgetlotsofspamanymore.invalid> wrote:
>> I would estimate the odds of being able to read a DVDR or RW after 10
>> years to be very small.
>
> Why do you say that? We can still read 3.5" floppies, after all. If
> you've still got the hardware you could probably still read a 5.25"
> floppy even.
>
> In 10 years we won't be able to read floppy disks, but it's hard to
> imagine not being able to read DVD media in that time.
>
> --keith
>
The Natural Philosopher
that is also my worry.
By the time I had costed out what keeping 10GB of data would cost on an
other medium, plus the hassle of it, I simply stuck a second disk on the
server...
If you are really paranoid, put a third disk on a machine somewhere
else.And trickle the data across the internet to it.
Grant Edwards
> On 2011-05-25, Mark <i...@dontgetlotsofspamanymore.invalid> wrote:
>>
>> I would estimate the odds of being able to read a DVDR or RW after 10
>> years to be very small.
>
> Why do you say that?
CDs have a variety of known aging-related failure mechanism.
> We can still read 3.5" floppies, after all.
Sometimes, if you're lucky. I've seen a lot of floppy disks go bad.
> If you've still got the hardware you could probably still read a
> 5.25" floppy even.
>
> In 10 years we won't be able to read floppy disks, but it's hard to
> imagine not being able to read DVD media in that time.
If you've read any of the test reports on how CDs and DVDs age, it's
not hard to imagine at all.
--
Grant Edwards grant.b.edwards Yow! How do I get HOME?
at
gmail.com
Rui Maciel
> that is also my worry.
>
> By the time I had costed out what keeping 10GB of data would cost on an
> other medium, plus the hassle of it, I simply stuck a second disk on the
> server...
>
> If you are really paranoid, put a third disk on a machine somewhere
> else.And trickle the data across the internet to it.
Indeed. Nowadays, HDs tend to go for around 0.10 euros/MB. Although DVD-
Rs tend to go around half that much, in the end the cost difference will
amount to peanuts. If we factor in how inconvenient it is to use DVDs to
store data when compared with simply pluging in a HD then there isn't much
left in favour of DVDs.
There is also the option to rely on any form of flash memory. It may be
more expensive than HDs and DVD-Rs in terms of price per GB but it appears
to be much more reliable than the other options in this sort of
application.
Rui Maciel
Keith Keller
> On Wed, 25 May 2011 07:25:14 -0700, Keith Keller
><kkeller...@wombat.san-francisco.ca.us> wrote:
>
>>In 10 years we won't be able to read floppy disks, but it's hard to
>>imagine not being able to read DVD media in that time.
>
> It's easy to imagine. The dyes used in CD & DVD media decay over
> time. In a good environment they can be made to last longer but I
> wouldn't rely on any dye or film based optical media lasting. I have
> had several fail after a very short time myself and this is not
> uncommon.
Bah, I apologize for completely misunderstanding--I read your comment as
"DVD drives will no longer work", not "DVD media will not work".
TomB
>
> Decades ago, someone well known in the computer business, but I forget
> just who, remarked
>
> Data should be stored magnetically, as God intended.
Amen to that. Nothing beats a good tape back-up.
--
"Programming today is a race between software engineers striving to
build bigger and better idiot-proof programs, and the Universe trying to
produce bigger and better idiots. So far, the Universe is winning."
~ Rick Cook
Dave U. Random
RM> If we factor in how inconvenient it is to use DVDs to store data
RM> when compared with simply pluging in a HD then there isn't much left
RM> in favour of DVDs.
Are you sure that you will be able to plug that 10-year old HD
in your future computer? Once popular buses, connectors, protocols, even
sockets for external units, tend to get replaced after a decade.
IMO, HDs are fine for archiving data, if you update your archives
every 6 years or so, using disks of newer technology.
(But they can certainly be used much longer for data backup purposes).
Rui Maciel
> Are you sure that you will be able to plug that 10-year old HD
> in your future computer?
Either through the SATA cable, HD dock or external HD case.... Yes. And
assuming that, somehow, the original hardware that you used with that HD
mysteriously vanished.
> Once popular buses, connectors, protocols, even
> sockets for external units, tend to get replaced after a decade.
USB is still going strong, although it was introduced in 1996. Parallel
ATA was introduced in 1986 and you can still buy motherboards which
support it. Serial ATA was introduced in 2003 and it won't go away for a
long, long time. So, I expect this issue to be covered.
> IMO, HDs are fine for archiving data, if you update your archives
> every 6 years or so, using disks of newer technology.
The deskop I work on has a single HD which I salvaged from a previous
desktop I owned, which was built around a AMD Athlon XP 2000+ processor.
That makes the HD to be around 8 or 9 years old. I've been using it
practically every single day since I've bought it. So, if a HD takes up
that level of beating without exhibiting a single problem then I believe
it is safe to assume that it's possible to use HDs to backup data for
periods over 6 years. And this without adding RAID to the mix.
Rui Maciel
The Natural Philosopher
If it fails, it is mirrored. Change it and re mirror.
Robert Riches
While I don't remember the exact source, I have read that
DVD+/-Rs use a different, much more stable, dye than CD-Rs and
should last much longer.
--
Robert Riches
spamt...@jacob21819.net
(Yes, that is one of my email addresses.)
Kevin the Drummer
> else.And trickle the data across the internet to it.
I backup two machines. I mirror them to each other, and then
rsync the full data of each to their own external hard drive.
One machine has its external drive cycled with an off-site copy.
I have not yet sent my data to a storage cloud. I don't consider
most Internet entities to be stable, though maybe some are.
--
PLEASE post a SUMMARY of the answer(s) to your question(s)!
Unless otherwise noted, the statements herein reflect my personal
opinions and not those of any organization with which I may be affiliated.