tar backup
Norbi
I am writing a script to do a backup, in the case the backup failed (bad
tape, or write protected tape, etc etc), then it should warn the operator
that the backup was not successful, and ask if they want to retry or not. I
use error output of tar. If backup was successful, err=0, if there was an
error, err<>0. This works fine in case there is no tape in the drive, or the
tape is bad or if there was a write error... BUT when I put in a write
protected tape the err=0(from tar)!! As if everything worked fine, and the
backup was written to the tape. Why is this happening and how can I fix it?
I am getting the message from the system that write failed, because of
write-protected tape, but still tar thinks everything was ok!
I am running sco open server 5.0.5
any help or advice is much appreciated
thanx
NF
Bill Vermillion
The supertar products handle all this quite well, in addition they
backup the /dev files which standard tar doesn't.
--
Bill Vermillion - bv @ wjv . com
Tony Lawrence
See http://pcunix.com/SCOFAQ/scotec1.html#tar_cpio
http://pcunix.com/Bofcusm/1656.html
http://pcunix.com/Bofcusm/1658.html
http://pcunix.com/Reviews/supertars.html
which might save us from repeating an entire thread again.
--
Please note new phone number: (781) 784-7547
Tony Lawrence
SCO/Linux Support Tips, How-To's, Tests and more: http://pcunix.com
Free Unix/Linux Consultants list: http://pcunix.com/consultants.html
Mike Brown
There is no reasonable way to trust a tar backup without a way of checking
it. This is why all the supertars will read back the tape and do a bit level
comparison with the same file on the hard drive. I have seen cases where
the backup proceeded with no apparent error, but the verify fails.
You could set up the tar command to place a "check" file at the end of the tape,
move the file on the hard disk, then restore that single file and compare it.
The main advice is to download and try a supertar first.
Mike
--
Michael Brown
The Kingsway Group
Bill Vermillion
>> The situation is the following: I am writing a script to do
>> a backup, in the case the backup failed (bad tape, or write
>> protected tape, etc etc), then it should warn the operator
>> that the backup was not successful, and ask if they want
>> to retry or not. I use error output of tar. If backup was
>> successful, err=0, if there was an error, err<>0. This works
>> fine in case there is no tape in the drive, or the tape is bad
>> or if there was a write error... BUT when I put in a write
>> protected tape the err=0(from tar)!! As if everything worked
>> fine, and the backup was written to the tape. Why is this
>> happening and how can I fix it? I am getting the message from
>> the system that write failed, because of write-protected tape,
>> but still tar thinks everything was ok! I am running sco open
>> server 5.0.5
>
>There is no reasonable way to trust a tar backup without a way
>of checking it. This is why all the supertars will read back the
>tape and do a bit level comparison with the same file on the
>hard drive. I have seen cases where the backup proceeded with no
>apparent error, but the verify fails.
>You could set up the tar command to place a "check" file at the
>end of the tape, move the file on the hard disk, then restore
>that single file and compare it.
That wont guarantee anything except that the 'check' file is
correct. You can have data read from a disk, get some corruption
in transisit to the tape drive, and when the data is written to the
tape the checksum will be computed on tape for the corrupted data.
In those instances you will be able to read the tape end to end but
a restore of the corrupted file will proceed as normal but could
fail in use if it's a binary, or have corrupted information it if
it was a data file.
So if the above scenario occured your single 'check' file would
only succeed in giving you a false positive - and that could be
pretty bad.
>The main advice is to download and try a supertar first.
Not first - but as the only way - or find/build your own bit-level
checking mechanism.
Bill
Mike Brown
I agree completely. I suspect the trouble likely comes between
the tape head and the tape, not from the hard drive to the tape
drive. With such high tape density any misalignment causes
problems, which the tape drive will try to correct by ECC or
rereading the tape. Some drives will flash a LED to indicate
it is using ECC for correction. Since the drives do read
after write checking they can mark a section of the tape bad,
then continue on with the backup further down. The end result
is that the drive will continue to heal and retry with no
outward indication until it really can't use that tape
anymore. The supertars tend to help out with this issue
by tracking number of uses, average backup rate and the bit
level verify. If you notice you backup rate decreasing by
10% or more, try a new tape.
The trick with placing a check file at the end assumes a good
tape drive and tape media. I have found it useful when I need
to send someone a tape copy and they do not have a supertar.
Bill Vermillion
>> In article <3DB407D7...@tkg.ca>, Mike Brown <mi...@tkg.ca> wrote:
>> >Norbi wrote:
>> >> The situation is the following: I am writing a script to do
>> >> a backup, in the case the backup failed (bad tape, or write
>> >> protected tape, etc etc), then it should warn the operator
>> >> that the backup was not successful, and ask if they want
>> >> to retry or not. ...
[Lucretia Deletia weild her knife here]
>> >There is no reasonable way to trust a tar backup without a way
>> >of checking it.
[snip snip]
>> I have seen cases where the backup proceeded with no
>> >apparent error, but the verify fails.
>> >You could set up the tar command to place a "check" file at the
>> >end of the tape, ....
>> That wont guarantee anything except that the 'check' file is
>> correct. ...
>I agree completely. I suspect the trouble likely comes between
>the tape head and the tape, not from the hard drive to the tape
>drive.
I don't agree with that. In all my past experiences, broadcast,
recording and computers, I've been working with magnetic tape
longer than many people on this list have been alive.
As CE of a recording studio we were in the $50,000/year column on
purchases from Ampex, Scotch and Agfa [none of which exist any
more]
> With such high tape density any misalignment causes
>problems, which the tape drive will try to correct by ECC or
>rereading the tape.
Today's tape are marvels of engineering with such things as metal
particles that are ceramic coated to resist oxidation and make them
stronger than a pure particle.
The newer machines are pretty much self-aligning.
You can have bad tape - but when a bit does not match on the tape
when compared to the HD I'd bet on some electrical discturbance
between the time it left the HD head and arrived at tape
electronics. I think I mentioned that if this occurs you will get
a proper checksum on tape as it is computer against the corrupt
data. And a tape verify - a read from end to end - will show the
tape is OK.
If the error occured between the head and the tape then the
checksums on tape would not match or you would get a tape read
error.
> Some drives will flash a LED to indicate
>it is using ECC for correction.
And also when they need cleaing.
>Since the drives do read after write checking they can mark
>a section of the tape bad, then continue on with the backup
>further down. The end result is that the drive will continue to
>heal and retry with no outward indication until it really can't
>use that tape anymore.
That's more like a HD than tape drive.
>The supertars tend to help out with this issue
>by tracking number of uses, average backup rate and the bit
>level verify. If you notice you backup rate decreasing by
>10% or more, try a new tape.
The number of passes on tape and a rejection of the tape when NNNN
passes have been made goes back much further than any of the
supertar programs.
If backup rate decreases it can also be caused by a drive that has
become fragmented badly by creation of lots of small files and then
having the files scattered in pieces. That is more likely to
happen on older file system than on modern file systems - many
based on the FFS which does it's best to allocate files all over
the disk to keep this from happening. I say this because slow
backup does not always mean tape problems.
I always like to use something like Iozone 2 [not the 3 version
with it's overwhelming output] to benchmark the hd file syste
read/write when intalling a new system. Then if things slow down
you can run it again. The nice thing about iozone is that it
writes through the file-system and you will see how the fs works in
real life. Many hd benchmarks show only raw throughput. I see
many using dd if=<drive> to find out how fast the system is. But
that only show HW capability and you'll find filesystem throughput
to be just a fraction of that.
On an older system a few years ago I had both an original S51 file
sysem [the old Xenix and early Unix FS] and a modern FFS.
I could get speeds up to 5 to 10 times faster on the FFS side than
I could on the S51 side. This was just two different FSes on the
same drive.
IOW I'm trying to point out that slow backup is not neccesarily an
indication of a tape problem.
>The trick with placing a check file at the end assumes a good
>tape drive and tape media. I have found it useful when I need
>to send someone a tape copy and they do not have a supertar.
Ah - but the supertars can be read on a normal system. The only
gotcha is that if you compress the files before backup on a
supertar they automatically decompress on retreival and on a normal
system you would have to do this yourself.
The only time I've seen an anomoly was several years ago when
I extracted via a normal tar on a Pentium based BSD system that was
created on an SGI Indy - with a MIPS 4440 chip and using LoneTar.
The files appeared to extract properly but upon further examination
they were all compressed files. A rename of the files to give them
a .Z extension and an uncompress fixed that. If you had forced
decompression you would have a namespace collision.
Never investigated that one further but it was interesting. The
'file' command can come in handy at times like that.
Mike Brown
In reference to DAT drives, check out
http://www.seagate.com/support/kb/tape/datmdtch.html
The section on RAW ( Read - After - Write ) is what I was hinting at.
Some of the DAT drives do flash for a tape cleaning, but flash a different
pattern or LED when correcting RAW failures or using ECC for correction.
I have some DAT tapes that 'work', but have bad sections on them. If
you use a utility like Lone Tar's tape sizer you find that the tape
holds less data then spec'd for. A backup to it takes longer because
the drive is rewriting the data. The RAW procedure, I think, makes its
decisions based on analog SNR rather than a hard read error. As the tape
media degrades, you end up with more rewrites, and lower write speeds.
The marketing/engineer who was describing the process claims this to
be a GREAT advantage, because it 'safely' increases the life span of
the media. If you do not use 100% of the tape for the backup, then
the area near the beginning of the tape experiences wear while the end
of the tape may never be reached. By rewriting and 'skipping' over
a few bad sections at the beginning of the tape the backup data gets
moved further on to new area on the tape.
The rough rule of thumb I used at a 10% decrease in backup speed assumed
the problem of disk fragementation was cancelled out because I compare
one night backup from a clients HD to another nights backup from the
same HD. Also, most backups are done at night when there should be
no other programs running to cause different system loads.
The last assumption I make is that the data travelling on the SCSI
bus does not undergo random errors because ( at least on the Compaq
servers I use ) they have parity protection. The data on the drive
is ECC detected and corrected, and the transfer mechanism has
parity or checksums at every point along the way.
In an attempt to test some of these common issues, I have have picked
up tapes from customers and backed up to the tape from one of my
servers. The test directories are on a raid 1+0 array, and are
not fragemented much. In any case the data is available from the
drives at a rate much higher than the tape can receive it. Some
of the tapes reduce the write speeds by over 25%, yet back up and
restore without error ( well, any noticable error ). A size test
also indicates reduced capacity, such as turning off HW compression
and only getting 10.8 Gbytes on a 12 Gbyte tape.
With so much magic going on, I really insist on a product like
Backup Edge.
Any DAT experts out there that can comment on my theories?
Bill Vermillion
>> In article <3DB5A916...@tkg.ca>, Mike Brown <mi...@tkg.ca> wrote:
>> >Bill Vermillion wrote:
....
>> You can have bad tape - but when a bit does not match on the tape
>> when compared to the HD I'd bet on some electrical discturbance
>> between the time it left the HD head and arrived at tape
>> electronics. I think I mentioned that if this occurs you will get
>> a proper checksum on tape as it is computer against the corrupt
>> data. And a tape verify - a read from end to end - will show the
>> tape is OK.
>> If the error occured between the head and the tape then the
>> checksums on tape would not match or you would get a tape read
>> error.
>> > Some drives will flash a LED to indicate
>> >it is using ECC for correction.
>> And also when they need cleaing.
>> The number of passes on tape and a rejection of the tape when NNNN
>> passes have been made goes back much further than any of the
>> supertar programs.
>> If backup rate decreases it can also be caused by a drive that has
>> become fragmented badly by creation of lots of small files and then
>> having the files scattered in pieces. That is more likely to
>> happen on older file system than on modern file systems - many
>> based on the FFS which does it's best to allocate files all over
>> the disk to keep this from happening. I say this because slow
>> backup does not always mean tape problems.
....
>> IOW I'm trying to point out that slow backup is not neccesarily an
>> indication of a tape problem.
>In reference to DAT drives, check out
>http://www.seagate.com/support/kb/tape/datmdtch.html
>
>The section on RAW ( Read - After - Write ) is what I was
>hinting at. Some of the DAT drives do flash for a tape cleaning,
>but flash a different pattern or LED when correcting RAW
>failures or using ECC for correction.
Stopped using Seagate drives a long time ago when Sony came out
with the 7000 series and their higher head-wheel speed made them
out-perform anything on the market. Last Seagate I used was a
Scorpion - 5.25" full-height form fact that took a cartridge with
4 DAT tape to give you 96MB backup. Thanks for that info.
>I have some DAT tapes that 'work', but have bad sections on them. If
>you use a utility like Lone Tar's tape sizer you find that the tape
>holds less data then spec'd for. A backup to it takes longer because
>the drive is rewriting the data. The RAW procedure, I think, makes its
>decisions based on analog SNR rather than a hard read error.
Tapes are cheap enough that if I got an error and a 'tape erase'
followed by a backup failed, I'd junk the tape.
>As the tape media degrades, you end up with more rewrites, and
>lower write speeds. The marketing/engineer who was describing
>the process claims this to be a GREAT advantage, because it
>'safely' increases the life span of the media.
Cheaper is not better when it comes to your data. As I said above
- tapes are cheap.
>If you do not use 100% of the tape for the backup, then the area
>near the beginning of the tape experiences wear while the end
>>of the tape may never be reached. By rewriting and 'skipping'
>over a few bad sections at the beginning of the tape the backup
>data gets moved further on to new area on the tape.
IMO that's false economy.
>The last assumption I make is that the data travelling on the SCSI
>bus does not undergo random errors because ( at least on the Compaq
>servers I use ) they have parity protection. The data on the drive
>is ECC detected and corrected, and the transfer mechanism has
>parity or checksums at every point along the way.
Have not used the Compaqs. I've not seen ECC on the controllers
I've used. Point taken on ECC.
>With so much magic going on, I really insist on a product like
>Backup Edge.
Yup. Never leave home without a supertar.
Mike Brown
--SNIP--
Generally use and resell SONY drives, I have been happy with their
reliability and performance over the years. I end up seeing all
the various drives on the market because HP/Dell/Compaq ship
by specification not manufacture. Despite their origin I think all
the new drives ( 12/24 and 20/40 ) have the same RAW feature,
I just found the Seagate link first.
The forensic testing on the tapes is for my own curiosity mainly,
once a tape or drive starts acting up I recommend replacing it.
The problem has been that with so much 'magic' going on, you
do not know when a tape starts having a problem. That was why
I was interested in transfer speed, and rather than saving money
just replacing the tape. The RAW algorith will rewrite data
to the media as much as 120 times to skip over a bad section,
without indicating the problem ( other than flashing a LED ).
I do not what to be in the position of explaining to a customer
that a critical backup is junk, but I saved you $20 on a tape.
Bill Vermillion
>> Cheaper is not better when it comes to your data. As I said above
>> - tapes are cheap.
>> >If you do not use 100% of the tape for the backup, then the area
>> >near the beginning of the tape experiences wear while the end
>> >>of the tape may never be reached. By rewriting and 'skipping'
>> >over a few bad sections at the beginning of the tape the backup
>> >data gets moved further on to new area on the tape.
>> IMO that's false economy.
....
>The forensic testing on the tapes is for my own curiosity mainly,
>once a tape or drive starts acting up I recommend replacing it.
>The problem has been that with so much 'magic' going on, you
>do not know when a tape starts having a problem. That was why
>I was interested in transfer speed, and rather than saving money
>just replacing the tape. The RAW algorith will rewrite data
>to the media as much as 120 times to skip over a bad section,
>without indicating the problem ( other than flashing a LED ).
>I do not what to be in the position of explaining to a customer
>that a critical backup is junk, but I saved you $20 on a tape.
Agree completely. The DDS-4s are not cheap but on DDS-3 I've put
customers in touch with a place that sold used one-pass DDS tapes.
They are the tapes used by the manufacturers to test each DAT drive
before it is shipped. Think of it as pre-qualified ;-). I audio
on critical sessions we'd sometimes make a complete pass on the
tape to be sure it was as close to perfect as it could be.
We'd not do that often but there was period of time when Ampex
was having a problem where we were about the only studio reporting
the problem. It was low-level noise - sounded like faint rumbling
to the untrained ear.
It was asperity noise and we were returning abovt 50% of all tapes
received for almost 6 months. [List was $250 for 16 min audio at 30
IPS - our cost was about $130 - if you think data tapes are
expensive].
You could hear it on our Studer A-800 [$70K], or the Steven 821A
[about $55K] but you wouldn't notice on the cheap [$30K] MCI.
Ampex even sent a FE to our location and that's what it took to
prove to them we did have legitimate concerns. Up to that point
they assumed it was some problem on our side.
In the end it turns out it was a bad bearing in one of the
calendaring machines at the Opelika plant.
We just kept all our equipment and monitors at 110 per cent :-)
We found it somewhat interesting [and perhaps a sad comentary on
others] that no one else had notice or complained. And
That was a problem with analog in that the medium affected the
message [with apollgies to McLuhan]. You could tell the different
manufacturers of tape just by the sound. Digital elminated all
the media influences, but OTOH once the signal is corrupted it's
gone - while in analog you can usually get something back.
Enough OT for now.