Problems with backups/restores
Erhard Wolloner
i40. The backups were done on a HP NetServer LH Pro with a HP SureStore i24.
Both machines have OpenServer 5.0.5, and both devices are used in
compressing mode.
The problem is that some files, which were backed up with the i24 and are
restored with the i40, do not have the same file size any more (of course
also their sum is also different). Obviously their way of compressing is
different, but I have never heard of such an incompatibility of tapes from
one and the same manufacturer.
I would be grateful for any kind of hint.
Jeff Hyman
1. If you backup on the 1st tape drive file named "FILE",
then restore the same file to a different directory on the
"same" CPU using the "same" tape drive... do the sums match?
# cd /orig
# tar cvfb /dev/rStp0 20 FILE
# cd /tmp
# tar xvfb /dev/rStp0 20
# sum -r /orig/FILE /tmp/FILE
2. Repeat #1 on the 2nd CPU. Same results?
3. # tape getcomp /dev/rStp0 on CPU 1
# tape getcomp /dev/rStp0 on CPU 2
Both have hardware compression ON?
4. # tape -a 0 setcomp /dev/rStp0 on CPU 1
# tape -a 0 setcomp /dev/rStp0 on CPU 2
By turning OFF hardware compression, does the problem go away?
5. What is the Man/Model of the actual 4mm tape?
Best Regards,
Jeff Hyman, President
.--.
__________________________ .-. | | __________________________________
Lone Star Software Corp. | | | | .-. E-Mail: je...@cactus.com
Cactus International, Inc. | |_| | | | Sales: 800.525-8649
509 E. Ridgeville Blvd. _ |___ |_| | Support: 301.829-1622
Mt. Airy, MD 21771 _| ~- | ___| Fax: 301.829-1623
http://www.CACTUS.com \, _} | | http://www.LONE-TAR.com
------------------------- \( -- | | -----------------------------------
| |
Bill Vermillion
<Woll...@t-online.de> wrote:
No - that is not obvious. In compatible data compression in the DDS
area went away about the time the name was changed from DAT to DDS.
Have you performed a diff [if text] or a cmp [if binary] to verify
what the problem is.
You did NOT mention the programs used to back up or restore.
Could they have been different. Could someone have use dd with
different block factors or options to pad out the blocks.
Until you compare the files it is not ovbvious what the problem is.
[I don't want to belabor that point but you only mentioned file
size differences].
Could one have been using HW compresson and the other using SW.
I have seen some programs store compressed files on tape with no
extension to them indicating compression, as there was an
assumption the files were compressed. A file <filename> showed
what I thought was an uncompressed file was actually compressed.
Running an uncompression program manually - of course with a rename
to avoid attempting to save a file with the same name as the
compressed file - revealed the file to be perfectly fine.
So beside the tape drives you used could you mention the program
and also verify that the options used for saving both were the
same?
--
Bill Vermillion - bv @ wjv . com
Joe DeBiso
front, an index at the end with "air" in the middle. Most backup software
will null fill the "air" making the restored file bigger. Could this be
your issue? I know BackupEdge will deal with this problem.
Hope that helps!
Joe DeBiso
"Erhard Wolloner" <Woll...@t-online.de> wrote in message
news:9rdoqi$ui0$00$1...@news.t-online.com...
-----= Posted via Newsfeeds.Com, Uncensored Usenet News =-----
http://www.newsfeeds.com - The #1 Newsgroup Service in the World!
Check out our new Unlimited Server. No Download or Time Limits!
-----== Over 80,000 Newsgroups - 19 Different Servers! ==-----
Jean-Pierre Radley
| There is a concept called "Sparse Files". his is where a file has data in
| front, an index at the end with "air" in the middle. Most backup software
| will null fill the "air" making the restored file bigger. Could this be
| your issue? I know BackupEdge will deal with this problem.
"index at the end"? I don't think so.
The blocks actually in use are, as usual, listed in the inode table.
--
JP
Tony Lawrence
>
> There is a concept called "Sparse Files". his is where a file has data in
> front, an index at the end with "air" in the middle. Most backup software
> will null fill the "air" making the restored file bigger.
Wow, that's the worst explanation of sparse files I've ever seen.
Sparse files were actually a space saving "trick" introduced in Unix
filesystems a long, long time ago. The need sprang from hashed files,
which is probably where "index" got into your muddled explanation.
A hash is an access/storage method where a mathematical function is
applied to a key. The number that results from that function is used as
the record number, or offset of the file. For example, suppose we had
the keys mary and tom (with associated data, of course), Our hash
function turns the word "mary" into the number 45, and turns "tom" into
128. Pretending that the data stored is 512 bytes for each record,
you'd find mary's data 512 * 45 bytes from the start of the file, and
tom's at 512 * 128 bytes. This sort of "indexing" with hashed keys
gives incredibly fast access to records (there are issues with how to
deal with keys that hash to the same value, biut we'll ignore that
here).
A good hash function is going to generate widely disparate numbers
(that's one of the ways to minimize the duplication problem). So rather
than 45 and 128, we'd really get something like 2 and 438,785. Now
suppose that these were the only data stored in the file so far: it
would be a pretty big file, over 200 megabytes (433,785 * 512), but
there's really only 1024 bytes of real data in it- a whole bunch of
wasted space.
Now we turn to the way Unix file systems work. Without getting into too
much detail, and without getting too much into the confusion of
indirect, double indirect etc here, the Unix inode has pointers to the
places where a files data can be found. The first ten pointers point
directly to data blocks, the next points to indirect blocks which in
turn point to real data blocks and so on.
So, the "mary" data ends up in the second data block (assuming 1k blocks
here) and the "tom" data ends way out in one of the double indirect
blocks somewehre. None of the other pointers are being used. No data
needs to be stored, so no need to waste space: this is a sparse file.
If you look at it with "ls -l" it looks like it's 200+ MB, but if you
removed it, you wouldn't gain 200 MB of space. If you do something that
reads it sequentially, the driver just returns nulls for the data that
isn't there. And that's the problem: ordinary tape utilities write
those nulls, using up 200+ MB of tape, and if it is restored with the
same non-aware utility, the data blocks actually get allocated and
filled with ascii 0's- now you really have used 200 MB of space.
The Supertars are smarter than this and do not write or restore the
nulls.
--
Tony Lawrence
SCO/Linux Support Tips, How-To's, Tests and more: http://pcunix.com
Bill Vermillion
>> There is a concept called "Sparse Files". his is where a file has
>> data in front, an index at the end with "air" in the middle. Most
>> backup software will null fill the "air" making the restored file
>> bigger.
>Wow, that's the worst explanation of sparse files I've ever seen.
Agreed.
>Sparse files were actually a space saving "trick" introduced in Unix
>filesystems a long, long time ago. The need sprang from hashed files,
>which is probably where "index" got into your muddled explanation.
I don't know if it was a trick but it really is just the Unix way
of doing only what is needed. And I suspect it may have come
before the concept of hashed files - normally associated with
database functions.
>A hash is an access/storage method where a mathematical function
>is applied to a key. The number that results from that function is
>used as the record number, or offset of the file.
But sparse files don't have to be assocaited with hashed files - at
least not as I understand them - so feel free to point me in the
proper direction if I'm wrong.
>For example, suppose we had
>the keys mary and tom (with associated data, of course), Our hash
>function turns the word "mary" into the number 45, and turns "tom" into
>128. Pretending that the data stored is 512 bytes for each record,
>you'd find mary's data 512 * 45 bytes from the start of the file, and
>tom's at 512 * 128 bytes. This sort of "indexing" with hashed keys
>gives incredibly fast access to records (there are issues with how to
>deal with keys that hash to the same value, biut we'll ignore that
>here).
Let's take something like a file created with something as simple
as a BASIC program [yes - you used to see that on Unix systems].
Using randon I/O - and for easy of math computation lets say the
data is a 100 byte record. Write record number 1 and you start at
the front. Then just call the identify the second record as record
number 1000. The system will write blocks that logically map to
a position about 1 million bytes down the file but in reality
the blocks used for the data could be adjacent to the data written
for block one. First record will take block of data [typical will
be 1024 bytes as most times two physical 512 byte blocks are
allocated at once, so two physical blocks are used]. The second
record will have to allocate data and you may use 2 or 4 physical
blocks - depending on whether that record spans a logical 1024 byte
block or not.
So you have used between 4 and 6 phsyical blocks, or 1 to 3 logical
blocks, and they may all be adjacent on the hard drive. A simple
c program will also create this - so the hashing doesn't come into
play at all - at least not as I see it.
> A good hash function is going to generate widely disparate numbers
>(that's one of the ways to minimize the duplication problem).
>So rather than 45 and 128, we'd really get something like 2 and
>438,785. Now suppose that these were the only data stored in the
>file so far: it would be a pretty big file, over 200 megabytes
>(433,785 * 512), but there's really only 1024 bytes of real data in
>it- a whole bunch of wasted space.
That is correct and this assumes something like a database as good
design requires this. I was only commenting on the fact that
the hashing method is not the only way to create sparse files.
>Now we turn to the way Unix file systems work. Without getting into
>too much detail, and without getting too much into the confusion of
>indirect, double indirect etc here, the Unix inode has pointers to
>the places where a files data can be found. The first ten pointers
>point directly to data blocks, the next points to indirect blocks
>which in turn point to real data blocks and so on.
It is a confusing subject isn't it.
I'll leave the rest as is because its a good explanation of a
confusing subject.
BDS
"Bill Vermillion" <bi...@wjv.com> wrote in message
news:GM2uL...@wjv.com...
<snip> Let's take something like a file created with something as simple
> as a BASIC program [yes - you used to see that on Unix systems].
>
Whaddya mean "used to"? :>)
-BDS
Tony Lawrence
> But sparse files don't have to be assocaited with hashed files - at
> least not as I understand them - so feel free to point me in the
> proper direction if I'm wrong.
.. stuff deleted..
> Let's take something like a file created with something as simple
> as a BASIC program [yes - you used to see that on Unix systems].
>
> Using randon I/O - and for easy of math computation lets say the
> data is a 100 byte record. Write record number 1 and you start at
> the front. Then just call the identify the second record as record
> number 1000.
But there would be absolutely no reason whatsoever for you to choose
1000 as the position of the second record *unless* you were using a
hashing algorithm. Logically, the second record would be, well, the
second record.
> That is correct and this assumes something like a database as good
> design requires this. I was only commenting on the fact that
> the hashing method is not the only way to create sparse files.
No, but it's generally the only logical reason one would arbitrarily
jump out to some high byte position. Why else would you do such a
thing? Maybe I'm just too tired (I had a 22 hour day a couple of days
back), but I can't think of any other reason.
The purpose of hashing is ultrafast access- no other access method can
even begin to approach it for random keys- it's really not a "database"
thing (unless you consider any access to data a database), Back in the
days of slow media I wrote programs that used hashed storage for no
other reason than that- it let me get my data back quickly.
A side effect of hashing is the sparse file if the file system supports
it. I really cannot think of any other reason that one would create
data in this manner.. someone less burned out than I can feel free to
remind me of such a reason..
I suppose there is the case (which I've certainly used where I could)
where some sequentially assigned number is part of the data- in that
case you can use that number in the same manner as a hash, and it is, in
effect, a degenerate hash- the function is just times 1 or plus zero-
but to my mind that's a hash even though you don't actually do any
math.. purists can disagree, of course, but I ignore purists anyway :-)
I'm pretty sure that the rationale for sparse files was hashing- I'm not
sure whether the driving force was efficiency in creating and reading or
to save space, but certainly both space and media speed were much more
critical then than now.