I have a problem and a question related to tapes.
The problem first. I have a big tgz file which I would like to
write to a tape. The file was created on a different box, and it
was transferred on the network to the box which has the tape
drive. The tape box doesn't have enough disk space to untgz the
file. So I tried to copy the tgz file to the tape by doing a
cat file.tgz > $TAPE
It didn't work. It wrote 1865 bytes less (the file is over 2GB) and
complained about a write error. Next I tried dd:
dd if=file.tgz of=$TAPE bs=1M count=3000
It didn't work either. It wrote over 800KB less and complained of
"Invalid argument".
I tried different tapes (the tapes are good) and two different drives.
Vendor: EXABYTE Model: EXB-85058HE-0000 Rev: 0112
Type: Sequential-Access ANSI SCSI revision: 02
Vendor: EXABYTE Model: EXB-85058SQANXR1 Rev: 0808
Type: Sequential-Access ANSI SCSI revision: 02
Here is the dmesg of the scsi card:
scsi1 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.1
<Adaptec 2940 SCSI adapter>
aic7870: Single Channel A, SCSI Id=7, 16/255 SCBs
Anybody knows why catting or dd-ing to a tape drive doesn't write
out all the data?
And now the question. How can I go to the end of a tape? The mt
command has the "seek" subcommand, but it goes only as far as the
end of the last file. How can I go *beyond* that?
Thanks, Vilmos
Supposing your tape is /dev/st0 you do
tar -c /dev/st0 file.tgz
Davide
> Supposing your tape is /dev/st0 you do
>
> tar -c /dev/st0 file.tgz
>
> Davide
The problem with this is that it would create a tgz inside a tar.
This is not what I want.
Thanks, Vilmos
No reason that you shouldn't be able to do that. Can you in fact tar or
cpio to the drive?
>
> And now the question. How can I go to the end of a tape? The mt
> command has the "seek" subcommand, but it goes only as far as the
> end of the last file. How can I go *beyond* that?
You can't go beyond the eot mark. The tape hardware just will not.
>
> Thanks, Vilmos
--
Tony Lawrence
Free SCO, Mac OS X and Linux Skills Tests:
http://aplawrence.com/skillstest.html
> I have a problem and a question related to tapes.
...
> cat file.tgz > $TAPE
> It didn't work. It wrote 1865 bytes less (the file is over 2GB) and
> complained about a write error. Next I tried dd:
> dd if=file.tgz of=$TAPE bs=1M count=3000
That can't work, a tape is a character device, take a look at the
permissions. You can't access a tape randomly.
You need to use tar/cpio to write/read from it, you can even record
the blocks while "taring" to SCSI tape and later use mt to seek fast
to some file, restoring it.
Michael Heiming
--
Remove +SIGNS and www. if you expect an answer, sorry for
inconvenience, but I get tons of SPAM
Huh? I must be missing something here. Where is he trying to access it
randomly? He's just sending a byte stream; what's wrong with that?
Writing to a tape with dd is in fact common enough that it's often used
in faqs as an example of what NOT to do (tar to a file, compress that,
dd to tape).
And you can find examples of using tapes in "dd" man pages:
http://www.gnu.org/manual/fileutils-4.1/html_node/fileutils_39.html for
example.
But his blocking could be screwing him up..
Tapes have a block size. If you try to dd with a bs= that's different
from the tape drive's block size, it won't work right. You can
typically change the block size with "mt -f /dev/$TAPE setblk $BYTES".
If a tape has already been written, you should figure out the block size
before trying to read it. GNU tar (or cpio, afio, BRU, etcetera) should
do that automagically.
--
Matt G|There is no Darkness in Eternity/But only Light too dim for us to see
Brainbench MVP for Linux Admin /
http://www.brainbench.com / "He is a rhythmic movement of the
-----------------------------/ penguins, is Tux." --MegaHAL
...
>> That can't work, a tape is a character device, take a look at the
>> permissions. You can't access a tape randomly.
> Huh? I must be missing something here. Where is he trying to access it
> randomly? He's just sending a byte stream; what's wrong with that?
Sure he isn't, in this case. Just to point out, that he should take a look
at the permissions of his tape device...
...
> And you can find examples of using tapes in "dd" man pages:
> http://www.gnu.org/manual/fileutils-4.1/html_node/fileutils_39.html for
> example.
> But his blocking could be screwing him up..
Perhaps, never used dd to write to tape and I wouldn't suggest using it,
even if it looks from the URL as if it should work.;)
Correct. I keep forgetting which OSes default to variable tape block
and which don't.
What can permissions have to do with failing AFTER writing x many bytes?
As I said, it could be block size, but that's why I asked if he can in
fact tar to it.
>
>
>>And you can find examples of using tapes in "dd" man pages:
>>http://www.gnu.org/manual/fileutils-4.1/html_node/fileutils_39.html for
>>example.
>
>
>>But his blocking could be screwing him up..
>
>
> Perhaps, never used dd to write to tape and I wouldn't suggest using it,
> even if it looks from the URL as if it should work.;)
There's nothing wrong with using dd to write tape. Again, other than
the possible problem with his block size.
Sorry, if this wan't obvious, I wanted to point out that a tape is a
character device, which can be seen by looking at the permissions.
> As I said, it could be block size, but that's why I asked if he can in
> fact tar to it.
Yes, he'd better use tar.;)
> There's nothing wrong with using dd to write tape. Again, other than
> the possible problem with his block size.
Wouldn't suggest using it, tar has many advantages over using dd to
write to a tape.
Michael Heiming
To which FAQs are you referring, and what's wrong with a snippet like
this anyway?
cd / && find . {some args} |
egrep {some filter} |
cpio -ocB |
gzip |
dd ibs=1K obs=1024K of=$TAPE
Cheers,
Chris
--
@s=split(//,"Je,\nhn ersloak rcet thuarP");$k=$l=@s;for(;$k;$k--){$i=($i+1)%$l
until$s[$i];$c=$s[$i];print$c;undef$s[$i];$i=($i+(ord$c))%$l}
What's wrong is that you've compressed the the whole archive and then
written out to a media that can have bit errors. If there is a problem,
you lose the entire archive. That's the warning that you'll find in the
aforementioned faqs. There's also the obvious limitation that you can't
grab a particular file from that tape.
Not that people don't do this :-). If you understand the risks and
limitations and still want to do it, go right ahead.
>>There's nothing wrong with using dd to write tape. Again, other than
>>the possible problem with his block size.
>
>
> Wouldn't suggest using it, tar has many advantages over using dd to
> write to a tape.
Well, yeah, but in fact a very common use is this:
tar cvf /tmp/mytar.tar
dd if=/tmp/mytar.tar of=/dev/tape
This is often when you want to make multiple tapes of the same tar data;
do the tar once, dd it as many times as you like. The resultant tapes
can be extracted with tar.
Another very common use is across networks
tar cvf - . | rcmd otherbox dd of=/dev/tape
I couldn't tell you how many times I've done that. Not so often now in
recent years because there are often easier ways, but still..
So "Wouldn't suggest using it" leaves people with the wrong impression,
I think.
> On Tue, 04 Feb 2003 21:54:34 GMT, Tony Lawrence staggered into the Black
> Sun and said:
>> Vilmos Soti wrote:
>> 0: Single Channel A, SCSI Id=7, 16/255 SCBs
>>> Anybody knows why catting or dd-ing to a tape drive doesn't write out
>>> all the data?
>> No reason that you shouldn't be able to do that. Can you in fact tar
>> or cpio to the drive?
>
> Tapes have a block size. If you try to dd with a bs= that's different
> from the tape drive's block size, it won't work right. You can
> typically change the block size with "mt -f /dev/$TAPE setblk $BYTES".
> If a tape has already been written, you should figure out the block size
> before trying to read it. GNU tar (or cpio, afio, BRU, etcetera) should
> do that automagically.
In particular, try it with BYTES=0 (asking for variable block sizes).
The Linux scsi (at least) tape driver discards incomplete blocks when
using fixed block size. So, unless the file you're writing out
is *exactly* N blocks long, you'll lose data.
>> Tapes have a block size. If you try to dd with a bs= that's different
>> from the tape drive's block size, it won't work right. You can
>> typically change the block size with "mt -f /dev/$TAPE setblk $BYTES".
>> If a tape has already been written, you should figure out the block size
>> before trying to read it. GNU tar (or cpio, afio, BRU, etcetera) should
>> do that automagically.
>
> In particular, try it with BYTES=0 (asking for variable block sizes).
> The Linux scsi (at least) tape driver discards incomplete blocks when
> using fixed block size. So, unless the file you're writing out
> is *exactly* N blocks long, you'll lose data.
Why does it discard those blocks? I don't understand why it would
make sense.
BTW I tried to write to the tape after setting the blocksize to 0
(which is variable), and the writing took 15 hours instead of 1.
I tried to read back, and the speed was like that from a floppy.
The data is a bit over 2GB so I killed the operation. Now I tried
to set the tape blocksize to 16384, set the dd blocksize to 16k,
and it seems there are still problems.
# mt status
SCSI 2 tape drive:
File number=0, block number=0, partition=0.
Tape block size 16384 bytes. Density code 0x0 (default).
Soft error count since last status=0
General status bits on (41010000):
BOT ONLINE IM_REP_EN
# mt tell
At block 0.
# mt rewind
# dd if=output.tgz of=$TAPE bs=16k
dd: writing `/dev/siteread/EXABYTE': Invalid argument
142897+1 records in
142897+0 records out
#
A question. Anyone has a good document which describes the intricacies
of tapes? Like block size, partitions, densities, etc. I have a
good grasp of them (or at least I hope...), but I would like to
read a good paper about it.
Thanks for everyone's help, Vilmos
> Yes, he'd better use tar.;)
>
>> There's nothing wrong with using dd to write tape. Again, other than
>> the possible problem with his block size.
>
> Wouldn't suggest using it, tar has many advantages over using dd to
> write to a tape.
This is a one time backup. The archive was created on a machine,
and I transferred to another box which has 8mm drives. The problem
is that this tape box doesn't have enough disk space to untgz
the whole archive. Also, I have to write multiple copies. This
is why I chose to create the archive on the box where the data
resides (and has plenty of disk space) and transfer only once
to the tape box and transfer it to multiple tapes.
Vilmos
> This is a one time backup. The archive was created on a machine,
> and I transferred to another box which has 8mm drives. The problem
> is that this tape box doesn't have enough disk space to untgz
> the whole archive. Also, I have to write multiple copies. This
> is why I chose to create the archive on the box where the data
> resides (and has plenty of disk space) and transfer only once
> to the tape box and transfer it to multiple tapes.
Perhaps I don't get it, but you don't need to use any compression
with tar, it works perfectly without, you can even get a single
file out of zillions from a tape.
> tar cvf - . | rcmd otherbox dd of=/dev/tape
Works fine with tar.;)
$ tar -cvb 128 . | ssh host "cat - > /dev/tape"
> So "Wouldn't suggest using it" leaves people with the wrong impression,
> I think.
Just my experience, tar simply works on every *nix system and I like
suggesting things that I'm using and actually work.
> Tony Lawrence <to...@pcunix.com> wrote:
> ...
>> What can permissions have to do with failing AFTER writing x many bytes?
>
> Sorry, if this wan't obvious, I wanted to point out that a tape is a
> character device, which can be seen by looking at the permissions.
>
>> As I said, it could be block size, but that's why I asked if he can in
>> fact tar to it.
>
> Yes, he'd better use tar.;)
>
>> There's nothing wrong with using dd to write tape. Again, other than
>> the possible problem with his block size.
>
> Wouldn't suggest using it, tar has many advantages over using dd to
> write to a tape.
>
> Michael Heiming
mt -f /dev/st0 setblk 0
dd if=file.tgz of=/dev/st0 bs=10240
tar tvfz /dev/st0
This would effectively be the same as
mt -f /dev/st0 setblk 0
tar cvfz /dev/st0 /original_paths_tarred
tar tvfz /dev/st0
If you wanted to be pedantic, you would set the tape's block size to 10240.
But since tar likely does not change s the tape drive's blocking factor from
the (usually) default value of 'variable', I see no reason to change it
manually. Also, by setting the default blocksize to 'variable' (zero), it
ensures the tape can be fairly easily read by other systems.
I've been using dd for years reading and writing SCSI tapes with no
problems. Media errors are so rare, I don't recall any. I've dd'ed
zip files, cpio archives, tar archives, plain files and other files
to and from tape drives with no problems.
The only time I had problems was reading, on BeOS, a tape written by
Solaris 2. And the problem was the default blocksize on the SCSI tape
driver on BeOS, which was non-zero.
In fact, this is precisely how we used to make bootable tapes for
Motorola VME systems. The first file on the tape was the kernel,
the second file was the initial RO file system, and the rest of the files
were the archives to be installed.
Fest3er
ne...@roaima.freeserve.co.uk asked:
> To which FAQs are you referring, and what's wrong with a snippet like
> this anyway?
> cd / && find . {some args} |
> egrep {some filter} |
> cpio -ocB |
> gzip |
> dd ibs=1K obs=1024K of=$TAPE
Tony Lawrence <to...@pcunix.com> wrote:
> What's wrong is that you've compressed the the whole archive and then
> written out to a media that can have bit errors. If there is a problem,
> you lose the entire archive.
Yes, agreed. So I always but always do a cpio read to verify that the
tape is readable. (I use this to generate an online listing of the tape,
so it has its benefits, too.)
> That's the warning that you'll find in the
> aforementioned faqs.
Can you cite a reference, please? Prior to my last post I went googling
but didn't find anything terribly obvious.
> There's also the obvious limitation that you can't
> grab a particular file from that tape.
Huh?
cd / && dd ibs=1024K if=$TAPE |
zcat |
cpio -idcvumB './some/file/to/restore'
cd / && dd ibs=1024K if=$TAPE |
zcat |
cpio -idcvumB './some/directory/to/restore/*'
No, I can't find anything. Most FAQS aren't well indexed; Google sees
the hit, but whatever you want is buried deep within.
>
>
>>There's also the obvious limitation that you can't
>>grab a particular file from that tape.
>
>
> Huh?
>
> cd / && dd ibs=1024K if=$TAPE |
> zcat |
> cpio -idcvumB './some/file/to/restore'
>
> cd / && dd ibs=1024K if=$TAPE |
> zcat |
> cpio -idcvumB './some/directory/to/restore/*'
>
> Cheers,
> Chris
Right: but you have to read the whole tape in and uncompress it. The
fact that you are doing it in a pipe doesn't change anything.
If this were an ordinary tar or cpio archive, the extraction algorithm
looks at headers, sees that it isn't the file it wants, picks up the
pointer to the next header, blissfully ignores that many bytes, and then
checks again.
The generally cited problem (aside from the bit errors) is that a dd
doesn't necessarily match the block size used on the tape, which means
that you will read back more than you put out there. That will be null
padded of course, but your uncompression can get screwed up by those
extra bytes. There are, of course, ways to deal with that, too.
I dunno: personally, I'd rather compress individual files on their way
out to tape. Obviously compressing individually lets you take advantage
of indexing to do fast seeks and restores, so that's a disadvantage of
dd'ing a compressed archive, but I'm not ready to go to the mat over it.
I think it's relatively unimportant: I mean it's not like you were
wanting to use emacs instead of vi :-)
[ne...@roaima.freeserve.co.uk replied with an example showing how this
should be achieved]
> Right: but you have to read the whole tape in and uncompress it. The
> fact that you are doing it in a pipe doesn't change anything.
ISTM that the overhead here is only the uncompression process.
> If this were an ordinary tar or cpio archive, the extraction algorithm
> looks at headers, sees that it isn't the file it wants, picks up the
> pointer to the next header, blissfully ignores that many bytes, and then
> checks again.
If it were a disk file, I'd agree that seeking is much faster than
reading data and ignoring it. I remain to be convinced that a seek is
any faster than corresponding reads when using tape I/O [1].
Cheers,
Chris
[1] Ignoring, of course, hardware generated EOF/EOT tape markers
But you have to READ the entire tape for that to happen.
>
>
>>If this were an ordinary tar or cpio archive, the extraction algorithm
>>looks at headers, sees that it isn't the file it wants, picks up the
>>pointer to the next header, blissfully ignores that many bytes, and then
>>checks again.
>
>
> If it were a disk file, I'd agree that seeking is much faster than
> reading data and ignoring it. I remain to be convinced that a seek is
> any faster than corresponding reads when using tape I/O [1].
Many tape drives support what they now call "fast seek", and you can, in
fact, skip by unwanted bytes much more quickly than reading them. It's
not a matter of "convincing" you, it's a hardware feature that you
probably have if your drive isn't very old. A Google of "tape fast
seek" will lead you to relevant material.
This lets you, for instance, restore any specific file from the middle
of a tape very, very quickly. It's real, it works, and you'd love it :-)
>> Sure he isn't, in this case. Just to point out, that he should take
>> a look
>> at the permissions of his tape device... ...
>
> What can permissions have to do with failing AFTER writing x many bytes?
>
> As I said, it could be block size, but that's why I asked if he can in
> fact tar to it.
Yes, it was the problem (I am the OP). Steve Wampler provided the
correct answer in the pan.2003.02.05....@noao.edu
message. I set the tape blocksize to zero (variable) and life
is shiny again.
Thanks Steve.
Vilmos
I'd check what block size tar will use on your system by default (man tar)
and then do:
zcat output.tgz | dd bs=Xk of=$TAPE
which should have the effect of giving you a tape which tar can then
read without assistance.
HTH
John
--
The Linux Emporium - the source for Linux in the UK
See http://www.linuxemporium.co.uk/
We had a woodhenge here once but it rotted.
Provided the block size is acceptable. I've had tapes produced like
that which cause tar to crash and need to be read with:
dd if=/dev/tape | tar xvf -
If you check what block size tar is expecting and specify that to dd
when writing to tape the problem is avoided.
>> BTW I tried to write to the tape after setting the blocksize to 0
>> (which is variable), and the writing took 15 hours instead of 1.
>> I tried to read back, and the speed was like that from a floppy.
>> The data is a bit over 2GB so I killed the operation. Now I tried
>> to set the tape blocksize to 16384, set the dd blocksize to 16k,
>> and it seems there are still problems.
>
> I'd check what block size tar will use on your system by default (man tar)
> and then do:
>
> zcat output.tgz | dd bs=Xk of=$TAPE
>
> which should have the effect of giving you a tape which tar can then
> read without assistance.
Finally I could write all the data. I read back in two different
drives, compared to the original one, and everything was fine. Even
the speed was ok, so that 1hour->15hours was maybe caused by something
else.
Vilmos
> We had a woodhenge here once but it rotted.
Did it rot completely?