back-up verification
RichardL
I use cpio to back-up about 21Gb of data overnight. I verify the back-up
using dd to read the tape to /dev/null with a simple
dd if=/dev/rStp0 of=/dev/null
To write the 21Gb data to tape takes about 3.5 hours (so I get about 1.75Mb
/ sec to the DLT - about what I would expect). The "dd" verification takes
about double this using a blocking factor of 80b and about an hour less if I
use no blocking factor at all.
I wondered if anyone could enlighten me as to how I might use blocking
factors to increase throughput of the data in the dd verification process if
at all possible.
It's SCO 5.0.6 running on a Compaq(HP) DL350, Dual Processor, 20/40Gb DLT
drive (SCSI) and Compaq EFS 5.48.
Cheers
Richard
Bill Vermillion
RichardL <no...@jose.com> wrote:
>I use cpio to back-up about 21Gb of data overnight. I verify the
>back-up using dd to read the tape to /dev/null with a simple
>dd if=/dev/rStp0 of=/dev/null
All you are doing is verifying that the tape is readable. If data
was corrupted between the disk drive and the time it arrived at the
tape interface, the crc written to tape will be computed against
the corrupt data. That really is no verify at all.
>To write the 21Gb data to tape takes about 3.5 hours (so I get
>about 1.75Mb / sec to the DLT - about what I would expect). The
>"dd" verification takes about double this using a blocking factor
>of 80b and about an hour less if I use no blocking factor at all.
You have the system - you can try other blocking sizes.
>I wondered if anyone could enlighten me as to how I might use
>blocking factors to increase throughput of the data in the dd
>verification process if at all possible.
If you want true file verifcation use one of the supertar products.
--
Bill Vermillion - bv @ wjv . com
Fabio Giannotti
> -----Original Message-----
> From: listm...@xenitec.on.ca [mailto:listm...@xenitec.on.ca]On
> Behalf Of RichardL
> Sent: Thursday, December 11, 2003 9:19 AM
> To: distri...@xenitec.on.ca
> Subject: back-up verification
>
>
> Hi,
>
> I use cpio to back-up about 21Gb of data overnight. I verify
> the back-up
> using dd to read the tape to /dev/null with a simple
>
> dd if=/dev/rStp0 of=/dev/null
>
Hi Richard,
I don't see how the above "dd" command verifies anything. (Except perhaps
that the tape is readable.)
If your data is at all important to you, use LoneTar or BackupEdge or Ctar.
They actually do byte-by-byte verification.
Fabio
Stuart J. Browne
"RichardL" <no...@jose.com> wrote in message
news:bra1sg$13m$1$8300...@news.demon.co.uk...
Apart from the fact that this isn't really doing anything useful in terms
of verification (you can read data? right.. but is the data useful?), it's
because the block size you are using is tiny.
You've also not said what you are using to *DO* the backup, or what command
line you are using, which doesn't help us in giving you decent block values
to use.
Myself, and most (if not all) of the other people here will recommend that
you drop your un-named custom written backup routines, and pick up one of
the Super-Tar packages (Backup Edge, Lone Tar, etc.).
These packages are fantastic, have excellent recovery procedures, and best
of all (for you) decent verify routines! (I use backup edge, and it's
bit-level verify is very nice, and simple to use).
bkx
John DuBois
>using dd to read the tape to /dev/null with a simple
>
>dd if=/dev/rStp0 of=/dev/null
>
>To write the 21Gb data to tape takes about 3.5 hours (so I get about 1.75Mb
>/ sec to the DLT - about what I would expect). The "dd" verification takes
>about double this using a blocking factor of 80b and about an hour less if I
>use no blocking factor at all.
>
>I wondered if anyone could enlighten me as to how I might use blocking
>factors to increase throughput of the data in the dd verification process if
>at all possible.
I'm not sure what you're asking. The obvious thing to do is just try a larger
block size. With my DLT drive, I use a read/write block size of 100MiB and a
tape block size of 128KiB.
John
--
John DuBois spc...@armory.com KC6QKZ/AE http://www.armory.com/~spcecdt/
RichardL
I provide support to a user base for whom over many years we have developed
a very simple menu based, system administration system that suits them. I
try to add value to my companies offerings by doing this without increasing
the cost of their purchase. This product set is, unfortunately, at end of
life. I will not be able to sell or even implement any new administration
systems - even simple back-up systems.
I have only 1 customer for whom this amount of data is an issue.
I only wondered if the utilisation of blocking factors might help me and I
must admit I have little understanding of how such things work, only that
many moons ago a [Xenix] guru told me that blocking factors improved the
performance of "dd". I don't mind experimenting on this if I have some
reasonable figures on which to work.
To backup it's a simple:
find / -depth -print | cpio -ocvB >/dev/rStp0
John - thanks for your input, I will experiment with settings about these
sizes.
Regards
Richard
Bob Bailin
I'm sorry, I just don't understand this logic.
You do charge your user base for your support services, don't you?
You do believe you're providing some sort of backup procedure for
them that protects their investment in case of disaster or simple
hardware failure?
They do believe they're protected by performing these backups on
a regular basis?
Well, to answer these last 2 q's for you: you're not and they're not.
We're talking about a one-time US$200-300 investment in software,
plus the cost of your time to install it. You've probably spent more
time on this issue already than it would have taken to install a
supertar product.
Adding this program to your existing administration menu is as
easy as substituting the "find ... | cpio" command with "edgemenu"
(in the case of Microlite's BackupEdge with which I'm most familiar).
Alternatively, you can set up the whole process as a cron job, so
that all they have to do is change tapes daily.
Any other action on your part is ultimately a waste of time (and money).
Just my two cents.
Bill Vermillion
>I use cpio to back-up about 21Gb of data overnight. I verify the
>back-up using dd to read the tape to /dev/null with a simple
>dd if=/dev/rStp0 of=/dev/null
>To write the 21Gb data to tape takes about 3.5 hours (so I get
>about 1.75Mb / sec to the DLT - about what I would expect). The
>"dd" verification takes about double this using a blocking factor
>of 80b and about an hour less if I use no blocking factor at all.
That seems wafully slow for a DLT - what model is this and how old
is it. However - before we blame the tape drive run some
performance tests on your hard drive and see what it's real
throughput is. It >might< be the limiting factor. If it can't
read the data fast enough that could be it.
I say that because I just checked logs on a Linux system
using Lone-Tar with a VXA on an IDE controller and it shows
that it backuped up 37.3GB in 1 hour, 10 minutes and 58 seconds.
>I wondered if anyone could enlighten me as to how I might use
>blocking factors to increase throughput of the data in the dd
>verification process if at all possible.
>It's SCO 5.0.6 running on a Compaq(HP) DL350, Dual Processor,
>20/40Gb DLT drive (SCSI) and Compaq EFS 5.48.
Model number of the DLT might help. Have you ever considered
a commercial application. The SuperTars - BackupEdge and LoneTar
have free demos?
The problem with taking 3.5 hours to write and a dd verification
at twice that mean you put 10 hours wear on your drive every night.
That will surely shorten your expected drive lifetime.
John DuBois
RichardL <no...@jose.com> wrote:
>I only wondered if the utilisation of blocking factors might help me and I
>must admit I have little understanding of how such things work, only that
>many moons ago a [Xenix] guru told me that blocking factors improved the
>performance of "dd". I don't mind experimenting on this if I have some
>reasonable figures on which to work.
>
>To backup it's a simple:
>
>find / -depth -print | cpio -ocvB >/dev/rStp0
>
>John - thanks for your input, I will experiment with settings about these
>sizes.
Just remember that the read/write (dd) block size is independent of the tape
block size (unless you're using variable tape block size, which is a bad idea),
and that it's the tape block size in effect when you *write* that determines
the size of the tape blocks you read.
I found that write speed varied by more than 3:1 with tape block sizes from
8192 to 131072. I didn't do read speed tests.
RichardL
>
> You do charge your user base for your support services, don't you?
>
> You do believe you're providing some sort of backup procedure for
> them that protects their investment in case of disaster or simple
> hardware failure?
>
> They do believe they're protected by performing these backups on
> a regular basis?
>
> Well, to answer these last 2 q's for you: you're not and they're not.
>
> We're talking about a one-time US$200-300 investment in software,
> plus the cost of your time to install it. You've probably spent more
> time on this issue already than it would have taken to install a
> supertar product.
>
> Adding this program to your existing administration menu is as
> easy as substituting the "find ... | cpio" command with "edgemenu"
> (in the case of Microlite's BackupEdge with which I'm most familiar).
> Alternatively, you can set up the whole process as a cron job, so
> that all they have to do is change tapes daily.
>
> Any other action on your part is ultimately a waste of time (and money).
>
> Just my two cents.
>
> >
Richard