We are toying with the idea of modifying VMS BACKUP
to support tape striping and tape shadowing, what does
it mean?
Tape shadowing:
----------------------
The ability to write two (or more) copies of a tape
saveset concurrently.
Tape striping
------------------------
Backup of large data stores can be substaintially sped
up by writing the save set to multiple tape drives.
Restoring the saveset will require all tapes making the
strip to be loaded into the drives.
What do you think?
I'm looking for feedback like yes this is a good idea or
no it is a waste of time. Please do not ask questions
about implementation as we are in a VERY!!! early
stage of planning and obviously this is not a commitment
that we'll actually do it....
Thank you for your feedback.
Guy Peleg
OpenVMS engineering
When you need a backup for this site as well as off-site storage, this
functionality would be great since with one command, you get 2 identical
tapes, one which can be sent off-site.
Question: would it be possible to have one output to tape and a second
output to disk save set ?
> Backup of large data stores can be substaintially sped
> up by writing the save set to multiple tape drives.
If I understand properly, you'd backup the first half of the disk on
tape 1 and second half of disk on tape 2 and do this concurently ?
There would be issues with truly ensuring that no file is updated during
the process, unless you can garantee that one file will always be backed
up to the same tape in one shot.
Cosnider this scenario:
File has 100 blocks right in middle of disk.
Backup 2 will start its task by writing blocks 50-100 to tape 2 right away.
Some process then adds records to the file, extending it to 125 blocks.
Backup 1 eventually gets to that file towards the end of its task, and
backsup the file header and first 50 blocks.
When you restore it, the file headed indicates it is 125 blocks. You get
file header and 50 blocks from tape1, but will only see 50 blocks in tape2.
> Tape shadowing:
> ----------------------
>
> The ability to write two (or more) copies of a tape
> saveset concurrently.
This would be highly useful, although the same goal might be
achieved for many sites by a supported BACKUP/COPY. The latter
approach has the advantage of susceptibility to being applies
as an afterthought.
> Tape striping
> ------------------------
>
> Backup of large data stores can be substaintially sped
> up by writing the save set to multiple tape drives.
>
> Restoring the saveset will require all tapes making the
> strip to be loaded into the drives.
This would seem extremely hard to implement so that it would be
compatible with the first one, since different output streams
would hit EOT at different times due to vagaries in media and
drives. Even if you could write it successfully, the results
would not be interchangeable, since one could not use tape B17
with tape A16 on the restore, and thus you would need unique
names for the NxM tapes leading to user confusion even if the
software were flawless.
I said no questions about the implementation....;-)
Do not know yet, but should be possible.
>
> > Backup of large data stores can be substaintially sped
> > up by writing the save set to multiple tape drives.
>
> If I understand properly, you'd backup the first half of the disk on
> tape 1 and second half of disk on tape 2 and do this concurently ?
I do not know yet but I envision dividing the work different, in
the case of two tapes, odd VBNs can go to one tape and even
to the other, again we are in the very early stage of thinking about
it. The goal is to speed up the amount of time it takes to backup
the data by parallelism...
Good....it means job security for me ;-)
> would hit EOT at different times due to vagaries in media and
> drives. Even if you could write it successfully, the results
> would not be interchangeable, since one could not use tape B17
> with tape A16 on the restore, and thus you would need unique
> names for the NxM tapes leading to user confusion even if the
> software were flawless.
I assume that we'll have strict set of rules for striping, maybe
all tapes must be identical? not sure yet as I said very early
stages of thinking...
Saveset Manager has the possibility to make a copy of a saveset from
one tape to another, this looks like what Larry indicated with
BACKUP/COPY.
Regards,
Bart Zorn
I know we're not allowed to ask questions but...
When VMS mounts a tape, does it know how much capacity a tape device
actually has ?
If not, it would be hard for it to start to know how to split the backup tasks.
Perhaps the user would need to provide some capacity for each device in
the backup command.
Lets say I have a TK50 and a TK70 on my all mighty microvax II:
BACKUP/IMAGE/NOALIAS mydisk: mua0:/capacity=95meg,mub0:/capacity=170meg
This way, backup would know how to split the workload between the two
drives and where to split the disk.
And one would also assume that backup would be able to ask for a new
tape when one of the tape devices runs out of tape. (this takes care of
a tape being a bit shorter than antitipated, as well as situations where
totakl capacity of the 2 drives is still less than the data to be backed up).
Serriously though, where is backup technology really going ? Is tape
really sticking around ? or will flash eventually take over ? Or will
disk become the prevalent backup medium ?
the shadowing project has immediate and long term value. The need for 2
copies of a backup will always be there, no matter what backup medium is used.
But the striping idea may become moot as backup medium change and backup
becomes a really quick operation.
One thing that would REALLY be neat though is if a new output parameter
were added.
instead of having /SAVE, if we could have /ZIP it would be really nice.
BACKUP/IMAGE dua0: mua0:mysaveset.zip/ZIP
Not only would this compress the output data, but it would also make it
readable on any other platform which has a UNZIP command.
Seems like a good idea to me. (I'm more interested in cutting my backup
window by striping to two drives simultaneously than I am in tape shadowing,
but I don't see shadowing causing me any harm.)
-- Alan
Warning: mostly non-VMS related -- if this is bothersome, please skip
this post.
Tape's still here. They're no longer your father's 9 track reel tapes. :)
Modern tape libraries are pretty nice; 80 MB/sec (compressed) and 900+
GB (compressed) with even faster and higher capacity tape drives on the
horizon.
Yes, the vendors these days are aggressively pushing disk-based
solutions. They do have their place, but much of the incentive for
vendors in pushing it has been for generating new revenue. Natural
response to break out of the slump in IT purchases due to the dot.com
bust and shaky economy earlier in this decade.
Disk-based backups make more sense in circumstances where you need to
get a copy of data off production drives fast as possible, then stream
to slower media. Or where you do large amounts of restores involving a
lot of random access. Or where the allowable time to restore is fairly
small due to internal service level agreements (SLAs).
Modern disks for backups are pretty decent; very good random access time
(as can be expected), more shock-proof (due to 2.5" laptop drive form
factor) than conventional desktop drives, hot plug support, small form
factor, and so forth.
I understand this is popular in financial markets and healthcare due to
demanding availability requirements, but often used as an intermediate
staging area with an additional backup to tape as extra insurance.
They also make sense in a number of other circumstances. Yet, tape isn't
quite dead yet, and not looking like it's going to be crippled or dead
in five or ten years. LTO/AIT/etc appears to have roadmaps that looks
pretty vibrant and realistic.
Haven't really looked at flash stuff; that would probably be pretty
nice, but I think it's currently too costly for price/gigabyte. Even a
small backup setup today will be storing terabytes of data. So price has
to be much better to make it more realistic for the average place.
My 400 GB tapes costs about US 18.4 cents per gigabyte -- equivalent to
USD $73 for a 400 GB hard drive. That's pretty good. (But, yes, the tape
drives costs USD $12,000-$19,000 each.)
I'm afraid all of the above stuff has no direct relation to VMS, so I
will just say that the first VMS-based tape drive I ever owned was a
TK50 hooked up to my VAXstation 3100/GPX M38 that I bought years ago.
Included were several 'genuine DEC' VMS 5.4 tapes. Ah, fond memories.
First tapes I ever owned and used on a VMS system was the 9-track reel
tapes and also some fond memories of doing MOUNT/ALLOC MUA0 ... and
BACKUP operations.
At work today, we still use technology in the same form factor as these
TK50s but with significantly more advanced 'guts' (compression
algorithms, more precise head positioning, better chemical properties,
and so forth).
-Dan
> Tape shadowing:
Nice.
> Tape striping
> ------------------------
>
> Backup of large data stores can be substaintially sped
> up by writing the save set to multiple tape drives.
>
> Restoring the saveset will require all tapes making the
> strip to be loaded into the drives.
If the tape set is made with N drives, can it be restored with N-1
(or fewer) drives? Sounds like a reliability risk, if not.
------------------------------------------------------------------------
Steven M. Schweda (+1) 651-699-9818
382 South Warwick Street sms@antinode-org
Saint Paul MN 55105-2547
> [...]
>
> Haven't really looked at flash stuff; that would probably be pretty
> nice, but I think it's currently too costly for price/gigabyte. Even a
> small backup setup today will be storing terabytes of data. So price has
> to be much better to make it more realistic for the average place.
>
> [...]
Aside from the price issues speed is an issue too. Some months ago I've
seen some numbers on the SanDisk web site (don't remember the URL):
- 3...4 MB/s for "standard" flash memory,
- 8...10 MB/s for "professional" flash memory,
- 15...20 MB/s for "ultra high speed" flash memory.
Michael
--
Real names enhance the probability of getting real answers.
My e-mail account at DECUS Munich is no longer valid.
Unless there are new tape technologies I have not heard about,
tapes can never be as identical as disks (where such a "same size"
requirement can be met). Even with identical lengths measured to
the angstrom level, tapes stretch and the space taken up by X bits
of data will vary from one run to the other. One tape will run
out before the other.
Indeed where I used to work 9 years ago we had the first option already
availiable!
(I am amazed Wayne Sewell has not responded to this, he is usually
sharp as a button at spotting opportunities to publicise his
creations!)
Tapesys has this option already available. I know, as in the past I
used it and found it a real boon. In my new job (not very new actually,
I have been here 7 years now), I don't need it and haven't seen any
installation of it in many years, but I am sure it is still around.
Maybe if this is still in the planning, cerebral phase, and you haven't
yet put finger to keyboard about it, you could save a lot of time and
effort re-inventing it, by offering to buy a copy of their tape
shadowing software, and get it out in half the time. (I believe that is
very similar to is how SLS was born).
Just an idea..
- John
I would rather see the effort spent a new file system that includes
enhancements that can make backup better and easier for all OpenVMS users.
Whatever happened to Snapshot services?
> We are toying with the idea of modifying VMS BACKUP
> to support tape striping and tape shadowing, what does
> it mean?
> Backup of large data stores can be substaintially sped
> up by writing the save set to multiple tape drives.
>
> Restoring the saveset will require all tapes making the
> strip to be loaded into the drives.
I am concerned about the possible increased risk caused by
a damaged tape. If by "striping", you mean writing pieces
of each file on different tapes (like disk striping), then
a single bad tape means (I think) that you lose everything.
For example, let's say that a disk backup takes two tapes
to hold everything. Half of the files are on the first
tape and half are on the second (ignoring boundary conditions,
etc.). If the second tape is damaged, but my desired file
is on the first one, I can restore it OK.
If striping causes pieces of my file to be spread across
two tapes, then no files from either tape can be restored
if one tape is bad.
Can you address this fear?
Thanks,
Alan
John Vottero wrote:
> I would rather see the effort spent a new file system that includes
> enhancements that can make backup better and easier for all OpenVMS users.
> Whatever happened to Snapshot services?
Snapshot backups was one of the design goals of Spiralog. When it was
dropped we were promised Snapshot services - then that was dropped.
Again there were claims of problems but a Google search finds this rebuttal:
=====
Newsgroups: comp.os.vms
From: "Kevin Playford" <kevin.playf...@virgin.net> - Find messages by
this author
Date: Wed, 1 Nov 2000 20:00:44 -0000
Local: Wed, Nov 1 2000 8:00 pm
Subject: Re: What happened to Snapshot Services?
I guess my question was a loaded one as I'd worked on the product
previously.
The core of Snapshot Services still lives, I believe, in the Windows NT
product.
As for the product not working in a cluster. Well I'm not going to argue
against
the posted view about problems working in a cluster. There were problems but
I don't believe they were insurmountable problems.
Water under the bridge I'm afraid.
Kevin
Ex Project Lead for VMS Snapshot Services
========
DEC/Compaq/HP didn't need much of an excuse to cut a project throughout
the never ending downsizing.
--
Alan Greig
I think both of these will be useful.
I know that you only asked for comments on these two changes, but
since you're in there making changes, I'd like to see:
1. BACKUP/STATISTICS like SORT/STATISTICS
2. BACKUP ... BACKUP1.ZIP/ZIP
which someone else already mentioned. Of course, the zip would
have to keep the VMS file attributes, too.
--
Dale Dellutri <ddelQ...@panQQQix.com> (lose the Q's)
Clearly, it can be addressed. RAID 0+1 or RAID 5 are both applicable.
Just write one or more parity tapes in parallel with your data tapes.
Or one could simply realize that this is a niche product and the people
who need performance may be willing to live with the increased risk.
If I save my backup on a save set that spans 5 reels, and I lose a reel,
the backup is, for many purposes, just as useless whether those are 5
reels in sequence or 5 reels striped.
Another feature that I wish VMS Backup could have would be to allow
multiplexing of multiple data streams onto the tape. This is a
feature that I have seen in other backup products from other vendors.
This way you can have multiple threads gathering data from multiple
files and another thread gathering them and streaming them all to the
tape drive in a multiplexed fashion(and doing the data integrity
calculations). This provides a significant speedup in the feeding of
the tape drive(s). This kind of feature could also be immensely
helpful for sending data to multiple tapes. In fact you could have the
multiplexor thread(s) directing data traffic to multiple outputs in
parallel almost as easily as to a single stream to one.
I see the biggest challenge with the multiple drive output scenario
being the issue of how do you read it back in if you have fewer drives
available on another system when you need to do a disaster recovery
restore?
Tape shadowing sounds like a great idea. I have the same comment that
has already been made about how do you deal with EOT when you know the
tapes will not be identical in length? Do you do the End-of-Volume
processing for all the media in the shadowset when the shortest one
hits EOT? What if the media are really dissimilar (TK88,TK89 or other
pairs like that)???
Robert
Antonio
http://it.openvms.org
>
>Well, were I work at the moment, I don't really need it, but I am sure
>that in general it will be well received.
>
>Indeed where I used to work 9 years ago we had the first option already
>availiable!
Absolutely. tapesys (or rather the embedded product tshad) has been able to do
this since at least 1992, when I first started working with software partners.
Indeed, tshad was the first product I worked on when I started consulting.
Looking at the comments in the source, tshad was first created in 1988.
>
>(I am amazed Wayne Sewell has not responded to this, he is usually
>sharp as a button at spotting opportunities to publicise his
>creations!)
Yeah, I was kind of asleep at the wheel on this one. Thanks for mentioning my
name. I have news scanning software running in background scanning for certain
keywords, and it sent me mail when my name appeared. :-)
>
>Tapesys has this option already available. I know, as in the past I
>used it and found it a real boon. In my new job (not very new actually,
>I have been here 7 years now), I don't need it and haven't seen any
>installation of it in many years, but I am sure it is still around.
Not only that, but it is running on itanium! Not yet released, as testing is
still in progress, but there are no currently open bugs on it.
>
>Maybe if this is still in the planning, cerebral phase, and you haven't
>yet put finger to keyboard about it, you could save a lot of time and
>effort re-inventing it, by offering to buy a copy of their tape
>shadowing software, and get it out in half the time.
Or better yet, just encourage people to buy tapesys/tshad and get the
functionality today. :-)
>(I believe that is
>very similar to is how SLS was born).
yes, the original sls was a snapshot of tapesys from sometime back in the 80s.
They have diverged since, but there are still many similarities.
Interestingly, tapesys and sls are coming back together in a sense. Since sls
is not being taken to itanium and tapesys is already running on it, many of the
sls customers are considering tapesys as the closest alternative. The sls
databases are very similar to tapesys 5.2, so the database conversion from sls
to tapesys 6.2 is virtually no different from the conversion during an upgrade
from older tapesys. I have successfully converted several live customer
sls databases.
===============================================================================
Wayne Sewell, Tachyon Software Consulting (281)812-0738 wa...@tachysoft.com
http://www.tachysoft.com/www/tachyon.html and wayne.html
===============================================================================
Jake Blues:"You traded the Caddy for a microphone? ...... Okay, I can buy that."
Where did you hear this rumor? This is a dangerous rumor to be
spreading without any verification from good sources.
Robert
Antoniov wrote:
> Hi,
> I think both features may be interesting for VMS users. But in my mind
> it's more important backup should make a sort of snapshot before start
http://www.hpl.hp.com/hpjournal/dtj/vol8num2/vol8num2art3.pdf
"The Spiralog file system for the OpenVMS operating system
incorporates a new technical
approach to backing up data. The fast,
low-impact backup can be used to create
consistent copies of the file system while
applications are actively modifying data.
The Spiralog backup uses the log-structured
file system to solve the backup problem. The
physical on-disk structure allows data to be
saved at near-maximum device throughput
with little processing of data. The backup
system achieves this level of performance
without compromising functionality such as
incremental backup or fast, selective restore."
I was part of the early Spiralog external field test and it screamed
along in most situations.
--
Alan Greig
Yes. You do EOV processing for all reels when any one of them approaches
EOT. That's how TAPESYS does it today.
The alternatives do not bear much contemplation. If you decouple logical
EOV from physical EOV, for instance, how are you going to support
skip record reverse that spans physical volumes?
Just how much operator intervention are you going to tolerate in the
neighborhood of the first EOT marker?
I do not speak for VMS engineering. It just seems obvious to me that
there's exactly one reasonable implementation path.
John Briggs
Just to be contrary, I'd vote this is waste of time, or at least asking
here in comp.os.vms is a waste of time.
Why?
1) TAPESYS already exists, works well, and is well supported
2) I think there are other priorities the limited resources of VMS
Engineering should be working on, not the least of which is making our
porting lives easier -- fork() comes to mind.
3) At times it's mentioned that for a particular VMS feature request, a
'business case' is needed or should be submitted. Folks who lurk here
I suspect are not the ones in control of the $$$, so what's the point
of asking lots of VMS enthusiasts here if they would like a new
feature? Where's the business case for tinkering with BACKUP now after
some 20 years after it's introduction (VMS 2.5 I think)?
Your post quoted no context showing anyone talking about any rumor.
Do not assume that everybody's newsreading software is identical
to your own, or that they receive posts in the same order.
>
>We are toying with the idea of modifying VMS BACKUP
>to support tape striping and tape shadowing, what does
>it mean?
>
>Tape shadowing:
>----------------------
>
>The ability to write two (or more) copies of a tape
>saveset concurrently.
>
As has been mentioned elsewhere in the thread, tapesys customers have had this
capability for the last 17 years. tapesys itself does not perform tape
shadowing, but includes an embedded optional product called tshad. There is no
extra cost for tshad. As far as I know there never has been. If you have the
cdrom, all you have to do is install it.
Usage is trivially easy. One merely adds "copies == 2" (or some other number
greater than 1, limited to the number of physical tape drives that are
available) to the backup definition file. If copies > 1, tapesys automatically
activates tshad, allocates the multiple drives and tapes, and performs the
concurrent backup. Since the duplicate tapes have separate records in the
tapesys database, they can independently be marked as onsite and offsite, and
the offsite copy can processed by the tapesys autovault system with no operator
intervention. At the same time, the onsite copy is treated as just a regular
tape. And both copies appear in the tapesys history and summary information,
making them available for restores.
I have described the inner workings of tshad before, but here it is again.
When you create the shadow set, a virtual tape drive is created and bound to
the specified number of physical tape drives. (Note: tshad operates at driver
level, so you cannot mismatch drive types. They have to all be the same type.)
VMS backup sees only the virtual drive and thinks it is dealing with a single
drive with a single tape and a single label. (The multiple labels on the
multiple tapes are handled behind the scenes.) Therefore tshad works with any
version of vms backup.
tshad works at the qio level. Any qio issued to the virtual tape drive by
backup (read, write, rewind, setmode, sensemode, whatever) is replicated to
each of the physical drives. When all of the duplicate operations complete,
the virtual operation is posted complete. If all return codes are good, the
return code for the virtual op is good. A failure on any drive becomes a
failure on the virtual drive, so backup performs its normal error handling,
even though not all drives actually had an error. This may cause unnecessary
rewriting on some of the drives, but the savesets remain logically identical.
It is possible for different drive/tape combinations to reach end of tape at
different times, especially open reel tapes, which can be radically different
in length. But the error processing above holds in this case. When the
shortest reel hits end of tape, they all hit end of tape, even though all but
one may actually have tape left. At this point, tapesys loads the next set of
tapes, and backup continues, just as if there had just been a single drive.
Again, all savesets continue to be logically identical.
Wayne
Here is the message I was replying to:
24. Antoniov
Nov 7, 12:16 pm show options
Newsgroups: comp.os.vms
From: "Antoniov" <anton...@shs-av.it> - Find messages by this author
Date: 7 Nov 2005 09:16:37 -0800
Local: Mon, Nov 7 2005 12:16 pm
Subject: Re: Request for feedback - BACKUP enhancement
Reply | Reply to Author | Forward | Print | Individual Message | Show
original | Report Abuse
Hi,
I think both features may be interesting for VMS users. But in my mind
it's more important backup should make a sort of snapshot before start
it.
Any system administrator 24x7 would like to backup while system is
running and everyone knows how dangerous is /IGNORE=INTERLOCK
qualifier. I even heard, next OpenVMS version don't support this
qualifier.
Antonio
http://it.openvms.org
I agree that striping will require (or at least allow) using parity tape
I can assure you that next version of VMS will support
/ignore=interlock !!!
->Tape striping
->------------------------
->
->Backup of large data stores can be substantially sped
->up by writing the save set to multiple tape drives.
->
->Restoring the saveset will require all tapes making the
->strip to be loaded into the drives.
->
->What do you think?
This one sounds very useful for cutting the backup window time.
As our SAN grows that's always the issue - will the full backups complete
before work starts tomorrow. Cutting that window in half would be a big win.
I know that some EBS products do this but we'd rather stick with
VMS backup.
--
-- Carl Karcher, Waisman Computing Services, Waisman Center, UW-Madison
-- karcher.n...@waisman.wisc.edu
While I've never seen the product, nor am I knowledgable about it's
capabilities, The third party product TapeSys claims to write to
multiple tape drives.
Possibly you see the chance for additional capability.
From my perspective, having software vendors who target VMS with their
products is a good thing for VMS. Cutting their throats by duplicating
their products and incorporating the capability into the base VMS
license costs is a real good way to drive such vendors away from VMS.
If some customers really need the capability, and won't deal with ISVs,
then make a deal to re-sale the product directly through HP.
I'm sure there are many other things VMS engineering could be working
on, for example, a better file system.
> Tape striping
> ------------------------
>
> Backup of large data stores can be substaintially sped
> up by writing the save set to multiple tape drives.
>
> Restoring the saveset will require all tapes making the
> strip to be loaded into the drives.
>
> What do you think?
>
> I'm looking for feedback like yes this is a good idea or
> no it is a waste of time. Please do not ask questions
> about implementation as we are in a VERY!!! early
> stage of planning and obviously this is not a commitment
> that we'll actually do it....
>
> Thank you for your feedback.
>
> Guy Peleg
> OpenVMS engineering
As for tape striping, don't know if anything currently exists. I'm
guessing that you'd need to have several parity tapes as in RAID5. I'd
think that customers who would use this, might also want more than one
copy, what you're calling tape shadowing. I have to wonder whether many
would have enough drives to do such.
--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. Fax: 724-529-0596
DFE Ultralights, Inc. E-Mail: da...@tsoft-inc.com
170 Grimplin Road
Vanderbilt, PA 15486
On Mon, 7 Nov 2005 12:22:13 +0200, Guy Peleg
<guy.peleg@remove_this_header@hp.com> wrote:
> Hi folks,
>
> We are toying with the idea of modifying VMS BACKUP
> to support tape striping and tape shadowing, what does
> it mean?
>
> Tape shadowing:
> ----------------------
>
> The ability to write two (or more) copies of a tape
> saveset concurrently.
>
>In a previous article, "Guy Peleg" <guy.peleg@remove_this_header@hp.com> wrote:
>
>->Tape striping
>->------------------------
>->
>->Backup of large data stores can be substantially sped
>->up by writing the save set to multiple tape drives.
>->
>->Restoring the saveset will require all tapes making the
>->strip to be loaded into the drives.
>->
>->What do you think?
>
>This one sounds very useful for cutting the backup window time.
>As our SAN grows that's always the issue - will the full backups complete
>before work starts tomorrow. Cutting that window in half would be a big win.
>
>I know that some EBS products do this but we'd rather stick with
>VMS backup.
>
My answer to the tape striping issue is the same as my earlier response about
the tape shadowing issue: get tapesys to handle it for you. Available right
now, and currently running on itanium.
True, tapesys doesn't have actual tape striping as described above, but it has
another feature that can increase your backup throughput in a similar fashion:
parallel backups.
Sure, tape striping would help when backing up a single large saveset. But
most sites have quite a few savesets that they want to back up.
The basic problem for a system manager to solve is: "I have a certain number of
gigabytes on a certain number of disks that must all be backed up to tape on a
certain schedule. I want all these backups to be performed in the shortest
possible time."
As others have mentioned, and the original poster admitted, fanning out a
single backup saveset to multiple drives greatly complicates the recovery
process. And only one saveset can be done at a time.
If you have multiple tape drives available, the goal is simply to keep all of
them spinning at full speed until all of the backups are done.
Having a number of *instances* of vms backup equal to the number of tape drives
will accomplish this as easily as tape striping, and the resultant savesets are
exactly like the savesets we are now using. When restoring, you only have to
mount the one tape containing the beginning of the saveset. If it spans
volumes, backup will ask for the next one.
You have a list of disks to back up. You break them up into a set of batch
jobs equal to the number of tape drives.
Just writing the command procedures to do this will allow you to increase your
thoughput time.
The disavantages to this manual approach are:
1. you must keep track of which disks are in which backup job and make sure
all of them are accounted for
2. you must do manual load balancing; if one backup job contains only big
disks and another only small ones, the tape drive assigned to the latter will
finish way early and sit idle for hours while the other keeps grinding; each
backup job should roughly contain about the same amount of data
3. if the backup load for a disk drastically changes (many files added, many
files deleted) load rebalancing may be required.
You can do all this manually, or you can get tapesys and activate parallel
backups. Say you have 40 disks of varying sizes to back up and 5 tape drives.
You simply create a system backup file (.sbk) to define the backup job.
Specify all 40 disks in whatever order you want and add:
$ parallel_backups == 1
$ n_drives == 5
Set it to kick off at a particular time via the builtin backup job scheduler
of tapesys and that's it.
When the backup job starts, sysbak will notice that parallel backups are in
effect, and kick off 5 clones of itself, one for each tape drive. The master
process will then begin assigning disks to back up, using the list in the
parameter file. Each clone will accept the next assignment, perform the
backup, and then ask for the next in the queue. All five clones are doing this
in parallel. The tapes spin continuously. If a clone gets a short backup, it
is quickly back to ask for another. If it gets a long backup, it grinds until
it is finished. There is no idle time, until *all* backups have ether been
done or are in progress. As each clone runs out of backups, it terminates.
When all clones are gone, so is the master.
To me, this seems just as efficient as processing each saveset linearly and
splitting it out to multiple drives. And a lot less complicated.
I find the requirement that all tapes be loaded simultaneously for
restoration to be too dangerous.
A bad tape drive or a different tape configuration at a DR site could
make the backup useless.
If the restoration can be done with a subset of the original tape
drives, this feature would be usefull.
Personally, I would much rather see BACKUP support host based
compression of the data. That way, time does not need to be wasted
sending blocks of zeros to the backup device.
--
-----------------------------------------------------------------------
Chris Scheers, Applied Synergy, Inc.
Voice: 817-237-3360 Internet: ch...@applied-synergy.com
Fax: 817-237-3074
a few years ago, I did see a hardware solution (RAIT0, 1 and 5 [T means
tapes}). IMHO tape RAITs are a good solution. Modern RAIDset do handle
disks with different sizes and will store all RAIDset relevant data
within first blocks of every disk. A future BACKUP RAITset could it do
too. The easiest way is to end if the first tape is full. Another
solution is like DLTsage, You will have a media information block(s) at
first. This block contains number of free blocks, error counter, etc.
For a RAITset it could be expanded wit a set identifier, number of
members, RAIT level, etc.
Best regards R. Wingert
This approach worked well -- it didn't have to be complicated since the
variability in amount of data backed up each time was sufficient to
make a close approximation good enough. The main benefit was that more
often than not as many as 3 or 4 or 6 backup streams on any given
system usually finished within 10-15 minutes of each other.
Once upon a time a project manager for ABS came to visit us when ABS
was in its early days and promised us that ABS would have a capability
like this built into it. Sad to say they never implemented it.
ABS is pretty good for what it does, but it's kindof like OS/JCL --
lots of bells and whistles that you have to stand on your head, pat
your tummy and whistle to make it roll over and sit up when you want it
to.
I will admit -- that with the various design goals and strategies
required to backup a wide variety of configurations and types of
systems, it is challenging to build something that is easy to drive and
smart at helping you make the process as efficient as possible.
What I'd like to see is something with the relative ease and speed of
Legato Networker combined with the data reliability/recoverability
features of VMS Backup, and the smarts of Tapesys with built in
performance tracking and adaptive scheduling.
I've worked with Tapesys in its early days, SLS, Networker, and ABS --
they all have some great features. I just wish there was something
that combined all of the best features and lost the ugly ones.
Robert
Antonio
http://it.openvms.org
>
>I've worked with Tapesys in its early days, SLS, Networker, and ABS --
>they all have some great features. I just wish there was something
>that combined all of the best features and lost the ugly ones.
>
In the case of tapesys, it may be that some of the ugly features you recall are
gone already.
Be aware that tapesys has changed a *lot* since its early days. Even 5.2 is
ancient now, given the radical changes introduced with 6.x in *this*
millenium. For instance, the parallel backups mentioned earlier didn't exist
in 5.x, nor did support for ODS-5 and TCP/IP and ICC. And early versions
certainly didn't have the backup API, so the entire backup mechanism is cleaner
now.
I often have this problem. A while back somebody who had used tapesys in the
eighties was posting that it was a great product, and it would be even better
if <annoying-behavior> could be changed. I would reply "Uh, it hasn't done
that for about 14 years now. So consider your request granted."
Wayne
===============================================================================
Wayne Sewell, Tachyon Software Consulting (281)812-0738 wa...@tachysoft.com
http://www.tachysoft.com/www/tachyon.html and wayne.html
The recommended snapshot is to dismount the disk and mount it privately.
For situations where that is not possible, there is nothing worthwhile
for VMS to do automatically, since the problem is too complex. Beyond
the simple issue of consistency within a file when backing up index
structures as well as data, there is also the problem of consistency
between files which have related data and need to be backed up in a
consistent fashion, even if they are on separate disks.
So at this point, a human who understands the applications on the
machine must design a backup plan that will work. When implementing
that plan, the technique of making shadow copies and removing them
from the shadow set for the backup has proven a useful tool for the
task, but no complete solution from VMS is possible.
The bitmap processing for shadow sets added in recent years helps
to make the shadow set manipulation run more quickly, but no fully
automated general backup mechanism for complex systems is possible.
But if you don't use /IGNORE=INTERLOCK and the file is open for write,
it won't be saved at all! What if the file in question is open for
write, but no one is writing to it? Then you've missed copying a
perfectly good file. And after restore, you won't be able to read data
from the file, because it won't be there at all!
So it's "dangerous" either way. But with /IGNORE=INTERLOCK, at least
there's a chance you'll get a good copy. Without it, you've got
nothing.
The important thing is too be aware of this problem and take
appropriate measures.
<big snip>
> You simply create a system backup file (.sbk) to define the backup job.
> Specify all 40 disks in whatever order you want and add:
>
> $ parallel_backups == 1
> $ n_drives == 5
>
> Set it to kick off at a particular time via the builtin backup job scheduler
> of tapesys and that's it.
>
>
> When the backup job starts, sysbak will notice that parallel backups are in
> effect, and kick off 5 clones of itself, one for each tape drive. The master
> process will then begin assigning disks to back up, using the list in the
> parameter file. Each clone will accept the next assignment, perform the
> backup, and then ask for the next in the queue. All five clones are doing this
> in parallel. The tapes spin continuously. If a clone gets a short backup, it
> is quickly back to ask for another. If it gets a long backup, it grinds until
> it is finished. There is no idle time, until *all* backups have ether been
> done or are in progress. As each clone runs out of backups, it terminates.
> When all clones are gone, so is the master.
>
What if you have a variety of tape drives, some the latest and fastest,
others older, slower, but still going strong?
Can you send certain disks which need to be available ASAP to the
fastest drives?
There is currently no mechanism for this built into parallel backups. The pool
of drives to use in a sysbak is specified by the "media type" associated with
the job. For instance, a media type of DLT3XT would have a list of drives that
could operate at that density. As this point, there is a single media type for
the whole job. The list is just a list; there is no mechanism for "preferred
drives".
Such a mechanism could certainly be requested via the enhancement list.
Parallel backups itself came about from such a customer request.
In the meantime, there is a way to do pretty much what you want, if you don't
mind splitting the priority and regular disks into two backup jobs and
splitting the fast and slow drives in a similar fashion. Without knowing the
exact disk and drive ratios, it is hard to predict whether this would help or
not.
Anyway, you would create two media types instead of one: FAST_DLT and SLOW_DLT.
You would assign the drives to the two lists accordingly. Then create two
backup jobs, one containing the hot backups and and using the FAST_DLT media
type. The other job would back up the regular stuff with the slow drives.
Say you had five fast drives and five slow drives.
$ tape oper add/media=fast_dlt/drives=(drive1,drive2, ...)/phys=dlt
$ tape oper add/media=slow_dlt/drives=(drive1,drive2, ...)/phys=dlt
(In this simple scenario, we will ignore other possible qualifiers, such as
/location, /density, and /node.)
So then you have two backup definition (.sbk) files instead of one, and two
independent jobs.
Both .sbk files contain
$ parallel_backups == 1
$ n_drives == 5
as before, but different media types:
$ media_type == "fast_dlt"
and
$ media_type == "slow_dlt"
and different lists of disks to back up, of course.
Basically the hot backups would compete with each other for the fast drives,
and the less critical backups for the slow drives.
> We are toying with the idea of modifying VMS BACKUP
> to support tape striping and tape shadowing,
Great!
> Tape shadowing:
> ----------------------
>
> The ability to write two (or more) copies of a tape
> saveset concurrently.
What about reading, i.e. just like disk volume shadowing? If someone
has data important enough to justify tape shadowing, he probably needs
to be able to restore it quickly. Imagine what would happen if, close
to finishing the restore, he has to start over with the other tape? It
would be nice to have a tape volume set which behaves like a disk volume
set!
> Tape striping
> ------------------------
>
> Backup of large data stores can be substaintially sped
> up by writing the save set to multiple tape drives.
Rdb can already do this with the RMU/BACKUP command. I think it is now
possible to have such a multi-stream backup to disk as well. What about
implementing that, i.e. have the possibility to do a multi-stream backup
(to split the load across several processors), but to disk as well as to
tape.
> I seem to remember that RMU/BACKUP already has the option of using two
> tapes in a striping pair?
More than 2.
> I am concerned about the possible increased risk caused by
> a damaged tape. If by "striping", you mean writing pieces
> of each file on different tapes (like disk striping), then
> a single bad tape means (I think) that you lose everything.
>
> For example, let's say that a disk backup takes two tapes
> to hold everything. Half of the files are on the first
> tape and half are on the second (ignoring boundary conditions,
> etc.). If the second tape is damaged, but my desired file
> is on the first one, I can restore it OK.
>
> If striping causes pieces of my file to be spread across
> two tapes, then no files from either tape can be restored
> if one tape is bad.
>
> Can you address this fear?
Parity tape? :-)
> If /IGNORE=INTERLOCK won't be supported, and there is no snapshot
> service available, VMS backups will become nearly impossible to
> accomplish in 24x7x365 shops.
If it is literally 24x7x365, then yes. However, if you can shut off
disk access long enough to cleanly break a shadow set, then there is no
problem (especially using MINICOPY).
AFAIK RDB writes what in VMS would be a file to a single tape, files
are being spread across tapes....I thought about writing different LBNs
to different tapes....we'll see....now I'm getting to commenting on
implementation
which I try to avoid as possible ;-)
As for being able to have 2 output, I would also need this. Right now I do a
backup/record to tape, then another backup to disk with a /Since="beginning
of tape backup"/backup (I keep the time the tape backup began in a symbol)
to backup the same files, to a disk location (actually through DECnet).
So YES this would be very useful to me but the 2 output must be of different
types (a tape drive and a DECnet location).
Syltrem
"Guy Peleg" <guy.peleg@remove_this_header@hp.com> wrote in message
news:436f...@usenet01.boi.hp.com...
> Hi folks,
>
> We are toying with the idea of modifying VMS BACKUP
> to support tape striping and tape shadowing, what does
> it mean?
>
> Tape shadowing:
> ----------------------
>
> The ability to write two (or more) copies of a tape
> saveset concurrently.
>
> Tape striping
> ------------------------
>
> Backup of large data stores can be substaintially sped
> up by writing the save set to multiple tape drives.
>
> Restoring the saveset will require all tapes making the
> strip to be loaded into the drives.
>
> What do you think?
>
"Phillip Helbig---remove CLOTHES to reply" <hel...@astro.multiCLOTHESvax.de>
wrote i
> Rdb can already do this with the RMU/BACKUP command. I think it is now
> possible to have such a multi-stream backup to disk as well. What about
> implementing that, i.e. have the possibility to do a multi-stream backup
> (to split the load across several processors), but to disk as well as to
> tape.
They charge you a lot extra for it though :-( Earlier in the year a customer
saw that after their Rdb upgrade they could now backup to multiple disks as
well as tape drives and they thought it was the bee's knees for shorttening
backup elapsed time! (Until they realized they had to stick their hand in
their pocket)
Regards Richard Maher.
"Phillip Helbig---remove CLOTHES to reply" <hel...@astro.multiCLOTHESvax.de>
wrote in message news:dkqq9d$7kq$1...@online.de...
"Guy Peleg" <guy.peleg@remove_this_header@hp.com> wrote
> AFAIK RDB writes what in VMS would be a file to a single tape, files
> are being spread across tapes....I thought about writing different LBNs
> to different tapes....we'll see....now I'm getting to commenting on
> implementation
> which I try to avoid as possible ;-)
That is also my understanding, with obvious issues for different storage
area/file sizes. But! You could uses RMU to backup a database to multiple
spindles and then use your wiz-bang BACKUP to send the resulting files
across the available Tape Drives.
Regards Richard Maher
"Guy Peleg" <guy.peleg@remove_this_header@hp.com> wrote in message
news:4370...@usenet01.boi.hp.com...
I'm not sure if you might be developing for something that
may not be used much going forward.
Regarding your customer base, are many of them large shops?
Maybe what you see going forward is folks backing up to large
disk farms - large customers - (is all the rage you know and certain
to shift that direction) so maybe hitting multiple tape drives for
shadowing and multiple tape drives with striping is rather moot.
Couple references:
http://searchstorage.techtarget.com/originalContent/0,289142,sid5_gci1016117,00.html
In the 2004 Purchasing Intentions Survey (to be published in the November issue
of Storage), 600 users from industries such as government, healthcare and
finance were asked a variety of questions about how they are spending their
storage dollars. When it comes to disk-to-disk backup, 53% of respondents are
increasing their spending. Only 31% of respondents are increasing their use of
tape, down from 56% in 2003.
http://informationweek.com/story/showArticle.jhtml?articleID=171201537
Chicago Merc Ditches Tape Backups For Disks Sept. 29, 2005
The Chicago Mercantile Exchange Inc. has experienced exponential growth in its
IT infrastructure as electronic trading volume for its futures and options
contracts soars. In the past two-and-a-half years, its storage area network has
grown from 4 terabytes to more than 180 terabytes, the number of servers has
grown from 500 to more than 2,000 -- half of them Linux -- and its expanded
from one data center to three.
That's raised a fundamental question -- where to store all that data? Chicago
Merc's infrastructure is made up of four tiers of storage: Tier 1 for critical
application data and databases that require high performance, such as trading
systems; Tier 2 for noncritical production databases; Tier 3 for long-term
archiving, such as that required for regulatory retention; and Tier 4, a
tape-based system for backup and recovery.
The sore point was Tier 4. The Chicago Merc had two StorageTek tape silos that
were controlled by mainframe-based software. Tape backups were slow and
resulted in a high rate of backup and restore failures. Also, there were only
two silos supporting three data centers, so an additional silo would need to be
purchased.
The Merc hit upon the idea of adding a new tier 4, with disks replacing tapes
as the medium for backup and restores, and bumping tape to tier 5, where it
would be used only for off-site storage and disaster recovery.
Your target might move for what you are suggesting for
development as "tape based systems for backup and recovery" are
certain to be disk based thing from now on and you would have
appropriate RAID levels in the disk backup farm. (Unless of course a
VERY large customer is asking for it - sure go ahead ;-).
I'd rather see something like differentials:
http://groups.google.com/group/comp.os.vms/msg/47c904d60324c301?dmode=source&hl=en
Newsgroups: comp.os.vms
Subject: WRITEMAP.SYS
Date: 24 Jun 2003 14:30:48
Since host-based mini-merge is coming to a future version of
VMS, I thought why not combine the tracking of "writes in progress"
on to a WRITEMAP.SYS?
The purpose would be twofold:
1) Allowing a BACKUP/MODIFIED_BLOCKS backing up only blocks
that have been written to since last full IMAGE.
(Let's face it, many of us are backing up databases
every night and in some/many of these databases little
changes night to night).
2) Memory required to track "writes in progress" could
be reduced and fixed in size.
The layout of WRITEMAP.SYS would be an index file each data node
would track 127 blocks at a time. Each 127 block to be tracked
would be a 9 byte entry.
etc.
http://groups.google.com/group/comp.os.vms/msg/33a06987c229ebfa?dmode=source&hl=en
>
>
> 2) Memory required to track "writes in progress" could
> be reduced and fixed in size.
>
Make that:
Memory required to track blocks modifed for mini-copy
could be reduced and fixed in size.
----
Rob
I have no tape drive at home, so backup to disk is a necessity. I feel
that VMS Engineering's limited resources can best be spent elsewhere.
In my last job, databases went to disk first, and then backed up to tape
"at leisure". Not only were the databases back up ASAP, but if a tape
backup barfed during the night it didn't affect database availability.
One valuable advantage of this method in practice was that the DBAs were
responsible for the backups to disk, and the system management team for
getting those backups off to tape. IOW, each team was in charge of what
it specialised in.
> I have no tape drive at home, so backup to disk is a necessity.
I invested in a DAT drive when I first got my home Alpha. Not cheap, but
I'd bought the Alpha when the USD was high and disks still pricy, so
felt it was worthwhile. The advantage since then has been that I can use
my Alpha as a staging post for backups of my other systems, and can buy
extra tapes cheaply.
> I feel
> that VMS Engineering's limited resources can best be spent elsewhere.
I'm inclined to agree, though that would change if enough customers said
"Yes please". There has to be a solid business case either way.
Question:
For a very large sequential file opened for append.
When backup gets to it, I assume it gets the file header, complete with
the number of blocks/extends and location of end of file in the last
block, and then proceeds to copying th header and the blocks to tape.
Right ?
If so, wouldn't that constitute a proper snapshot ? When you restore
the file, the original number of blocks are retored with the original
eof offset in the last block, no matter how many blocks were added to
the file while backup was copying it.
RE: why re-invent the tapesys wheel.
having a robust trustable full featured backup that comes with the OS is
one of the advantages VMS has over Unix and Windows. So it can be
argued that working to improve backup with more features can be a good
marketing tool to sell VMS to more customers.
Whether VMS management simply licence the stuff from tapesys or do it
themselves is open for debate.
Well, I trust you understand the ramifications of doing so: a huge file
could take hours to write to tape, and where does VMS store the "deltas"
while the "original" file is being read?
There is a case where /IGNORE=INTERLOCK sort of saves the day: database
tablespace files. You put the database (or tablespace) into backup mode
for the duration of the save operation. At least for Oracle, this causes
buffers to be flushed out to disk, even though the file(s) remain open
for write. So, you *CAN* safely /IGNORE=INTERLOCK and still get the
file(s) with full integrity.
--
David J Dachtera
dba DJE Systems
http://www.djesys.com/
Unofficial OpenVMS Hobbyist Support Page:
http://www.djesys.com/vms/support/
Unofficial Affordable OpenVMS Home Page:
http://www.djesys.com/vms/soho/
Unofficial OpenVMS-IA32 Home Page:
http://www.djesys.com/vms/ia32/
Coming soon:
Unofficial OpenVMS Marketing Home Page
I'll second that.
If ABS had everything SLS has, including SLS's flexibility (*_SBK.COM
files allow certain backup parameters to be determined at batch-job run
time, such as MEDIA_TYPE, the FILES_n virtual array, etc), it could then
be an acceptable replacement for SLS.
As it is, ABS is not "ready for prime time". SLS beats it hands-down,
IMO.
From what I've seen, TapeSys has some neat features as well.
None of the three, however, could handle our STK configuration "out of
the box" without extensive custom coding and DCL-jacketing. It should
have extraordinarily easy to resolve the shortfalls in SLS, and ABS
would have been a perfect change to extend SLS rather than gut it to
virtual uselessness and just expose the MDMS "underbelly".
Naturally, ABS is being taken forward to I64, but not SLS. Guess they
WANT to keep losing business.
Guy,
Based on these postings, perhaps it would be in the best interests of
your group to conduct a formal poll of the user base to see what sites
need that neither BACKUP, SLS, ABS or third-party solutions can
currently offer.
It'd be better than trying to guess.
Don't misconstrue - y'all come up with some *GREAT* ideas.
There's bound to be some true wisdom to be gleaned by seeing what issues
are being faced and seeking solutions to those issues.
...IMHO.
>>From: kar...@thuria.waisman.wisc.edu (Carl Karcher)
>>X-Newsgroups: comp.os.vms
>>Subject: Re: Request for feedback - BACKUP enhancement
>>Date: 7 NOV 2005 11:19:48 GMT
>>Organization: Waisman Center, University of Wisconsin-Madison
>
>
>>In a previous article, "Guy Peleg" <guy.peleg@remove_this_header@hp.com> wrote:
>>
>>->Tape striping
>>->------------------------
>>->
>>->Backup of large data stores can be substantially sped
>>->up by writing the save set to multiple tape drives.
>>->
>>->Restoring the saveset will require all tapes making the
>>->strip to be loaded into the drives.
>>->
>>->What do you think?
>>
>>This one sounds very useful for cutting the backup window time.
>>As our SAN grows that's always the issue - will the full backups complete
>>before work starts tomorrow. Cutting that window in half would be a big win.
>>
>>I know that some EBS products do this but we'd rather stick with
>>VMS backup.
>>
>
>
>My answer to the tape striping issue is the same as my earlier response about
>the tape shadowing issue: get tapesys to handle it for you. Available right
>now, and currently running on itanium.
>
<description of parallel backups deleted>
Sorry to reply to my own post, but I forgot to mention something: you can
combine parallel backups with tape shadowing in any configuration you want as
long as the number of parallel backups times the number of copies equals the
total number of tape drives available.
So if you have 8 tape drives, you can have:
a. 8 parallel backup jobs each writing 1 copy of the savesets
b. 4 parallel backup jobs each writing 2 copies of the savesets on 2 tapes
c. 2 parallel backup jobs each writing 4 copies of the savesets on 4 tapes
d. a single backup job writing 8 copies of the savesets on 8 tapes
Not sure how many people would want c or d, but I'm sure somebody somewhere
does. Also not sure what the shadowing performance would be writing identical
records to 8 tape drives simultaneously.
Can't try it, don't have that many. :-)
> Antoniov wrote:
>>
>> The /IGNORE=INTERLOCK qualifier may be very dangeorus because you can
>> save a file while somebody is writing in it. After restore you couldn't
>> read data from file!
>> I agree this is dangerous so in my mind VMS should make a snapshot
>> before backup!
>
> Well, I trust you understand the ramifications of doing so: a huge file
> could take hours to write to tape, and where does VMS store the "deltas"
> while the "original" file is being read?
>
> [...]
Isn't that (very) similar to what is accomplished with "mini-merge" when
a (previous) shadow-set member is "re-integrated"?
Michael
--
Real names enhance the probability of getting real answers.
My e-mail account at DECUS Munich is no longer valid.
How so?
In mini-merge, a bitmap is used to track the 127-block segments that
have write activity between clearings of the bitmap.
In snapshot, the system stores the original blocks in a secondary
location on a reserved volume when changes to the "base" occur until the
snapshot is released.
Looks like two entirely different critters to me, although snapshots
look to be a logical offshoot of write-bitmaps.
Redundant Array of Independent Tapes (RAIT) - no, its not just fantasy.
http://www.thic.org/pdf/Oct00/stk.mfisher.pdf
I believe its patented though.
While I did not response to every one of the replies
(or the 100 extra messages sent directly to me) I sure
read it all.....
We have some research to do, although it looks like there
is an interest in these features (while implementation is
yet TBD) and I'll keep you guys posted.
Here is a quick update on latests happenings with
BACKUP:
1. Starting with V8.3 BACKUP will support encrypting
savesets using AES encryption.
2. Latest BACKUP ECO shipped couple of weeks
ago, updated the BACKUP$_OPERSPEC message
to include requesting PID and target device, this helps
identify the process making the request when parallel operations
are being performed.
3. BACKUP has been enhanced to support Dynamic Volume
Expansion - preserve the logical size and expansion size of the
disk. Also we added two new qualifiers /LIMIT & /SIZE
to allow specifying the expansion size and logical size regardless
of the values stored in the saveset. This functionality will ship with
V8,3 and with the next round of BACKUP ECOs (if you need
it earlier contact your support center).
Currently we are in the process of improving the way BACKUP
is handling I/Os to reduce the disk queue load.
Again thank you for your feedback.
Guy Peleg
OpenVMS Engineering
Will backup savesets created on Alpha 8.3 be readable on the unupdated
VAX BACKUP now stuck at 7.3 ? (assuming no compression is used).
When distributing products/data, will folks have to use VAX-BACKUP to
ensure that the save sets can be read by both VAX and Alpha nodes ?
AES encryption support is currently only available on V8.3. A saveset
encrypted using AES can only be decrypted on V8.3. If you encrypt
using one of the DES algorithms you can decrypt on any version.
As for Dynamic Volume Expansion - saveset created using the
modified code could be read by prior versions including VAX
(it was coded with maintaining compatibility in mind )
Guy
>+In article <436f...@usenet01.boi.hp.com>, "Guy Peleg" <guy.peleg@remove_this_header@hp.com> writes:
>+
>+> Tape shadowing:
>+> ----------------------
>+>
>+> The ability to write two (or more) copies of a tape
>+> saveset concurrently.
>+
>+This would be highly useful, although the same goal might be
>+achieved for many sites by a supported BACKUP/COPY.
Do you mean *parallel* backup (or COPY) ?
Then I must disagree.
Making two sequential backups or copy of backup means:
- more operator(s) time
- more devices (tapedrives, places on disk) required to be
available
..and possibly both of above. True ?
Regards - Gotfryd
--
=====================================================================
$ ON F$ERROR("LANGUAGE","ENGLISH","IN_MESSAGE").GT.F$ERROR("NORMAL") -
THEN EXCUSE/OBJECT=ME
=====================================================================
HTH
Antonio
http://it.openvms.org
What I'd like is functionality to write to exactly ONE tape ;)
We've a single drive, and do unattended backup. All's well, until it
overflows, and then instead of a nice mail message saying what tapes to
feed today, and the results of last night's backups, the backup hangs
requesting tape2 - here no-one's listening, there's no second tape, we'd
just like it to finish. (the backup procedure works out which disks need
doing first, so failure to finish everything in one night is acceptable)
I wrote nightwatchman (it's on freeware somewhere) to handle replying to
these requests, but it's a nasty clunky solution.
Backup has a noassist qualifier which stops it asking for the first
tape, but it will still ask for the second.
How about a non-default /NOASSIST=ALWAYS or something.
Chris
Antoniov,
The snapshot can be avoid if you are using shadow-sets ! Make sure you
have 1 extra disk in the shadowset for backup. So if you are on two
sites, make sure you have 3 disks per shadow-set. Disks are inexpensive
nowadays. Before starting the backup, dismount one member of all
shadow-sets and backup these dismounted devices.
After backup (per disk for performance) mount the disk back in with
/MINICOPY !
This works fine. Do not relay on snapshot or other 'beautiful' tools in
SAN's. YOU DO NOT HAVE ANY CONTROL OVER IT. (Most of the time the
SAN-manager eather).
AvR
If I'm right this won't work for one saveset over more then one tapes,
does it ?
AvR
Gee, what you're asking for sounds like exactly the problem I had
with UNIX, it wouldn't run dump in the background and give me a
way to change tapes in the morning. dump wanted to do an explicit
open on /dev/tty to discuss changing tapes. If opened stdin/stdout
I could have set up a pipe to emulate OPCOM.
What you want is exactly what I don't want. The current behaviour
lets me start backups overnight, change tapes in the morning, and
rest assured that ALL my data is safely backed up.
Meanwhile, one simple "reply/abort" and you can actually get what
you're looking for.
OPCOM is a great friend.
One issue to consider in this however is $$$$$. Can you justify spending
the big bucks for disk shadowing licence only so you can do a backup ?
Wrong issue. The question is justifying the $$$$$ to allow you to run
24 x 365. If that's what you need, and you can justify spending the
$$$$$, then that's what you do.
A backup taken on a running system is a waste of time and effort. Data
files will be out of sync. Worthless.
Doing a backup is mandatory, for anyone with retained data. Protecting
against data loss with shadowing doesn't protect against the fire that
takes out the whole system. Only offsite data, and/or multiple sites
with data shadowed on both sites may do that. Even multiple sites do
not protect against loss of data through application and/or user errors.
I can see the tax people now. "What do you mean that you don't have
supporting data for this $20,000,000 deduction? Disallowed!"
--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. Fax: 724-529-0596
DFE Ultralights, Inc. E-Mail: da...@tsoft-inc.com
170 Grimplin Road
Vanderbilt, PA 15486
> The snapshot can be avoid if you are using shadow-sets !
However, it can be done so cleanly only if no files are open for write.
No they won't be out of sync if you close databases down,
>
> Doing a backup is mandatory, for anyone with retained data. Protecting
> against data loss with shadowing doesn't protect against the fire that
> takes out the whole system. Only offsite data, and/or multiple sites
> with data shadowed on both sites may do that. Even multiple sites do
> not protect against loss of data through application and/or user errors.
>
> I can see the tax people now. "What do you mean that you don't have
> supporting data for this $20,000,000 deduction? Disallowed!"
>
Giving that example is just being sensationalist.
Hmmm, asking delicately, what part of 'running system' didn't you
understand? If users need to be using the applications full time, then
data files, databases, and such are constantly being updated.
>> Doing a backup is mandatory, for anyone with retained data.
>> Protecting against data loss with shadowing doesn't protect against
>> the fire that takes out the whole system. Only offsite data, and/or
>> multiple sites with data shadowed on both sites may do that. Even
>> multiple sites do not protect against loss of data through application
>> and/or user errors.
>>
>> I can see the tax people now. "What do you mean that you don't have
>> supporting data for this $20,000,000 deduction? Disallowed!"
>>
>
> Giving that example is just being sensationalist.
Yes it is. That said, it's still valid, and regardless of value, lost
data is just that, lost data.
>I would like to thank everyone for their feedback.
>
>While I did not response to every one of the replies
>(or the 100 extra messages sent directly to me) I sure
>read it all.....
>
>We have some research to do, although it looks like there
>is an interest in these features (while implementation is
>yet TBD) and I'll keep you guys posted.
>
>Here is a quick update on latests happenings with
>BACKUP:
>
<changes deleted>
>
>Again thank you for your feedback.
>
While you are making improvements, could anything possibly be done with the
API? AFAIK, the interface has not changed in the slightest since the API was
first introduced in 7.1 or 7.2 or whichever it was. Don't get me wrong, the
backup API was a great enhancement and I use the hell out of it, but there is
more that could be done to make it extremely powerful for customized backup
operations.
As probably the heaviest user of the api on earth, I have asked for this stuff
a couple of times ready, but I thought I would try again.
There are two basic areas:
1. more callbacks and greater control of the backup operation
2. help with parsing the command line
Callbacks:
Nearly all of the existing callbacks are passive. Basically backup lets me
know when it does things, possibly with useful information, but there is very
little oportunity to change its behavior once the backup has started.
For instance, I would like to be able to specify the label that will go on the
tape in real time, on the first and especially subsequent volumes. To do that
now, I have to do system service and runtime library intercepts, a rather
fragile mechanism.
It would be awesome if there was a label generation callback. When backup is
in the process of deriving a label from the saveset name, use the callback to
see if I want to provide you with a label. If I do, forget the generated label
and write what I give you on the tape.
This would eliminate 90% of the special purpose code in the tapesys backup
module and would allow me to use established/supported interfaces for greater
reliability and immunity to vms version changes.
The backup command line:
The backup command is fairly complex and has a lot of qualifiers. The api does
not deal with the command line at all and expects all of the values to be
already parsed and placed into a fixed length parameter list, similar to the
item lists used for various system services. I guess the assumption is that
all custom backup programs will generate this list dynamically from internal
values. In truth I have done this in simple programs used for testing.
However, an equally likely scenario is that the custom backup program is a
wrapper around regular vms backup. This means that all backup parameters and
qualifiers are identical to those used by straight backup, even if the wrapper
doesn't do anything with them. So basically the same parsing is done,
resulting in an identical parameter list passed to backup$start in the api.
Unfortunately, the code to do this is in backup.exe, *not* backupshr. So it is
not available to someone using the api. All of the command line parsing and
parameter list building code has to be replicated from scratch, even though it
is logically identical to what is in backup.exe. Worse, you have to monitor
new versions of backup for new qualifiers and modify your code to add them to
the list. If you forget to do it, the customers complain that the new
qualifiers are being ignored.
It would be a great service to the api user to simply *move* the
parameter/qualifier processing code from backup.exe to backupshr and make it
available as a callable procedure (parse_backup_command_line or some such). No
input is required; the routine can simply use the cli routines to check all the
parameters/qualifiers as it always does. The output would be the parameter
list passed to backup$start.
So the custom backup program would call parse_backup_command_line, massage the
generated parameter list if needed, and call backup$start.
backup.exe would do basically the same thing. IIRC from examining the source,
the parsing code is a called procedure already, so this would be a case of
simply moving that procedure from one module to another and giving it a
transfer vector in the shareable image. And adding more error checking for us
irrational, unpredictable users, of course. :-)
I agree 100%. In fact, even BACKUP.EXE should have better label
handling. If you are not sure how many tapes your backup will require,
you need to still allocate a whole bunch of tapes and specify them in
the backup command and after the backup has completed, you need to
search the log file to see which tapes were actually used.
In a tape library with pre-initialised tapes, you don't want backup to
change tape labels on you.
Looks to me like current tape drives are hitting 100+ MB/sec.
I find it hard to believe any VMS system can keep 1 such drive
spinning much less 2.
In particular, backup's inability to do simultaneous read & write
(as mentioned by the Italians) has been a real killer.
> What you want is exactly what I don't want. The current behaviour
> lets me start backups overnight, change tapes in the morning, and
> rest assured that ALL my data is safely backed up.
I wasn't suggesting it as universal behaviour - your wishes are
perfectly reasonable.
But I don't want my backup starting tape 2 and filling it in prime-time.
I do want everything finished first thing, and I'm content with a
partial backup. I want an option to behave this way.
> Meanwhile, one simple "reply/abort" and you can actually get what
> you're looking for.
>
> OPCOM is a great friend.
That's all very well, but it requires someone logged in, privileged, and
trained.
Chris
Not really. If you have no operator (means also not OPA0 enabled) then the
requests aborts itself immediately with "no operator available..."
--
Peter "EPLAN" LANGSTOEGER
Network and OpenVMS system specialist
E-mail pe...@langstoeger.at
A-1030 VIENNA AUSTRIA I'm not a pessimist, I'm a realist
That example is reality, sensational or not.
Didn't particularly want to disable OPA0.
I can live with this (wrote nightwatchman to enable me to do so), but
don't like it.
Chris
This would interface with the XQO directly and any write IO to the disk
would get copied to tape sequentially with time stamps every X units of time.
This way, you could "replay" a disk's activity over a period of time and
stop at at the right time to take a look at what the disk looked like at
that very moment, then proceed and advance time and see what the disk
looked like later on etc etc.
(sort of a new definition of "incremental".)
Yep, this would probably require you start off with a BACKUP/PHYSICAL so
that all subsequent IOs would place the data in the same physical blocks.
Am I missing something? It's trivial to mount a disk into the shadow
set immediately after you dismount another one. Actually, if you have 2
offline, it shouldn't matter which is the oldest; surely if you have 3
members mounted at all times you won't have to rely on saving the
offline member which is the younger of the two (but still older than
your live disks). Nevertheless, you could mount them privately, get the
shadowing-generation number then mount the oldest into the shadow set.