Fun in the Sun with Solaris 9
Nigel P. Longbottom
I got the machine and put a Sun Ultra-3 card in it attached to a SDLT 320
Sun drive - I wanted fast backups as it contained 6 x 72Gb drives.
As the internal drives are not hardware RAID-5 and I wanted resilience, I
decided to go for Solaris 9's Disksuite aka Sun Volume Manager and would
mirror the disks. Even though they are on the same controller, this would at
least help with disk failures.
All fine so far - installed Solaris 9, latest patches etc, then configured
DiskSuite to mirror root and the other disks.
Stuffing a SDLT tape in the drive, I excitedly ran a ufsdump on all the UFS
filesystems and waited... And waited... I discovered I was getting 4Mb/s
backup speeds even on slices with large Oracle files (approx 35Gb).
Hmm -not good. Luckily we had Sun Gold Support so logged a call and received
a helpful chap who, after getting all the relevant details i.e. Explorer
files etc, said that the problem was that I was doing a UFS dump with the
filesystem mounted and that Sun didn't recommend that.
I asked if he was serious and he suggested that if I unmounted the
filesystems, I could get a proper backup. What about / I inquired - how do I
do that? Well, if you boot off a CD, you can back up the system that way? So
if I want to back up a Sun system, I have to drop to single user mode and
manually run a ufsdump command! What a load of tosh! He was persisitant and
stuck by his guns insisting that was the problem!
In the meantime - I had been doing some experimentation myself and
discovered that if I turned on logging and changed the mirroring read
strategy from round-robin to geometric, I got backup read speeds of between
10Mb/s to 30Mb/s (compressed)!
The moral is - do Sun really want to keep their customers by telling them
that the only way to backup a system is to stand by it and load CD's then
manually type the commands in? If someone from Sun is reading this - you
really need to get your people-facing guys sorted! IBM, Apple, HP and
Microsoft would be pissing themselves if this was the only way they
recommended people to backup their systems!!
Rich Teer
> Hmm -not good. Luckily we had Sun Gold Support so logged a call and received
> a helpful chap who, after getting all the relevant details i.e. Explorer
> files etc, said that the problem was that I was doing a UFS dump with the
> filesystem mounted and that Sun didn't recommend that.
>
> I asked if he was serious and he suggested that if I unmounted the
> filesystems, I could get a proper backup. What about / I inquired - how do I
> do that? Well, if you boot off a CD, you can back up the system that way? So
> if I want to back up a Sun system, I have to drop to single user mode and
> manually run a ufsdump command! What a load of tosh! He was persisitant and
> stuck by his guns insisting that was the problem!
IF you backup a file system while it is busy, how do you expect
it to be reliable and consistant? However, using UFS snapshots
might help in this scenario.
> In the meantime - I had been doing some experimentation myself and
> discovered that if I turned on logging and changed the mirroring read
> strategy from round-robin to geometric, I got backup read speeds of between
> 10Mb/s to 30Mb/s (compressed)!
Very good. Now, what has that got to do with backups?
> The moral is - do Sun really want to keep their customers by telling them
> that the only way to backup a system is to stand by it and load CD's then
Who on earth seriously uses CDs as a backup medium?
> manually type the commands in? If someone from Sun is reading this - you
Manually? No, sysadmins with a clue use a script, preferably
driven from cron.
> really need to get your people-facing guys sorted! IBM, Apple, HP and
> Microsoft would be pissing themselves if this was the only way they
> recommended people to backup their systems!!
Or perhaps you could purchase some 3rd party application, like
Veritas, to perform your backups.
And I'm sure a datacenter manager would be pissing himself if you
told him you wanted to backup a multi-GB file system using CDRWs...
Well, after he fired you, anyway.
--
Rich Teer, SCNA, SCSA
President,
Rite Online Inc.
Voice: +1 (250) 979-1638
URL: http://www.rite-online.net
Neil W Rickert
Hash: SHA1
Rich Teer <rich...@rite-group.com> writes:
>On Wed, 21 Jul 2004, Nigel P. Longbottom wrote:
>Very good. Now, what has that got to do with backups?
>> The moral is - do Sun really want to keep their customers by telling them
>> that the only way to backup a system is to stand by it and load CD's then
>Who on earth seriously uses CDs as a backup medium?
I think you misread that. He is talking about booting from CD so
that he can backup with the filesystems unmounted. That's where the
"load CD's" part comes in.
>> manually type the commands in? If someone from Sun is reading this - you
>Manually? No, sysadmins with a clue use a script, preferably
>driven from cron.
If you are first booting from the install CD's then it isn't going to
be easy with cron. Either the OP was given poor advice, or he
misinterpreted the advice he was given. I cannot tell which.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.3.6 (SunOS)
iD8DBQFA/wFfvmGe70vHPUMRAixvAJ98NzKLvq3tNNq1+Nn1olFyF1DxZQCg+cH5
m7FtuiH8IVYcKg+DSxaGTYQ=
=xBiP
-----END PGP SIGNATURE-----
Juhan Leemet
> On Wed, 21 Jul 2004, Nigel P. Longbottom wrote:
>> Hmm -not good. Luckily we had Sun Gold Support so logged a call and received
>> a helpful chap who, after getting all the relevant details i.e. Explorer
>> files etc, said that the problem was that I was doing a UFS dump with the
>> filesystem mounted and that Sun didn't recommend that.
>>
>> I asked if he was serious and he suggested that if I unmounted the
>> filesystems, I could get a proper backup. What about / I inquired - how do I
>> do that? Well, if you boot off a CD, you can back up the system that way? So
>> if I want to back up a Sun system, I have to drop to single user mode and
>> manually run a ufsdump command! What a load of tosh! He was persisitant and
>> stuck by his guns insisting that was the problem!
>
> IF you backup a file system while it is busy, how do you expect
> it to be reliable and consistant? However, using UFS snapshots
> might help in this scenario.
Yes, that's true, I understand "fssnap" can be used to get a reliable and
consistent backup for ufs filesystems, but...
In this case, he has actually mirrored everything, right? He said
"...configured DiskSuite to mirror root and the other disks."
What about if he were to "metaoffline" his mirrors? One at a time, natch.
Couldn't he then get a backup of a static "offline" mirror? Then when he
puts the mirror back "metaonline" then it will sync up again. The man page
for metaoffline even suggests this as a mechanism for doing online backups!
I can't see any extra risk here, except perhaps on the "work in progress"
while he has 1/2 mirror offline. If he loses one of the disks, he will
either have the "static copy", i.e. like after restoring backup, or he
will have the "real" dynamic copy, and he just syncs a new mirror.
Wouldn't that work? I'm planning to do this myself. Soon! Priorities...
In my case, I'm going around and mirroring all my root disks, and
then I'll be able to use the offline/online trick to get a clean
backup of / filesystem. The other ones I think I can umount, selectively.
--
Juhan Leemet
Logicognosis, Inc.
Paul Eggert
> Luckily we had Sun Gold Support so logged a call and received
> a helpful chap who, after getting all the relevant details i.e. Explorer
> files etc, said that the problem was that I was doing a UFS dump with the
> filesystem mounted and that Sun didn't recommend that.
>
> I asked if he was serious and he suggested that if I unmounted the
> filesystems, I could get a proper backup.
You have Sun Gold support. You have the right to competent advice,
but you're not getting it. Sounds like you should be going up the
management chain on this one. You wouldn't be doing Sun any favors by
letting this sort of service go without complaint.
If you'd like better advice about how to do backups of Oracle on
Solaris, you might start with the Sun Blueprint series of
best-practice documents. E.g., see Art Licht's paper
<http://www.sun.com/blueprints/0802/816-7395-10.pdf> and Selim Daoud's
<http://www.sun.com/blueprints/0702/816-5243-10.pdf>.
Rich Teer
> I think you misread that. He is talking about booting from CD so
> that he can backup with the filesystems unmounted. That's where the
> "load CD's" part comes in.
AH yes, that makes more sense. Thanks for the (obviously needed!)
interpretation!
Rich Teer
> Yes, that's true, I understand "fssnap" can be used to get a reliable and
> consistent backup for ufs filesystems, but...
>
> In this case, he has actually mirrored everything, right? He said
> "...configured DiskSuite to mirror root and the other disks."
>
> What about if he were to "metaoffline" his mirrors? One at a time, natch.
> Couldn't he then get a backup of a static "offline" mirror? Then when he
> puts the mirror back "metaonline" then it will sync up again. The man page
> for metaoffline even suggests this as a mechanism for doing online backups!
Yes, that would also work. Off the top of my head, I'm not sure
what all the pros and cons of fssnap vs metaoffline are, although
one obvioous benefit of the latter is that it'll work with any
file system, not just UFS (IIRC, only UFS supports fs snapshots
at present. I would be very surprised if DFS didn't, unless it
wasn't necesary by design...).
> I can't see any extra risk here, except perhaps on the "work in progress"
> while he has 1/2 mirror offline. If he loses one of the disks, he will
> either have the "static copy", i.e. like after restoring backup, or he
> will have the "real" dynamic copy, and he just syncs a new mirror.
True. And this risk can be mitigated by using 3-way mirrors.
That way, when you break off one of the sub-mirrors to back it
up, the "live" file system is still safe in a 2-way mirror.
It's one of those cost/benefits type of things.
Frank Cusack
> In the meantime - I had been doing some experimentation myself and
> discovered that if I turned on logging and changed the mirroring read
> strategy from round-robin to geometric, I got backup read speeds of between
> 10Mb/s to 30Mb/s (compressed)!
What is the geometric strategy? (use knowledge of proximity of last read
to last known head position?)
/fc
Nigel P. Longbottom
Cusack" <fcu...@fcusack.com> wrote:
Thanks for the response. After Richard's comments, I'm glad some people
jumped in to clarify. Don't worry - I'm not that bad to start backing up to
CD-RW and yes - my tone can be a bit subtle!
I suppose my real point was a dig at Sun support. My initial query was about
backup rates to tape but I ended up being told that the problem was due to
mounted filesystems. With the competition in UNIX markets, I would say Sun
is damaging itself by saying not to backup while filesystems are mounted
(and they were adamant about this). I've been doing this for years
(obviously while the filesystems are quiet i.e. Oracle is shut down and the
servers are not active 24x7) and have no problems when restoring from these
backups in a Disaster Recovery exercise.
The other systems I work on - namely IBM AIX are quite happy to suggest you
backup when the filesystems are 'quiescent' (as they happily term it).
As for geometric read strategy - it's one of the options that you can assign
to mirrors using the metaparam command. Its recommended if you have large
sequential reads.
Alan Coopersmith
|I suppose my real point was a dig at Sun support. My initial query was about
|backup rates to tape but I ended up being told that the problem was due to
|mounted filesystems. With the competition in UNIX markets, I would say Sun
|is damaging itself by saying not to backup while filesystems are mounted
|(and they were adamant about this). I've been doing this for years
|(obviously while the filesystems are quiet i.e. Oracle is shut down and the
|servers are not active 24x7) and have no problems when restoring from these
|backups in a Disaster Recovery exercise.
It's been a long time (almost 9 years) since I worked in Sun tech support
answering questions about backups, but I think the answer we gave then
is still basically true today - and is close to, but more detailed than
what you seem to have been told. The party line then was not "don't do
backups of mounted filesystems", but was actually "*ufsdump* of a mounted
filesystem is not reliable due to the way it scans the UFS structure
then dumps the data, going in below the filesystem level and may get
very confused if the filesystem layout changes between the read of the
UFS structures and the read of the data. To backup a filesystem without
unmounting, you can mirror it, then break the mirror temporarily, backup
the mirror and then resync, or use a different backup utility that is
designed to cope with change, such as Solstice Backup/Legato Networker."
The biggest change I know of since then is the addition of ufs snapshots
as yet another option.
--
________________________________________________________________________
Alan Coopersmith * al...@alum.calberkeley.org * Alan.Coo...@Sun.COM
http://www.csua.berkeley.edu/~alanc/ * http://blogs.sun.com/alanc/
Working for, but definitely not speaking for, Sun Microsystems, Inc.
Rich Teer
> I suppose my real point was a dig at Sun support. My initial query was about
> backup rates to tape but I ended up being told that the problem was due to
> mounted filesystems. With the competition in UNIX markets, I would say Sun
> is damaging itself by saying not to backup while filesystems are mounted
> (and they were adamant about this). I've been doing this for years
I think Sun support likes to err on the side of caution.
> The other systems I work on - namely IBM AIX are quite happy to suggest you
> backup when the filesystems are 'quiescent' (as they happily term it).
Using ufsdump on a quiescent file system is also "fairly" safe on
Solaris, but the advent of UFS snapshots has made this practice
obsolete.
Frank Cusack
> On 22/7/04 7:03 am, in article m3k6wwd...@magma.savecore.net, "Frank
> Cusack" <fcu...@fcusack.com> wrote:
>
>> What is the geometric strategy? (use knowledge of proximity of last read
>> to last known head position?)
>>
> to mirrors using the metaparam command. Its recommended if you have large
> sequential reads.
Thanks, but that doesn't answer the question. What *is* the geometric
strategy? metaparam(1M) doesn't say. Didn't find anything on Google.
/fc
Frank Cusack
Which I just noticed, is a strange omission. The other two algorithms are
described, although they are self-explanatory.
/fc
Juhan Leemet
> ...Sun is damaging itself by saying not to backup while filesystems are
> mounted (and they were adamant about this). I've been doing this for
> years (obviously while the filesystems are quiet i.e. Oracle is shut
> down and the servers are not active 24x7) and have no problems when
> restoring from these backups in a Disaster Recovery exercise.
The reason might be legal? If Sun support (under contract, being paid)
tells you that you can do a backup in a "dangerous" way, and you lose
some critical data or system, you might be tempted to sue? Malpractice or
some such: giving paid advice which was innacurate or led to the loss of
revenue, or whatever. So they may be "ass covering" to some extent.
I have also been doing this: (ufs)dump of "quiescent" mounted file systems
for years and years. I was quite surprised when I more recently read about
the extent of the "corruption" that might occur on a backup if the file
system changes while ufsdump is working "underneath the covers" (under the
filesystem, on the raw disk). Scary stuff. Now, we normally would keep a
couple of generations of full backups, but still, depending on how
"quiescent"?!? You could end up in a world of pain, as they say.
I'm most interested these days in using metaoffline for backup, since I'm
moving towards mirroring all my root disks. I can dismount/remount other
filesystems (e.g. RAID5) for the duration of its backup. More relaxed!
but IANAL
Nigel P. Longbottom
Cusack" <fcu...@fcusack.com> wrote:
>> Thanks, but that doesn't answer the question. What *is* the geometric
>> strategy? metaparam(1M) doesn't say. Didn't find anything on Google.
>
> Which I just noticed, is a strange omission. The other two algorithms are
> described, although they are self-explanatory.
>
> /fc
I tried looking for what this strategy is and still couldn't find an answer.
The Sun Docs tell you when to use it i.e. Minimal multiple access on large
sequential files but they don't mention what it is doing! I can guess but
wouldn't have the definitive answer.
On another note, I have been playing with fssnap and have tried it out and
it works great. Only small issue I can see is that you need a free
filesystem that you are not going to backup because you can't backup a
filesystem where the snapshot resides. On new boxes, I can create a specific
'/snapshot' filesystem and grep that out of the list of filesystems to back
up but on the current boxes, I'll have to come up with a workaround.
Rich Teer
> On another note, I have been playing with fssnap and have tried it out and
> it works great. Only small issue I can see is that you need a free
> filesystem that you are not going to backup because you can't backup a
> filesystem where the snapshot resides. On new boxes, I can create a specific
True - but you only backup one file system at a time. So, use
one of your existing partitions as the home for the snap shot
when you back up all the other partitions, and then use one of
the other partitions for the snap shot when you back up the
partition you usually use for snap shots.
Nigel P. Longbottom
Pine.SOL.4.58.04...@zaphod.rite-group.com, "Rich Teer"
<rich...@rite-group.com> wrote:
> On Sat, 24 Jul 2004, Nigel P. Longbottom wrote:
>
>> On another note, I have been playing with fssnap and have tried it out and
>> it works great. Only small issue I can see is that you need a free
>> filesystem that you are not going to backup because you can't backup a
>> filesystem where the snapshot resides. On new boxes, I can create a specific
>
> True - but you only backup one file system at a time. So, use
> one of your existing partitions as the home for the snap shot
> when you back up all the other partitions, and then use one of
> the other partitions for the snap shot when you back up the
> partition you usually use for snap shots.
True - but I was wondering if there is a neater way. I usually use a FOR
loop to decide which filesystems to backup i.e.
For FSYS in `df -Ik ufs | cleanup the output`
Do
ufsdump 0uf $TAPE `fssnap -o raw,bs=<snapsys> $FSYS`
some other checks
Done
As you can see, the 'snapsys' location would have to change if and when I
came to the existing filesystem that contained the snapshots. As such, I'd
have to add a new check i.e. If $FSYS = $snapsys then change snapsys to
something else temporarily. I'm wondering if there is a neater way.
Rene Stepanek
> True - but I was wondering if there is a neater way. I usually use a FOR
> loop to decide which filesystems to backup i.e.
>
> For FSYS in `df -Ik ufs | cleanup the output`
> Do
> ufsdump 0uf $TAPE `fssnap -o raw,bs=<snapsys> $FSYS`
> some other checks
> Done
>
> As you can see, the 'snapsys' location would have to change if and when I
> came to the existing filesystem that contained the snapshots. As such, I'd
> have to add a new check i.e. If $FSYS = $snapsys then change snapsys to
> something else temporarily. I'm wondering if there is a neater way.
I use a mounted nfs volume for my snapshots (and the backups). Works fine
so far.
Rene
Akop Pogosian
> "Nigel P. Longbottom" <n...@all.valid> writes in comp.sys.sun.admin:
> |I suppose my real point was a dig at Sun support. My initial query was about
> |backup rates to tape but I ended up being told that the problem was due to
> |mounted filesystems. With the competition in UNIX markets, I would say Sun
> |is damaging itself by saying not to backup while filesystems are mounted
> |(and they were adamant about this). I've been doing this for years
> |(obviously while the filesystems are quiet i.e. Oracle is shut down and the
> |servers are not active 24x7) and have no problems when restoring from these
> |backups in a Disaster Recovery exercise.
> It's been a long time (almost 9 years) since I worked in Sun tech support
> answering questions about backups, but I think the answer we gave then
> is still basically true today - and is close to, but more detailed than
> what you seem to have been told. The party line then was not "don't do
> backups of mounted filesystems", but was actually "*ufsdump* of a mounted
> filesystem is not reliable due to the way it scans the UFS structure
> then dumps the data, going in below the filesystem level and may get
> very confused if the filesystem layout changes between the read of the
> UFS structures and the read of the data. To backup a filesystem without
> unmounting, you can mirror it, then break the mirror temporarily, backup
> the mirror and then resync, or use a different backup utility that is
But how do you know the file system data structures on the disk are in
a consistent state at the moment when you're detaching the mirror?
Shouldn't the file system be unmounted first before detaching the
mirror?
-akop
Juhan Leemet
> Alan Coopersmith <al...@alum.calberkeley.org> wrote:
>> "Nigel P. Longbottom" <n...@all.valid> writes in comp.sys.sun.admin:
>> |...not to backup while filesystems are mounted...
>
>> ..."*ufsdump* of a mounted filesystem is not reliable due to the way it
>> scans the UFS structure then dumps the data, going in below the
>> filesystem level and may get very confused if the filesystem layout
>> changes between the read of the UFS structures and the read of the
>
> But how do you know the file system data structures on the disk are in a
> consistent state at the moment when you're detaching the mirror?
> Shouldn't the file system be unmounted first before detaching the
> mirror?
That is an interesting question. I expected to see a note in the
metadetach man page explicity saying that a sync is done before the mirror
is actually detached, to ensure that these system data structures on
the mirror disk are consistent. There is no such explicit note. It would
only make sense (to me) if there was a sync. Can anyone confirm?
Frank Batschulat
you know this when you issued a lockfs(1M) -f /filesystem
before doing the detach.
---
frankB
Frank Batschulat
[...]
> what you seem to have been told. The party line then was not "don't do
> backups of mounted filesystems", but was actually "*ufsdump* of a mounted
> filesystem is not reliable due to the way it scans the UFS structure
> then dumps the data, going in below the filesystem level and may get
> very confused if the filesystem layout changes between the read of the
> UFS structures and the read of the data. To backup a filesystem without
this is actually quite correct, ufsdump goes directly to the raw device
when scanning the filesystem meta data, which are usually on a mounted
filesystem held in the buffer cache for the most recent changes and
the buffer cache gets written from time to tome out to disk, thus ufsdump
likely can live in the past when doing a backup of the mounted filesystem
and thus it's manpage already states this possibility,
> unmounting, you can mirror it, then break the mirror temporarily, backup
> the mirror and then resync, or use a different backup utility that is
dont forgott to issue lockfs(1M) -f to flush the filesystems meta data
and in case logging is enabled to roll the log prior to breaking the mirror.
---
frankB
Alan Coopersmith
|dont forgott to issue lockfs(1M) -f to flush the filesystems meta data
|and in case logging is enabled to roll the log prior to breaking the mirror.
Right - I had forgotten about that in the years since I've had to do this.