Panic save very sloooooow
carl_...@hotmail.com
Swap space was not big enough to hold a memory image (2 gig) so last
weekend we put in a spare 10 gig and created an empty 5 gig partition
as our new default dump device (as per SCO TA).
Today we got our first panic. Operator said "Y" to dump to the
default and hit ENTER when it said to insert media. And nothing
happened for about eight minutes when it displayed one period. I was
not there, operator said we were getting some disk lights, but I
cannot attest to frequency. An hour later it had displayed a total of
nine periods and management made us reboot the system so users could
get back in.
A few days ago I posted here looking for a way to induce panics so I
could test this procedure but I decided that I should not do the
recommended changes on a production system. I did, however, use
sysdump to write a full memory image to /dev/dump and it took about
five minutes to run. Then I tested a script I wrote to preserve this
image in a dated disk file and this, too, took about 5 min.
Details:
SCO OSR 5.0.4 with all applicable patches, etc.
Four 800 mhz CPUs
2 gig RAM
Two SCSI RAID controllers:
- DAC960pg
- Adaptec/DPT 3400s - this is where the disk with the dump device
lives, though it is not part of an array.
Questions:
1) What does each period mean when it is saving the panic image?
2) Any ideas why it is so slow?
3) Any other ideas or suggestions (I'm open to anything!)?
4) I am thinking of paying Caldera to help us dig through the saved
dump files (once we have a couple) and diagnose the problem - any
thoughts on this?
Thanks Very Much!
Mike Brown
I never understood what was going on, but I did leave a server
for about 8 hours to finish a panic dump. It had not written
a full line of 'dots', so I gave up on it.
You seem to have a large investment in hardware, why are you
running such an old version of OS ?
Mike
--
Michael Brown
The Kingsway Group
Bela Lubkin
Each dot represents 1/80 of your total memory, so a complete dump should
fill one entire line of the console screen.
> 2) Any ideas why it is so slow?
Each dot is about 25MB on your system, so it might be slower than you
might be used to from another system. On the other hand, 8 minutes is
far too long for a mere 25MB.
Are there any other devices on the Adaptec/DPT 3400s controller where
the dump device lives? In other words: do you have any evidence that
this adapter is working normally?
You say you "created an empty 5 gig partition as our new default dump
device (as per SCO TA)". First, which TA? Second, show us how you've
informed the kernel of this -- excerpt from /etc/default/boot or
/etc/conf/cf.d/sassign. Finally, show us the device node for the device
you're dumping to, and output from `hwconfig -h`.
The dump could be slow due to the nature of the panic: if it was caused
by a problem in the disk drivers in the kernel, they might be in an
insane state. Or there may be a problem with interrupt delivery. I
believe the kernel dump code uses some tricks to continue writing even
if interrupts are not being received from the host adapter to which the
dump is being written; this would probably be slow (I'm not sure, have
never seen it in action).
> 3) Any other ideas or suggestions (I'm open to anything!)?
Can you tell whether the partial dump attempt wrote anything? Probably
not, but you can next time. Create a filesystem on the dump partition
or division; mount it; copy some files on; look at them. Do not set the
system up to automatically mount this filesystem (i.e. don't run `mkdev
fs` or whatever the scoadmin equivalent is). Just create enough stuff
to be recognizable. Then unmount it.
If the system panics, it should overwrite that "watchdog" filesystem.
If the slow dumping behavior persists, you'll probably kick out of it
before the dump completes, but at least you can tell whether it was
doing anything, by trying to mount that filesystem. If the dump was
writing anything it'll have trashed the beginning of the filesystem and
it'll be completely unrecoverable.
Another suggestion: link the kernel debugger into the kernel. Edit
/etc/conf/sdevice.d/scodb, change "N" to "Y". Now relink the kernel
_twice_ (it'll change some data table sizes the first time), then
reboot.
Once scodb is in, you have a new way to test, and new behavior in a
panic. To test: boot the system to single-user mode. Hit control-X on
the console to get into scodb. At the scodb prompt, type "sysdump()".
This calls the kernel function that writes a panic dump. It should go
through the usual motions. Does it? Do the dots move across the screen
at a reasonable pace? (Let it finish if it's fast; hit RESET or power
if it's slow, and go figure out why...)
New behavior: if the system panics, it will drop to a scodb prompt. "?"
gives brief help. You can get a call traceback with "stack". If you
can transcribe the panic traceback, someone here might be able to see
what the problem is.
NOTE: if the console is in graphics mode (X or DOS Merge, for instance),
a panic will drop to scodb, but you won't be able to see what you're
doing. It's best to leave the console in character mode when you're
trying to catch a panic in scodb.
> 4) I am thinking of paying Caldera to help us dig through the saved
> dump files (once we have a couple) and diagnose the problem - any
> thoughts on this?
Sounds good to me. ;-}
>Bela<
Mike Brown
-snip-
> NOTE: if the console is in graphics mode (X or DOS Merge, for instance),
> a panic will drop to scodb, but you won't be able to see what you're
> doing. It's best to leave the console in character mode when you're
> trying to catch a panic in scodb.
Must of missed the documentation on that gotcha, thanks for the advise.
carl_...@hotmail.com
>
>You seem to have a large investment in hardware, why are you
>running such an old version of OS ?
>
Thanks for your thoughts. The application is only certified to this
level of the opsys so we have no choice. Believe me, I would VERY
MUCH like to be on 5.0.6, or even .5, if only for the SCSI drivers...
So it goes.
carl_...@hotmail.com
wrote:
Bela!
Where to start? Thanks for the excellent information. Your thought
and time are greatly appreciated!
>Are there any other devices on the Adaptec/DPT 3400s controller where
>the dump device lives? In other words: do you have any evidence that
>this adapter is working normally?
Yes and the controller is fine. After rebooting I spent some time
toying with the system and suddenly realized that the automounted
filesystem on the back half of the disk was not mounted. It appears
that the disk died, and neither the default dump space nor the
filesystem were available. It appears to have just quit spinning some
time the night befoe. I found about 300k of error messages in my
backup log when it went offline while the nightly backup was copying
files from the filesystem. This further proves that the Machine God
has a sense of irony, and He likes to pick on ME!
>The dump could be slow due to the nature of the panic: if it was caused
>by a problem in the disk drivers in the kernel, they might be in an
>insane state.
We thought about something along these lines, so we changed the
default dump device to our DLT tape. It's on another controller and
we should be OK next time. Of course, now it has not had a panic in
days! More Divine Irony, I guess.
>Can you tell whether the partial dump attempt wrote anything? Probably
>not, but you can next time. Create a filesystem on the dump partition...
I'll go you one better. I thought of prepping my dump device by using
dd to write hex FFs to the whole thing while the system is up and
running. I guess I could do the same for the tapes, too.
>Another suggestion: link the kernel debugger into the kernel. Edit
>/etc/conf/sdevice.d/scodb, change "N" to "Y". Now relink the kernel
>_twice_ (it'll change some data table sizes the first time), then
>reboot.
I can't play around with this production system at all, but I've set
up a little 6 gig disk on an IDE in a P233 with SCO 5.0.4 to serve as
a testbed. I'm planning to put a spare DDS2 in it so I can try some
of this stuff out. There is a TA on linking your kernel so you can
produce a panic at will, and I plan to wring it out this weekend.
>> 4) I am thinking of paying Caldera...
>
>Sounds good to me. ;-}
As quickly as possible, I need to cut to the chase on this one. If I
get time to experiment with panics and using crash and scodb before I
bring in Caldera all the better but I need to get this one solved as
soon as I can. A couple of the panic screens have listed registers
and these have been transcribed (I hope faithfully) by the onsite
staff and I need to dig through these, too. You may find me here
again soon on this issue, and I'll post the solution (I'm SURE I'll
find it!) too...
Thanks again for the excellent input.
Mike Brown
The application should not care, in my experience the major change
has been the SCO..SCOANSI..ANSI console emulation changes. I have
tested MANY apps that claimed 5.0.0 compatibility under 5.0.4/5/6
and found no problems. The vendor should be happy to have support!
There has been some install hard/soft link problems, but in general
5.0.4/5/6 are the same.
Fight it out.
Nachman Yaakov Ziskind
FWIW (not very much) Veritas' Backup Exec ver. 7.3 has a SCO backup agent which
runs with 5.0.5 but not with 5.0.6. Inquires to the vendor produced the usual
"we only support 5.05."
Whatever.
--
_________________________________________
Nachman Yaakov Ziskind, EA, LLM aw...@egps.com
Attorney and Counselor-at-Law http://yankel.com
Economic Group Pension Services http://egps.com
Actuaries and Employee Benefit Consultants
Bela Lubkin
> FWIW (not very much) Veritas' Backup Exec ver. 7.3 has a SCO backup agent which
> runs with 5.0.5 but not with 5.0.6. Inquires to the vendor produced the usual
> "we only support 5.05."
But how does it fail? Whatever the problem, it can probably be fixed by
you, without their help.
>Bela<
Nachman Yaakov Ziskind
| Nachman Yaakov Ziskind wrote:
|
| > FWIW (not very much) Veritas' Backup Exec ver. 7.3 has a SCO backup agent
| > which runs with 5.0.5 but not with 5.0.6. Inquires to the vendor produced
| > the usual "we only support 5.05."
|
| But how does it fail?
It dies silently. It comes with a debug facility, which reported nothing
useful. Since I use B/Edge for the Unixen, I don't worry about it too
much ...
| Whatever the problem, it can probably be fixed by you, without their help.
Ha ha ha. Ho ho ho. :-)
For all I know, it checks the version number and, not finding what it wants,
commits hara kari. :-(
BTW, is there a useful way to spoof the version number, should this be the
issue?
Jean-Pierre Radley
| Bela Lubkin wrote (on Mon, Feb 25, 2002 at 08:05:40AM +0000):
| | Nachman Yaakov Ziskind wrote:
| |
| | > FWIW (not very much) Veritas' Backup Exec ver. 7.3 has a SCO backup agent
| | > which runs with 5.0.5 but not with 5.0.6. Inquires to the vendor produced
| | > the usual "we only support 5.05."
| |
| | But how does it fail?
|
| It dies silently. It comes with a debug facility, which reported nothing
| useful. Since I use B/Edge for the Unixen, I don't worry about it too
| much ...
|
| | Whatever the problem, it can probably be fixed by you, without their help.
|
| Ha ha ha. Ho ho ho. :-)
|
| For all I know, it checks the version number and, not finding what it wants,
| commits hara kari. :-(
|
| BTW, is there a useful way to spoof the version number, should this be the
| issue?
Make you own (temporary) uname command, and have it return
Release=XYZ, where XYZ is other than 3.2v5.0.6.
--
JP
Nachman Yaakov Ziskind
Oh? And do you really think that a complied binary utilizes a command like that?
Undoubtedly, there is a programmatic equivalent (the uname(C) man pages refer me
to uname(S), which I don't seem have on my 5.0.6 system), and creating a shell
script will accomplish, what? A call to a function won't invoke a shell script,
right?
I wish I understood programming better.