Server crashes - need help! :(
Farlander
I'm having a HUGE problems with SCO OpenServer 5.0.6. I give up, I
don't know where else to look: it keeps crashing every single day,
sometimes a couple of times a day.
Here's the situation: I have 12 Dual P3/1000MHz and one Single
P4/2000MHz servers. Every server has a number of Dialogic and Antares
boards in it. Every Dual P3 CPU server crashes with kernel panic in
either 'freeb' or 'msgcount'. And Single P4 CPU server crashes with
different reasons every time: sometimes it is "clkrepdone from
0xf01a24fa in hlted", sometimes - something else (need to look up all
the memory dumps now to recall)...
I've tried everything that I could - different STREAMS and Dialogic
drivers, different patches, different kernel settings - nothing seems to
have helped...
Does anybody knows whatta heck is going on with the servers??? I give
up - I spent a couple of months trying to figure it out :(
Any help would be GREATLY appreciated!
Regards,
Farlander.
Rainer Zocholl
>Hello, everyone!
> I'm having a HUGE problems with SCO OpenServer 5.0.6.
Since when?
>I give up, I don't know where else to look:
>it keeps crashing every single day,
>sometimes a couple of times a day.
Did they ever run stable?
What was changed after that?
ANY change may be relevant!
>Here's the situation:
>I have 12 Dual P3/1000MHz and one Single P4/2000MHz servers.
>Every server has a number of Dialogic and Antares boards in it.
Those?
http://www.intel.com/network/csp/products/1871web.htm
Antares 2000/50, 3000/50, 6000/50 ISA Speech Platform
This document describes a legacy product.
How strong is the power supply?
Each board took (at least) 2,6A from 5V!
>Every Dual P3 CPU server crashes with kernel panic in
>either 'freeb' or 'msgcount'.
>And Single P4 CPU server crashes with different reasons every time:
>sometimes it is "clkrepdone from 0xf01a24fa in hlted",
>sometimes - something else (need to look up all
>the memory dumps now to recall)...
all patches applied?
> I've tried everything that I could -
>different STREAMS and Dialogic drivers,
>different patches,
>different kernel settings -
>nothing seems to have helped...
CPU too fast? (no joke)
> Does anybody knows whatta heck is going on with the servers??? I
>give up - I spent a couple of months trying to figure it out :(
They never run stable since month?
13 boxes?
Wow...
> Any help would be GREATLY appreciated!
Simple attemps:
Change the RAM stripes
Change the powersuplies, check voltages for transientes
Remove as much hardware as possible. (Including all Antares boards).
Get in contact with board vendor.
Bela Lubkin
At least post the kernel stack traces...
>Bela<
Farlander
>> I'm having a HUGE problems with SCO OpenServer 5.0.6.
>
>
> Since when?
Ever since we switched to Dual CPUs. When all servers were single
P3-s everything worked fine. Duals - crash after crash after crash...
Same story with P4, only different kind of crashes now. On dual CPUs
it's always dual kernel panic, on single - it's just panic with 'Press
return to reboot' at the end.
> Did they ever run stable?
These particular servers? No. However sometimes they might work for a
month without crashes, sometimes they crash a couple of times per day :(
> What was changed after that?
> ANY change may be relevant!
The turning point was when we switched to dual CPUs. Ever since it
was a problem. The higher load on the servers - the higher the
possibility of a crash. And like I said, on dual CPUs all crashes are of
the same kind - either in 'freeb' or in 'msgcount'. And it looks like I
give up on P4s - doesn't handle the load (two P3s 1000 MHz have ~70% CPU
Idle most of the time under the same load, under which P4/2000MHz has
0-10% of CPU idle and it already slows down some functions. So for now
I'm looking into fixing the dual CPU ones...
> Those?
> http://www.intel.com/network/csp/products/1871web.htm
Yup...
> How strong is the power supply?
400w...
> Each board took (at least) 2,6A from 5V!
Hmmm... I need to check that. But even if it is power - why would it
always crash on the same function - freeb or msgcount? I figure it would
crash randomly...
> all patches applied?
OSS642a - Cron supplement (ver 1.0.0)
OSS643A: Socket Driver Supplement for OpenServer 5 (ver 1.0.0)
OSS645A: Audit Subsystem Security Supplement (ver 1.0.0)
OSS648A: Processor Supplement for OpenServer 5.0.6 (ver 1.0.0)
RS506A: Release Supplement for SCO OpenServer Release 5.0.6 (ver rs5)
RS506A: Software Manager Supplement (ver rs506a)
> CPU too fast? (no joke)
Well... Dual CPU, P3/1000MHz, 512 megz of RAM. Another one -
P4/2000MHz, performance is worse, so I'm giving up on this one. I don't
know whether 1000 MHz is 'too fast' or not :)
> They never run stable since month?
Nope... The higher the load - the sooner it's going to crash.
> Simple attemps:
> Change the RAM stripes
Won't work - the brand new server with DDR266 memory crashes in
exactly the same fashion, only still can write 'Press return to reboot'
at the end :)
> Change the powersuplies, check voltages for transientes
That I'll check, but something tells me that crashes would be in
random places should this be the case...
> Remove as much hardware as possible. (Including all Antares boards).
Can't do that, unfortunately :(
> Get in contact with board vendor.
With Dialogic? Been there, done that... Looks like it is about time
to try again, though...
Thanks!
Regards,
Farlander.
Farlander
Ok :) Here's the most frequent one:
KERNEL STACK TRACE FOR PROCESS 94:
STKADDR FRAMEPTR FUNCTION POSSIBLE ARGUMENTS
e0000844 e0000970 prf_task_s (0x4,0,0x1000,0xe)
e0000978 e0000994 cmn_err (0x3,got_RESERVEDFLT+0x26c,0xe,u+0x9d4)
e000099c e00009c8 k_trap (u+0x9d4)
e00009d4 kern_trap from 0xf0013ae5 in bcpalign
ax:dffda000 cx: 400 dx: 1ffda bx: 1000 fl: 10206 ds: 160
fs: 0
sp:e0000a04 bp:e0000a24 si:dffda000 di:c0120000 err: 0 es: 160
gs: 0
e00009dc e0000a24 bcpalign (tmpva_pages,0xc0120000,0x1000,0x1ffda)
e0000a2c e0000a50 dumpnextpa
(0xc0120000,u+0xb30,0x3,got_RESERVEDFLT+0x26c)
e0000a58 e0000b74 sysdump (0x4,0,0xfd8bd2b8,0xe)
e0000b7c e0000b98 cmn_err (0x3,got_RESERVEDFLT+0x26c,0xe,u+0xbd8)
e0000ba0 e0000bcc k_trap (u+0xbd8)
e0000bd8 kern_trap from 0xf005f234 in freeb
ax:ffffffff cx: 1 dx:f03560c4 bx:fd8bd2b8 fl: 10282 ds: 160
fs: 0
sp:e0000c08 bp:e0000c30 si:fd8c87d8 di: 0 err: 0 es: 160
gs: 0
e0000be0 e0000c30 freeb (0xfd8c87d8,0xfd8c87d8,0x1,0xf2d745e8)
e0000c38 e0000c48 freemsg (0xfd8c87d8,0xfd8c87d8,0xf2d745e8,0xf2d5c700)
e0000c50 e0000c88 sr_device (0xf2d5c700,0xfd8c87d8,0xfd8c87d8,0xf2d5c700)
e0000c90 e0000cb4 sramsendcm (0xf2d5c700,0xfd8c87d8,0xf27aaa00,0)
e0000cbc e0000cd4 _dlgn_send (0xfd8c87d8,0xfd8c87d8,0,0xfd8c87d8)
e0000cdc e0000cf4 _dlgn_putc
(0xf2d5c700,0xfd8c87d8,0xfce2df7c,streams+0x1998)
e0000cfc e0000d18 dlgnwput (0xfce2df7c,0xfd8c87d8,0xfce2bfb4,0)
e0000d20 e0000d44 putnext (0xfce2bfb4,0xfd8c87d8,streams+0x1998,0)
e0000d4c e0000d7c strputpmsg (inode+0x12de0,u+0xdcc,u+0xdc0,0)
e0000d84 e0000d9c strputmsg (inode+0x12de0,u+0xdcc,u+0xdc0,0)
e0000da4 e0000ddc msgio (0x2)
e0000de4 e0000de8 putmsg (0x80d5810,0x80d1b34,0x80d1ab4,0x80474d0)
e0000df0 e0000e10 systrap (u+0xe1c)
e0000e1c scall_noke from 0x80053348
ax: 56 cx: 4 dx: 0 bx: 80d5810 fl: 202 ds: 1f
fs: 0
sp:e0000e4c bp: 804742c si: 80d1b34 di: 80d1ab4 err: 56 es: 1f
gs: 0
And here's another one - for msgcount:
KERNEL STACK TRACE FOR PROCESS 87:
STKADDR FRAMEPTR FUNCTION POSSIBLE ARGUMENTS
e0000910 e0000a3c prf_task_s (0x4,0,0x1000,0xe)
e0000a44 e0000a60 cmn_err (0x3,got_RESERVEDFLT+0x26c,0xe,u+0xaa0)
e0000a68 e0000a94 k_trap (u+0xaa0)
e0000aa0 kern_trap from 0xf0013ae5 in bcpalign
ax:dffda000 cx: 400 dx: 1ffda bx: 1000 fl: 10206 ds: 160
fs: 0
sp:e0000ad0 bp:e0000af0 si:dffda000 di:c0110000 err: 0 es: 160
gs: 0
e0000aa8 e0000af0 bcpalign (tmpva_pages,0xc0110000,0x1000,0x1ffda)
e0000af8 e0000b1c dumpnextpa
(0xc0110000,u+0xbfc,0x3,got_RESERVEDFLT+0x26c)
e0000b24 e0000c40 sysdump (0x4,0,0,0xe)
e0000c48 e0000c64 cmn_err (0x3,got_RESERVEDFLT+0x26c,0xe,u+0xca4)
e0000c6c e0000c98 k_trap (u+0xca4)
e0000ca4 kern_trap from 0xf005fc7a in msgcount
ax: 0 cx: 78 dx: 0 bx: 0 fl: 10286 ds: 160
fs: 0
sp:e0000cd4 bp:e0000ce8 si:f3005e50 di: 7 err: 0 es: 160
gs: 0
e0000cac e0000ce8 msgcount
(0xfd8c84c0,0xfd4297f0,0xfd423e78,shlock_str_qnext)
e0000cf0 e0000d18 putnextqru (0,0,0x1,0)
e0000d20 e0000d44 queuerun (0xfd8c98a0,0x1)
e0000d4c e0000d54 runqueues
(u+0xdcc,inode+0x52d50,u+0x1148,region+0xcae0)
e0000d5c e0000d7c strputpmsg (inode+0x52d50,u+0xdcc,u+0xdc0,0)
e0000d84 e0000d9c strputmsg (inode+0x52d50,u+0xdcc,u+0xdc0,0)
e0000da4 e0000ddc msgio (0x2)
e0000de4 e0000de8 putmsg (0x80d5870,0x80d1b94,0x80d1b0c,u+0xe10)
e0000df0 e0000e10 systrap (u+0xe1c)
e0000e1c scall_noke from 0x80053348
ax: 56 cx: 2 dx: 0 bx: 80d5870 fl: 202 ds: 1f
fs: 0
sp:e0000e4c bp: 8047508 si: 80d1b94 di: 80d1b0c err: 56 es: 1f
gs: 0
I hope there's something useful in there that I'm missing...
Thanks!!!
Regards,
Farlander.
Rainer Zocholl
>Rainer Zocholl wrote:
>>> I'm having a HUGE problems with SCO OpenServer 5.0.6.
>> Since when?
> Ever since we switched to Dual CPUs.
>When all servers were single P3-s everything worked fine.
>Duals - crash after crash after crash...
Hm, i know that driver implementors sometimes forget
that there could be a second CPU accessing the hardware
at the "same" time...
this will happen especially under high load, of cause.
To make it clear:
If you turn the second CPU OFF in BIOS (or remove it or turn it off),
it is not crashing or was it "not craching" on older boards?
>Same story with P4, only different kind of crashes now.
But the P4 is a single IIRC, and double fast.
>On dual CPUs it's always dual kernel panic,
>on single -
Do you mean the P4 box with "single"?
>it's just panic with 'Press return to reboot' at the end.
That might have an other cause on the P4.
Have you increased the system logging level to see
how it goes to that "run" level?
>> Did they ever run stable?
> These particular servers? No. However sometimes they might work for
>a month without crashes, sometimes they crash a couple of times per
>day :(
:-(
>> What was changed after that?
>> ANY change may be relevant!
> The turning point was when we switched to dual CPUs.
>Ever since it was a problem.
>The higher load on the servers - the higher the possibility
>of a crash.
Ah, that's good! (To find the problem ;-))
>And like I said, on dual CPUs all crashes are
>of the same kind - either in 'freeb' or in 'msgcount'.
>And it looks like I give up on P4s - doesn't handle the load
>(two P3s 1000 MHz have ~70% CPU Idle most of the time under
>the same load,
load from what?
>under which P4/2000MHz has 0-10% of CPU idle and
>it already slows down some functions.
Wow. What are the boards doing? They have 4 -four- relatively
performant DSPs and still needs so much main CPU?
I have the feeling that the driver has a problem
with "software timed loop optimization" and not right implemented
spin locking.
Sometimes the programmer (on old CPUs) does not wait for a bit
to become active, because it was always active after fewer us
the -older- CPU needs to check at all, and that checking took
too much time.
If now a fast CPU is running, the bit is never set as
expected so the driver runs (later) into a wait loop which may
be much too long. Or it missed "pulsed" status bits (which are
"broken by design" but may be found (Assume: While DSP are busy
from the previous command, a status bit is set to "1". On old CPUs
it was sufficient to wait for the bit to become "0". On new -very fast-
CPUs the (now adays slow) DSPs did not have enough time to set the busy bit.
The CPU sees the (old) "0" and continues to sent the next command or what
ever. That may work realy stable for a long time (nextime the CPU asks
the bit is set from the preprevoius command, but the DSP or CPU idled),
but may cause the lost of parallel processing.
That would explain your performance problems with a single CPU.
(While one CPU is blocked in the -now- unlukely programed driver
the other CPU can still work. With only one CPU all is blocked.)
BTW:
Did you check the CPU heat?
If your boards generates such high loads and the cooling
is not good, the P4 will throttle it self.
>So for now I'm looking into fixing the dual CPU ones...
I think that there are different problems.
>> Those?
>> http://www.intel.com/network/csp/products/1871web.htm
> Yup...
The boards are ISA or PCI?
>> How strong is the power supply?
> 400w...
How many power (amperes) is on the 5V strap?
>> Each board took (at least) 2,6A from 5V!
> Hmmm... I need to check that. But even if it is power - why would
>it always crash on the same function - freeb or msgcount?
Good question. It did not become clear for me that it is always
the same funktion (same register, ever analysed the core dump (if there is
a core dump...)?)
>I figure it would crash randomly...
Maybe it depends on the "doing" of the software?
But if is really always almost the same function:
That's unlikely to be a hardware problem.
>> all patches applied?
> OSS642a - Cron supplement (ver 1.0.0)
> OSS643A: Socket Driver Supplement for OpenServer 5 (ver 1.0.0)
> OSS645A: Audit Subsystem Security Supplement (ver 1.0.0)
> OSS648A: Processor Supplement for OpenServer 5.0.6 (ver 1.0.0)
> RS506A: Release Supplement for SCO OpenServer Release 5.0.6 (ver rs5)
> RS506A: Software Manager Supplement (ver rs506a)
>> CPU too fast? (no joke)
> Well... Dual CPU, P3/1000MHz, 512 megz of RAM. Another one -
>P4/2000MHz, performance is worse, so I'm giving up on this one.
>I don't know whether 1000 MHz is 'too fast' or not :)
Hm, i assumed you had other, slower machines running before
without problems (except lack of performance/thruput).
>> They never run stable since month?
> Nope... The higher the load - the sooner it's going to crash.
>> Simple attemps:
>> Change the RAM stripes
> Won't work - the brand new server with DDR266 memory crashes in
>exactly the same fashion, only still can write 'Press return to
>reboot' at the end :)
Ok, if it's always the same function, then a hardware problem
is not very likely.
>> Change the powersuplies, check voltages for transientes
> That I'll check, but something tells me that crashes would be in
>random places should this be the case...
Yes, but have a look to the 5V.
"Modern" boards burdens the 3,3V more, so the power supllies delivers
less power(ampers) on the 5V.
>> Remove as much hardware as possible. (Including all Antares boards).
> Can't do that, unfortunately :(
Because the "panic" can be forced(really?) by "just" high load
it might be possible to make the box "allone" crashing, only
with the boards and no site equipment: Maybe the DSP software vendor has
a "stress" utility or knows a way to simulate stress (he should have
such softare).
Bela Lubkin
Well, that's clearly in the Dialog driver (and then a double-panic in
the panic dump writing code...!)
When it hits this double-panic (2nd panic in bcpalign()), has it printed
any of the dump-in-progress dots?
Again with the double-panic...
This doesn't have the Dialogic driver on the stack, but if it fed bad
data into a STREAMS queue, this could be related.
If the double-panics are happening after the dump has printed some dots,
there is something bad happening in the hardware. Something like a DMA
transfer being written to a wrong address, corrupting memory not owned
by the driver. The loop in sysdump() that calls dumpnextpage() and then
bcopy() (which we see here as "bcpalign") uses the same addresses over
and over. 0xc0110000 is the unchanging address of a disk buffer it's
using to stage writes. tmpva_pages is the unchanging virtual address at
which it is sequentially mapping every page of memory. The mapping
cannot fail (if no memory existed at that physical address, it would
just get all 0xff's). So if it double-panics after some dots have been
printed, something very strange is happening. In fact it's pretty
strange even if this is the first page.
What is the value of register CR2 in these dumps? That's the address it
got the fault on. Should be the same as either %esi or %edi in the last
trap frame in the stack trace (si:dffda000 di:c0110000 in the 2nd
example).
For that matter, how are you displaying these stacks?! Those are
crash(ADM) output. To get crash output on a panic, you would need a
finished panic dump, but these show the system going down in flames in
mid-dump! I could understand scodb traces, you could be using a serial
console and capturing the output, but crash output from a double-panic
in the dump code?!?
>Bela<
Fabio Giannotti
> -----Original Message-----
> From: listm...@xenitec.on.ca [mailto:listm...@xenitec.on.ca]On
> Sent: Sunday, December 29, 2002 12:50 PM
> To: distri...@xenitec.on.ca
> Subject: Re: Server crashes - need help! :(
>
>
> (Farlander) 29.12.02 in /comp/unix/sco/misc:
>
> >Hello, everyone!
>
> > I'm having a HUGE problems with SCO OpenServer 5.0.6.
>
> Since when?
>
> >I give up, I don't know where else to look:
>
> >it keeps crashing every single day,
> >sometimes a couple of times a day.
>
> Did they ever run stable?
> What was changed after that?
> ANY change may be relevant!
>
>
> >Here's the situation:
> >I have 12 Dual P3/1000MHz and one Single P4/2000MHz servers.
> >Every server has a number of Dialogic and Antares boards in it.
>
We had a problem with a dual-CPU system crashing and it turned out that
rs506a was installed before the SMP software was installed. I don't think
it says anything about fixing SMP problems in the rs506a docs, but it does.
Reinstalling rs506a solved the problem.
Fabio
Rainer Zocholl
>Rainer Zocholl wrote:
>>> I'm having a HUGE problems with SCO OpenServer 5.0.6.
>> Since when?
> Ever since we switched to Dual CPUs.
>When all servers were single P3-s everything worked fine.
>Duals - crash after crash after crash...
Hm, i know that driver implementors sometimes forget
that there could be a second CPU accessing the hardware
at the "same" time...
this will happen especially under high load, of cause.
To make it clear:
If you turn the second CPU OFF in BIOS (or remove it or turn it off),
it is not crashing or was it "not craching" on older boards?
>Same story with P4, only different kind of crashes now.
But the P4 is a single IIRC, and double fast.
>On dual CPUs it's always dual kernel panic,
>on single -
Do you mean the P4 box with "single"?
>it's just panic with 'Press return to reboot' at the end.
That might have an other cause on the P4.
Have you increased the system logging level to see
how it goes to that "run" level?
>> Did they ever run stable?
> These particular servers? No. However sometimes they might work for
>a month without crashes, sometimes they crash a couple of times per
>day :(
:-(
>> What was changed after that?
>> ANY change may be relevant!
> The turning point was when we switched to dual CPUs.
>Ever since it was a problem.
>The higher load on the servers - the higher the possibility
>of a crash.
Ah, that's good! (To find the problem ;-))
>And like I said, on dual CPUs all crashes are
>of the same kind - either in 'freeb' or in 'msgcount'.
>And it looks like I give up on P4s - doesn't handle the load
>(two P3s 1000 MHz have ~70% CPU Idle most of the time under
>the same load,
load from what?
>under which P4/2000MHz has 0-10% of CPU idle and
>it already slows down some functions.
Wow. What are the boards doing? They have 4 -four- relatively
performant DSPs and still needs so much main CPU?
I have the feeling that the driver has a problem
with "software timed loop optimization" and not right implemented
spin locking...(to proof: let it run with slower CPU clock resp. one CPU)
Sometimes the programmer (on old CPUs) does not wait for a bit
to become active, because it was always active after fewer us
the -older- CPU needs to check at all, and that checking took
too much time.
If now a fast CPU is running, the bit is never set as
expected so the driver runs (later) into a wait loop which may
be much too long. Or it missed "pulsed" status bits (which are
"broken by design" but may be found) (Assume: While DSP are busy
from the previous command, a status bit is set to "1". On old CPUs
it was sufficient to wait for the bit to become "0" again.
On new -very fast- CPUs the (now adays slow) DSPs did not have
enough time to set the busy bit.
The CPU sees the (old) "0" and continues to sent the next command or what
ever. That may work really stable for a long time (nextime the CPU asks
the bit is set from the preprevoius command, but the DSP or CPU idled),
but may cause the lost of parallel processing.
That would explain your performance problems with a single CPU.
(While one CPU is blocked in the -now- unlukely programed driver
the other CPU can still work. With only one CPU all is blocked.)
BTW(nothing to do with the panic):
Did you check the CPU heat?
If your boards generates such high loads and the cooling
is not good, the P4 will throttle itself...
>So for now I'm looking into fixing the dual CPU ones...
I think that there are different problems.
>> Those?
>> http://www.intel.com/network/csp/products/1871web.htm
> Yup...
The boards are ISA or PCI?
Doing DMA or not?
>> How strong is the power supply?
> 400w...
How many power (amperes) is on the 5V strap?
>> Each board took (at least) 2,6A from 5V!
> Hmmm... I need to check that. But even if it is power - why would
>it always crash on the same function - freeb or msgcount?
Good question. It did not become clear for me that it is always
the same funktion (same register, ever analysed the core dump (if there is
a core dump...)?)
>I figure it would crash randomly...
Maybe it depends on the "doing" of the software?
But if is really always almost the same function:
That's unlikely to be a pure hardware problem.
>> all patches applied?
> OSS642a - Cron supplement (ver 1.0.0)
> OSS643A: Socket Driver Supplement for OpenServer 5 (ver 1.0.0)
> OSS645A: Audit Subsystem Security Supplement (ver 1.0.0)
> OSS648A: Processor Supplement for OpenServer 5.0.6 (ver 1.0.0)
> RS506A: Release Supplement for SCO OpenServer Release 5.0.6 (ver rs5)
> RS506A: Software Manager Supplement (ver rs506a)
>> CPU too fast? (no joke)
> Well... Dual CPU, P3/1000MHz, 512 megz of RAM. Another one -
>P4/2000MHz, performance is worse, so I'm giving up on this one.
>I don't know whether 1000 MHz is 'too fast' or not :)
Hm, i assumed you had other, slower machines running before
without problems (except lack of performance/thruput).
>> They never run stable since month?
> Nope... The higher the load - the sooner it's going to crash.
>> Simple attemps:
>> Change the RAM stripes
> Won't work - the brand new server with DDR266 memory crashes in
>exactly the same fashion, only still can write 'Press return to
>reboot' at the end :)
Ok, if it's always the same function, then a pure hardware problem
is not very likely.
>> Change the powersuplies, check voltages for transientes
> That I'll check, but something tells me that crashes would be in
>random places should this be the case...
Yes, but have a look to the 5V.
"Modern" boards burdens the 3,3V more, so the power supplies delivers
less power(ampers) on the 5V!
If you have 4 boards, that's more than 10A, not talking about
spikes..
>> Remove as much hardware as possible. (Including all Antares boards).
> Can't do that, unfortunately :(
Because the "panic" can be forced(really?) by "just" high load
it might be possible to make the box "alone" crashing, only
with the boards and no site equipment: The DSP software vendor should
have a "stress" utility or knows a way to simulate stress (he should have
such software...).
Rainer Zocholl
Where one can read from that's clearly the Dialogic driver?
"prf_task_s cmn_err" gives several hits on groups.google.com, why?
>(and then a double-panic in the panic dump writing code...!)
>
>When it hits this double-panic (2nd panic in bcpalign()), has it printed
>any of the dump-in-progress dots?
>
>If the double-panics are happening after the dump has printed some dots,
>there is something bad happening in the hardware. Something like a DMA
>transfer being written to a wrong address, corrupting memory not owned
>by the driver.
If spin/cpu lock is not working...
or if there is a DMA already released while the DMA/copy
is still inprogress.
http://www.intel.com/network/csp/products/1871web.htm
"Antares 2000/50, 3000/50, 6000/50 ISA Speech Platform"
Hm there is no DMA mentioned, only 4MB "Global (dual ported) DRAM"
That may explain the high CPU load if all Data must got tho the
ISA bottleneck by CPU!
All boards share(may share?) one IRQ line...
Only
Intel386, Intel486, or Pentium processor-based,
IBM PC AT (ISA) bus orcompatible computer.
is mentioned..
The board is a 1996 design, dialogic was bought by intel,
the Antares Board seems to have been retired in Dec 2002?
The boards ship with software packages for
SCO UNIX*, UnixWare*, Windows NT*, and Windows 2000*.
OSR not mentioned?
Several independent, third-party companies offer developers
software products that are compatible with the Speech Processor
Boards, such as SpeechWorks*, Nuance*, IBM*, Lernout & Hauspie
(L&H)*, Philips Speech Processing*, InfoTalk*, Syrinx*, Locus
Dialogue*, Loquendo*, Phonetic Systems*, Vocalis* and T-NETIX*.
Several other faults possible...
>The loop in sysdump() that calls dumpnextpage() and then
>bcopy() (which we see here as "bcpalign") uses the same addresses over
>and over.
>0xc0110000 is the unchanging address of a disk buffer it's
>using to stage writes.
>tmpva_pages is the unchanging virtual address at which it is
>sequentially mapping every page of memory.
>The mapping cannot fail (if no memory existed at that physical address,
>it would just get all 0xff's).
Or a random value (sometimes the last value written) depending
on hardware design.
>So if it double-panics after some dots have been printed,
>something very strange is happening. In fact it's pretty
>strange even if this is the first page.
>What is the value of register CR2 in these dumps?
>That's the address it got the fault on.
>Should be the same as either %esi or %edi in the last
>trap frame in the stack trace (si:dffda000 di:c0110000 in the 2nd
>example).
>
>For that matter, how are you displaying these stacks?! Those are
>crash(ADM) output. To get crash output on a panic, you would need a
>finished panic dump, but these show the system going down in flames in
>mid-dump! I could understand scodb traces, you could be using a serial
>console and capturing the output, but crash output from a double-panic
>in the dump code?!?
Good question... ;-)
Bela Lubkin
Three routines with "dlgn" in their names: dlgnwput, _dlgn_putc,
_dlgn_send. The next two, sramsendcm and sr_device, are probably
utility routines within the "dlgn" driver; then it calls freemsg() which
is a thin wrapper around freeb(), which panics. So a utility routine in
the Dialog driver is passing a bad pointer (points to invalid memory, or
wasn't allocated by allocb(), or was already freed, etc.) to freemsg().
> "prf_task_s cmn_err" gives several hits on groups.google.com, why?
The routines prf_task_sw() and cmn_err() will show up in any stack trace
of a system panicing, they're part of the normal code path of the panic
procedure. Then if someone displays that stack with crash(ADM),
"prf_task_sw" gets truncated to "prf_task_s", giving a perfect match...
> The board is a 1996 design, dialogic was bought by intel,
> the Antares Board seems to have been retired in Dec 2002?
>
> The boards ship with software packages for
> SCO UNIX*, UnixWare*, Windows NT*, and Windows 2000*.
>
> OSR not mentioned?
"SCO UNIX"
>Bela<
scooter6
There is some merit to what Fabio is saying. In the past 18 months, we
have upgraded 25 servers to dual 850MHz Pentiums with Dialogic &
Antares boards.
We did run across some issues with the SMP stuff, primarily with NICs
though.
It all had to do with putting ALL the necessary patches on, in the
correct order and making sure you put the SMP License as well as the
SMP Patch in place correctly as well. Failure to do that, DID cause
issues with us using BOTH 5.0.4 and 5.0.5
HTH
Scott Ullmann
sull...@telespectrum.com
Farlander
> Well, that's clearly in the Dialog driver (and then a double-panic in
> the panic dump writing code...!)
>
> When it hits this double-panic (2nd panic in bcpalign()), has it printed
> any of the dump-in-progress dots?
Yes it did. Usually it prints at least one or two points. Sometimes
it prints almost all the points, then double panics - and then again
prints a few points.
On 13 servers in the same manner? I don't know... It would be kind of
strange, wouldn't it?
> What is the value of register CR2 in these dumps? That's the address it
> got the fault on. Should be the same as either %esi or %edi in the last
> trap frame in the stack trace (si:dffda000 di:c0110000 in the 2nd
> example).
0x0000000C in the first one, 0x0000000E - in the second one.
> For that matter, how are you displaying these stacks?! Those are
> crash(ADM) output. To get crash output on a panic, you would need a
> finished panic dump, but these show the system going down in flames in
> mid-dump! I could understand scodb traces, you could be using a serial
> console and capturing the output, but crash output from a double-panic
> in the dump code?!?
That's a very good question :) Please let me know if you find an
answer :)
But seriously, what I'm going to do now is I want to remove a few
boards from some of servers and see if they still will be crashing.
By the way, as far as I know, they have a patched version of STREAMS
driver installed, that SCO has made especially for us - some kind of
patch for some error on multiple CPU systems, I don't know any details,
unfortunately. What I know is when I've installed the same driver on the
single CPU server, I got exactly the same kind of crashes - in 'freeb'
and in 'msgcount'. When I rolled back the original driver the single CPU
(P4) crashed in 'hlted' two times, but since I've removed three boards
from it - seems to be pretty stable (but, on the other hand, the load on
it has not yet been as high as it usually is, so lets wait till the
holidays are over).
Regards,
Farlander.
Farlander
>>When all servers were single P3-s everything worked fine.
>>Duals - crash after crash after crash...
>
>
> Hm, i know that driver implementors sometimes forget
> that there could be a second CPU accessing the hardware
> at the "same" time...
> this will happen especially under high load, of cause.
>
> To make it clear:
> If you turn the second CPU OFF in BIOS (or remove it or turn it off),
> it is not crashing or was it "not craching" on older boards?
When the servers were 'upgraded' to dual CPU, they started crashing
right away - a few crashed per day. However crashes stopped, when the
second CPU was turned off.
>>Same story with P4, only different kind of crashes now.
>
>
> But the P4 is a single IIRC, and double fast.
Yes, supposed to be double fast :) But in my case it works so much
slower - I have to pull it back from production :(
>>On dual CPUs it's always dual kernel panic,
>>on single -
>
>
> Do you mean the P4 box with "single"?
Yes.
>>it's just panic with 'Press return to reboot' at the end.
>
>
> That might have an other cause on the P4.
Maybe. All I know is that on my Dual CPUs there's some special
STREAMS driver installed, that was made by SCO especially for us, as I
was told. When I've tried to install it in that single CPU P4, it
started crashing in the same functions as duals - 'freeb' and
'msgcount'. With the original one it crashed twice in 'hlted' and in
some other places - never twice in the same function, except for hlted.
> Have you increased the system logging level to see
> how it goes to that "run" level?
Hmmm... I don't know OSR that much, honestly - how do I do that? :)
>>And it looks like I give up on P4s - doesn't handle the load
>>(two P3s 1000 MHz have ~70% CPU Idle most of the time under
>>the same load,
>
>
> load from what?
Load from phone calls hitting the server. The system takes up to 90%
of CPU. Here is the snapshot of 'sar' output on the server that has just
a few active sessions on it:
22:03:48 %usr %sys %wio %idle (-u)
22:03:49 8 67 2 24
22:03:50 16 67 0 17
22:03:51 3 42 0 55
22:03:52 8 62 3 27
And here is the snapshot from another server - dual CPU - with twice
more active sessions:
22:04:51 %usr %sys %wio %idle (-u)
22:04:52 1 19 1 79
22:04:53 7 26 1 66
22:04:54 3 16 3 78
22:04:55 4 19 1 76
There is some difference, no? :)
> Wow. What are the boards doing? They have 4 -four- relatively
> performant DSPs and still needs so much main CPU?
That's a very good question. I don't know! :( And I don't even know
for sure what exactly is causing this load - Dialogic voice boards or
Antares DSP boards :(
> If now a fast CPU is running, the bit is never set as
> expected so the driver runs (later) into a wait loop which may
> be much too long. Or it missed "pulsed" status bits (which are
> "broken by design" but may be found) (Assume: While DSP are busy
> from the previous command, a status bit is set to "1". On old CPUs
> it was sufficient to wait for the bit to become "0" again.
> On new -very fast- CPUs the (now adays slow) DSPs did not have
> enough time to set the busy bit.
> The CPU sees the (old) "0" and continues to sent the next command or what
> ever. That may work really stable for a long time (nextime the CPU asks
> the bit is set from the preprevoius command, but the DSP or CPU idled),
> but may cause the lost of parallel processing.
> That would explain your performance problems with a single CPU.
> (While one CPU is blocked in the -now- unlukely programed driver
> the other CPU can still work. With only one CPU all is blocked.)
That is possible. I was thinking about trying to do what was said
here before by Bela Lubkin - install SMP support and license on P4 in
order to activate its hyperthreading (if it supports one, which I,
honestly, don't know how to find out) and see if it's going to change
anything :)
> BTW(nothing to do with the panic):
> Did you check the CPU heat?
> If your boards generates such high loads and the cooling
> is not good, the P4 will throttle itself...
Would that cause output of HW command to change anyhow?
>>>Those?
>>> http://www.intel.com/network/csp/products/1871web.htm
>>
>
>> Yup...
>
>
> The boards are ISA or PCI?
> Doing DMA or not?
ISA. No DMA, AFAIK.
> Maybe it depends on the "doing" of the software?
> But if is really always almost the same function:
> That's unlikely to be a pure hardware problem.
To me it looks like an error in STREAMS or Dialogic driver. Don't
know which one, though :(
>> Well... Dual CPU, P3/1000MHz, 512 megz of RAM. Another one -
>>P4/2000MHz, performance is worse, so I'm giving up on this one.
>>I don't know whether 1000 MHz is 'too fast' or not :)
>
>
> Hm, i assumed you had other, slower machines running before
> without problems (except lack of performance/thruput).
Well, long time ago (like, about 2 years ago or so) the servers were
running on single P3/700 MHz and never crashed :)
>> That I'll check, but something tells me that crashes would be in
>>random places should this be the case...
>
>
> Yes, but have a look to the 5V.
> "Modern" boards burdens the 3,3V more, so the power supplies delivers
> less power(ampers) on the 5V!
> If you have 4 boards, that's more than 10A, not talking about
> spikes..
4 boards... I have about 3 times more than 4 boards in the servers :)
>>>Remove as much hardware as possible. (Including all Antares boards).
>>
>
>> Can't do that, unfortunately :(
>
>
> Because the "panic" can be forced(really?) by "just" high load
> it might be possible to make the box "alone" crashing, only
> with the boards and no site equipment: The DSP software vendor should
> have a "stress" utility or knows a way to simulate stress (he should have
> such software...).
I have, actually, removed a few boards from the P4 server - it can't
handle much load anyway, so I've removed the ones that are not used.
Seems to be stable so far (a little bit over 24 hours now). If it stays
up for a few days - probably that's the reason the single CPU was
crashing: the power. But I still don't know what to do with dual CPU
ones :( Don't even know which direction to look at :(
Regards,
Farlander.
Bela Lubkin
> Bela Lubkin wrote:
>
> > Well, that's clearly in the Dialog driver (and then a double-panic in
> > the panic dump writing code...!)
> >
> > When it hits this double-panic (2nd panic in bcpalign()), has it printed
> > any of the dump-in-progress dots?
>
> Yes it did. Usually it prints at least one or two points. Sometimes
> it prints almost all the points, then double panics - and then again
> prints a few points.
I need to be really clear here: a double-panic while printing the dots
is possible (it could panic in the host adapter driver). What I can't
make sense of is a double-panic with the stack you showed: a trap in
bcpalign() (which is bcopy()) called from dumpnextpage(), with a
different panic already on the stack.
Seeing that makes me think there is some fundamental problem with the
hardware -- something like cache coherency problems.
> > If the double-panics are happening after the dump has printed some dots,
> > there is something bad happening in the hardware. Something like a DMA
> > transfer being written to a wrong address, corrupting memory not owned
> > by the driver.
>
> On 13 servers in the same manner? I don't know... It would be kind of
> strange, wouldn't it?
Yep.
How similar is the hardware on the 13 servers? (Specifically:
motherboards & CPUs)
> > What is the value of register CR2 in these dumps? That's the address it
> > got the fault on. Should be the same as either %esi or %edi in the last
> > trap frame in the stack trace (si:dffda000 di:c0110000 in the 2nd
> > example).
>
> 0x0000000C in the first one, 0x0000000E - in the second one.
See, that doesn't make sense. bcpalign is a label within assembly code:
allcopy+2D nop
bcpalign movl %ecx,%ebx
bcpalign+2 shrl %ecx,2
bcpalign+5 rep movs (%esi),(%edi)
bcpalign+7 andl %ebx,3
bcptail movl %ecx,%ebx
(scodb disassembly). There is only one memory reference in that code,
and it reads from %esi and writes to %edi. If we panic at that address,
%cr2 _has_ to match either %esi or %edi. Or you're looking at a corrupt
dump, or the hardware is seriously busted.
> > For that matter, how are you displaying these stacks?! Those are
> > crash(ADM) output. To get crash output on a panic, you would need a
> > finished panic dump, but these show the system going down in flames in
> > mid-dump! I could understand scodb traces, you could be using a serial
> > console and capturing the output, but crash output from a double-panic
> > in the dump code?!?
>
> That's a very good question :) Please let me know if you find an
> answer :)
Hmmm. This isn't a real situation, you're making it up to see what kind
of response you get?
I would seriously like to know how you're showing crash(ADM) output of a
double-panic that aborted sysdump() in mid-dump.
> But seriously, what I'm going to do now is I want to remove a few
> boards from some of servers and see if they still will be crashing.
>
> By the way, as far as I know, they have a patched version of STREAMS
> driver installed, that SCO has made especially for us - some kind of
> patch for some error on multiple CPU systems, I don't know any details,
> unfortunately. What I know is when I've installed the same driver on the
> single CPU server, I got exactly the same kind of crashes - in 'freeb'
> and in 'msgcount'. When I rolled back the original driver the single CPU
> (P4) crashed in 'hlted' two times, but since I've removed three boards
> from it - seems to be pretty stable (but, on the other hand, the load on
> it has not yet been as high as it usually is, so lets wait till the
> holidays are over).
If you're using a patched STREAMS driver, you're going to have to
identify it much more strongly. That changes the whole picture --
you're running something unique, who knows what kind of behavior it has?
Then again, you say you crashed in `hlted'?! That's assembly again,
only has two instructions in it. To panic there, the CPU's `HLT'
instruction would have to somehow screw up.
Many years ago I helped someone debug a panic immediately after a `HLT'
instruction. The cause was a very low-level hardware problem. They
were the vendors of a 386SX "daughterboard" that plugged into a 286
socket. 386 Xenix was panicing. It turned out that they were not
following the CPU bus protocol for interrupts closely enough. A device
raises an interrupt, eventually the CPU strobes `INTA' (interrupt
acknowledge), the device presents the IRQ number, the CPU reads it, then
strobes `INTA' again. The 386SX daughterboard's timing for presenting
the IRQ number was off, so the CPU would read a nonsense IRQ number.
Result: CPU is sitting idle (in a `HLT' instruction), an IRQ comes in,
CPU reads a nonsense IRQ number, tries to jump to a nonexistent offset
in the interrupt vector table. Kernel sees this as a panic in `HLT'.
The panic wasn't seen in other instructions because if the CPU was
running an instruction that actually did something, the timing was
different and the daughterboard's late presentation of the IRQ number
wasn't late. Only when the CPU was sitting idle in `HLT', with nothing
on its "todo" list but handle the next incoming IRQ, was it fast enough
to see the problem.
You should make sure you have the latest microcode updates for your
CPUs. I see our `p6update' supplement (oss471f) is from late 2000, so
it probably doesn't cover your CPUs. :-(
Meanwhile, if you're having trouble with a patched STREAMS driver, you
_need_ to go back up the support channel you got that patch from. I'm
not going to be able to help with that. Without knowing what you're
really running, I'm only guessing. It's not useful.
>Bela<
Bela Lubkin
> Load from phone calls hitting the server. The system takes up to 90%
> of CPU. Here is the snapshot of 'sar' output on the server that has just
> a few active sessions on it:
>
> 22:03:48 %usr %sys %wio %idle (-u)
> 22:03:49 8 67 2 24
> 22:03:50 16 67 0 17
> 22:03:51 3 42 0 55
> 22:03:52 8 62 3 27
>
> And here is the snapshot from another server - dual CPU - with twice
> more active sessions:
>
> 22:04:51 %usr %sys %wio %idle (-u)
> 22:04:52 1 19 1 79
> 22:04:53 7 26 1 66
> 22:04:54 3 16 3 78
> 22:04:55 4 19 1 76
>
> There is some difference, no? :)
Please be more specific in these statements. I think you mean "here is
a single P4 with a few sessions" and "here is a dual PIII with twice as
many sessions". Put those words right inline so we know what you're
talking about all the way through. Add in CPU clock speeds. Slowest
P4's are 1.4GHz, fastest PIII's are close to that, this could be
perfectly reasonable (or close to it). Sure, the _actual_ speeds are
probably farther apart, but if you don't spell it out we don't know.
OpenServer 5.0.6 prints a statement of clock speed at boot time. It
also prints CPU info. Like this:
# egrep 'cpu|clock' /dev/string/cfg
%cpu - - - unit=1 family=6 type=Pentium III
%cpuid - - - unit=1 vend=GenuineIntel tfms=0:6:8:10(2)
%clock - - - type=TSC/999519129Hz
%cpu - 255 - unit=2 family=6 type=Pentium III
%cpuid - - - unit=2 vend=GenuineIntel tfms=0:6:8:6(2)
(from a dual Pentium III 1GHz machine). Notice that this machine is
running different stepping CPUs (068A and 0686) -- generally not a good
idea, but seems to be working on this machine.
Show us your output from `egrep 'cpu|clock' /dev/string/cfg`, clearly
tagging each with your performance observations. (It would probably
also be good to start a separate thread on this topic, something like
"P4 slower than expected".)
Also, run this ad hoc performance test on each:
time awk 'BEGIN{j=10*1000*1000;for(i=0;i<j;i++)}'
I get about 3.25 seconds on a PIII-1GHz. 15.2 seconds on a Pentium Pro
200MHz. Ignore the "real" and "sys" time, just look at "user".
Run it several times, while the system is under load and while it isn't.
Performance while not under load should be pretty constant (+/- a few
percent). Under heavy load it might get 10-25% worse due to cache
sharing.
If performance varies by more than 25% from run to run, something is
wrong with the machine's memory subsystem. Some areas of memory are not
cached, for instance.
> That is possible. I was thinking about trying to do what was said
> here before by Bela Lubkin - install SMP support and license on P4 in
> order to activate its hyperthreading (if it supports one, which I,
> honestly, don't know how to find out) and see if it's going to change
> anything :)
Wow, that's an improbable option... I have not seen a case where
Hyperthreading helps OSR5 performance (keeping in mind that so far, OSR5
has no clue about HT, does not know how to schedule processes to CPUs
properly to keep each hardware CPU busy). If Rainer's idea is right, it
might actually help in this case. One "CPU" could block in a device
driver while the other continued to run applications or other driver
code. Interesting.
> > BTW(nothing to do with the panic):
> > Did you check the CPU heat?
> > If your boards generates such high loads and the cooling
> > is not good, the P4 will throttle itself...
>
> Would that cause output of HW command to change anyhow?
No, but it would definitely affect the outcome of the little `awk`
performance test.
>Bela<
Rainer Zocholl
>Rainer Zocholl wrote:
>>>When all servers were single P3-s everything worked fine.
>>>Duals - crash after crash after crash...
>>
>>
>> Hm, i know that driver implementors sometimes forget
>> that there could be a second CPU accessing the hardware
>> at the "same" time...
>> this will happen especially under high load, of cause.
>>
>> To make it clear:
>> If you turn the second CPU OFF in BIOS (or remove it or turn it
>> off), it is not crashing or was it "not craching" on older boards?
> When the servers were 'upgraded' to dual CPU, they started crashing
>right away - a few crashed per day. However crashes stopped, when the
>second CPU was turned off.
Ah! I read "update"... ;-)
Did you reapply the patches a suggested?
There seems to be a problem with 506a if switching to SMP
occurs after patching...
Bela, is there an easy way to check if the "SMP patches"
were done the "right" sequence?
>> But the P4 is a single IIRC, and double fast.
> Yes, supposed to be double fast :) But in my case it works so much
>slower - I have to pull it back from production :(
I assume there is a driver problem too.
(Remember the "missing-bit-problem, sound very much like that.)
Rainer Zocholl
>Rainer Zocholl wrote:
>>>And it looks like I give up on P4s - doesn't handle the load
>>>(two P3s 1000 MHz have ~70% CPU Idle most of the time under
>>>the same load,
>>
>>
>> load from what?
> Load from phone calls hitting the server. The system takes up to
>90% of CPU.
Sounds that there is polling involved and CPU is not released...
I don't think that the boards are generating a sustained
data thruput of 2..4MByte/s (The ISA bus is slow! 8MB/s would be peak IIRC)
or?
>Here is the snapshot of 'sar' output on the server that
>has just a few active sessions on it:
>22:03:48 %usr %sys %wio %idle (-u)
>22:03:49 8 67 2 24
>22:03:50 16 67 0 17
>22:03:51 3 42 0 55
>22:03:52 8 62 3 27
> And here is the snapshot from another server - dual CPU - with
>twice more active sessions:
>22:04:51 %usr %sys %wio %idle (-u)
>22:04:52 1 19 1 79
>22:04:53 7 26 1 66
>22:04:54 3 16 3 78
>22:04:55 4 19 1 76
> There is some difference, no? :)
there seems to be still many traffic on the box. (See wio)
BTW: What kind of discs? SCSI or IDE?
> That's a very good question. I don't know! :( And I don't even know
>for sure what exactly is causing this load - Dialogic voice boards or
>Antares DSP boards :(
Emm, are there two kind of boards?
I thought Dialogic was the vendor/resller of the Antares boards?
>> The CPU sees the (old) "0" and continues to sent the next command or
>> what ever. That may work really stable for a long time (nextime the
>> CPU asks the bit is set from the preprevoius command, but the DSP or
>> CPU idled), but may cause the lost of parallel processing.
>> That would explain your performance problems with a single CPU.
>> (While one CPU is blocked in the -now- unlukely programed driver
>> the other CPU can still work. With only one CPU all is blocked.)
> That is possible. I was thinking about trying to do what was said
>here before by Bela Lubkin - install SMP support
and reapply 506a! ;-)
>and license on P4 in order to activate its hyperthreading
>(if it supports one,
SCO 5.0.6 does not "know" about hyperthreading.
It "thinks" that there are really 2 independ CPUs.
Under some circumstances that may cause less thruput,
but is worth a try if you can "play" more freely arround
with the non active P4 box. Maybe you get the same panics now.
>and see if it's going to change anything :)
>> BTW(nothing to do with the panic):
>> Did you check the CPU heat?
>> If your boards generates such high loads and the cooling
>> is not good, the P4 will throttle itself...
> Would that cause output of HW command to change anyhow?
AFAIK: No.
I don't know how to check if the CPU is throttled.
I assume that i have to read some special registers with the debugger
or is there more elegant tool?
>> Doing DMA or not?
> ISA. No DMA, AFAIK.
I looked into the spec: No DMA...
So all data is drawn from the boards by polling via CPU!
>>> Well... Dual CPU, P3/1000MHz, 512 megz of RAM. Another one -
>>>P4/2000MHz, performance is worse, so I'm giving up on this one.
>>>I don't know whether 1000 MHz is 'too fast' or not :)
>>
>>
>> Hm, i assumed you had other, slower machines running before
>> without problems (except lack of performance/thruput).
> Well, long time ago (like, about 2 years ago or so) the servers
>were running on single P3/700 MHz and never crashed :)
Yepp. But the new machines have 1000MHz, that's almost 50% faster,
maybe the newer CPU has more and faster Firstlevel cache,
and DDR-RAM is faster than SD-RAM which is much faster than PS/2-RAM!
>> Yes, but have a look to the 5V.
>> "Modern" boards burdens the 3,3V more, so the power supplies
>> delivers less power(ampers) on the 5V!
>> If you have 4 boards, that's more than 10A, not talking about
>> spikes..
> 4 boards... I have about 3 times more than 4 boards in the servers :)
Uups. Where did get boards with so many ISA slots? ;-)
Better check at least the 5V specfication of the power supply!
Maybe get a fast oszilloscope and a hardware engineer and inspect
the line for spikes (on the board slots!)
Maybe the layout of the board is a little weak in the ISA area...
soldering extra wires may help.
But that would be (at least) the third, emm fourth problem...
we have so far:
1. (maybe) Activated SMP after patching (see crashing)
2. (maybe) a driver that is not SMP proof (see crashing, but "hung" is missing )
3. (maybe) a driver that does not expect such fast CPUs (See huge CPU load)
4. (maybe) overloaded power supply on 5V (see the more stable working with one board)
5. (maybe) blocking ISA access (high CPU load)
Bela Lubkin
> (Farlander) 02.01.03 in /comp/unix/sco/misc:
> > When the servers were 'upgraded' to dual CPU, they started crashing
> >right away - a few crashed per day. However crashes stopped, when the
> >second CPU was turned off.
>
> Ah! I read "update"... ;-)
> Did you reapply the patches a suggested?
> There seems to be a problem with 506a if switching to SMP
> occurs after patching...
>
> Bela, is there an easy way to check if the "SMP patches"
> were done the "right" sequence?
`customquery listpatches` should do it; you should see:
SCO:smp::1.1.1Ga rs506a.smp111.1.0a
in the list.
I sometimes prefer a more empirical approach. Look at the file lists
for both rs506a and SMP (bust open the cpio archives). Figure out what
files are updated by both. Now look at the target system and make sure
that file contents are right.
In the case of SMP -> rs506a, you should check
/etc/conf/pack.d/apic/Driver.o and /etc/conf/pack.d/crllry/Driver.a:
# what /etc/conf/pack.d/apic/Driver.o /etc/conf/pack.d/crllry/Driver.a
/etc/conf/pack.d/apic/Driver.o:
apic/Driver.o SCO OpenServer 5.0.6a 2001-02-26
/etc/conf/pack.d/crllry/Driver.a:
Driver.a(actkeychk.o) SCO OpenServer 5.0.6a 2001-02-26
Driver.a(ciprf.o) SCO OpenServer 5.0.6a 2001-02-26
[...]
>Bela<
Farlander
> I'm having a HUGE problems with SCO OpenServer 5.0.6. I give up, I
> don't know where else to look: it keeps crashing every single day,
> sometimes a couple of times a day.
OK, the single CPU P4 has just crashed again:
KERNEL STACK TRACE FOR PROCESS 450:
STKADDR FRAMEPTR FUNCTION POSSIBLE ARGUMENTS
e0000b38 e0000c64 prf_task_s (0x4,0,0x79,0xe)
e0000c6c e0000c88 cmn_err (0x3,got_RESERVEDFLT+0x26c,0xe,u+0xcc8)
e0000c90 e0000cbc k_trap (u+0xcc8)
e0000cc8 kern_trap from 0xf005fc48 in msgdsize
ax: 0 cx:fd1b9038 dx: 5301 bx: 79 fl: 10282 ds: 160
fs: 0
sp:e0000cf8 bp:e0000d10 si:fd1b9038 di: 0 err: 0 es: 160
gs: 0
e0000cd0 e0000d10 msgdsize (0xfd1b9038,0x79,inode+0x57490,u+0x7972)
e0000d18 e0000d98 strioctl (inode+0x57490,0x5301,0x8047588,0x3)
e0000da0 e0000dbc cdevioctl (inode+0x57490,0x5301,0x8047588,0x3)
e0000dc4 e0000de8 ioctl (0x1,0x808fb5c,0x1,u+0xe10)
e0000df0 e0000e10 systrap (u+0xe1c)
e0000e1c scall_noke from 0x80030cee
ax: 36 cx: 1 dx: 1 bx: 1 fl: 292 ds: 1f
fs: 0
sp:e0000e4c bp: 8047590 si: 808fb5c di: 1 err: 36 es: 1f
gs: 0
No double panic, though. Just normal panic with 'Press RETURN to
reboot'...
I have removed a couple of boards from the server, so the power
shouldn't be a big problem, I guess (unfortunately don't have anything
to measure it with :( ).
Dammit :((( This is getting really frustrating...
Regards,
Farlander.
Bela Lubkin
You said you were using some sort of custom modified STREAMS driver.
Either follow up with the people that gave that to you, or back it out
and see what happens with the original 506 (probably rs506a) edition of
the driver.
>Bela<
Farlander
No, this server has the original one.
> Either follow up with the people that gave that to you, or back it out
> and see what happens with the original 506 (probably rs506a) edition of
> the driver.
This is the one.
Regards,
Farlander.
Bela Lubkin
Ok. I can spend a few minutes looking at the dump, if you put it up for
FTP. Do this:
# $dump is /dev/swap or a file you've written the dump to:
sysdump -i $dump -fumbh -o - | bzip2 > /tmp/dump.bz2
Then email me the location of the compressed dump. It should be
somewhere in the 4-10MB size range.
The problem is almost certainly provoked by the drivers for the telecom
and DSP cards you're using -- whether the underlying cause is a bug in
those drivers or a bug in OSR5 STREAMS that is specially provoked by
those drivers.
>Bela<
Rainer Zocholl
>Hello, everyone!
but email address is not working....
so i have to ask here:
Is the problem solved?
If, how?