SCO 5.0.4 Slows and vhand spins wildly.
Con Torrisi
Con Torrisi
Bretts-IT Team Leader
Webmaster - Systems Administrator
Mobile: 0418 880103
Work: 61 7 3361 0533
Home: 61 7 3263 2523
URL: http://www.bretts.com.au/~con _\\|//_
Email: c...@bretts.com.au (-0-0-)
---------------------------------------------------------ooO-(_)-Ooo----
Bela Lubkin
Con Torrisi wrote:
> This is a multi-part message in MIME format.
and for that reason, I almost didn't respond. Fix your news poster.
> I administer several SCO 5.0.4 Enterprise servers with 3 of them running =
> as Web Servers. Two of them are run almost identical software and were =
> set up at about the same time. One system has displayed a problem I have =
> never encountered before.
> First the system seems to hang. It could be hours or days after a =
> reboot.
> Then after about 5 minutes it will respond but it is obviously under =
> stress.
> It may or may not recover completely after about 30 minutes.
> During this time the "vhand" system program runs wildly using up all =
> possible CPU time.
> No other process is growing in size or using up very much CPU resource.
> It doesn't matter if there is a telnet session, so not the telnet death =
> problem.
> Sar reports only a small amount of swap is used up and real memory also =
> drops by a small amount.
What's happening with availrmem and availsmem while this goes on?
Easy way to track this (OSR5.0.4 only): link /usr/bin/sar to
/usr/bin/scosar. Then run `scosar -E 1 100` or whatever. You can run
this on already-collected (by 5.0.4) sar data files. BTW, in OSR5.0.5,
this becomes part of the `sar -r` output:
$ sar -r 1 3
SCO_SV vagabond 3.2v5.0.4 Pentium 04/04/1998
09:07:33 freemem freeswp availrmem availsmem (-r)
09:07:34 2381 148000 9134 20899
09:07:35 2381 148000 9134 20899
09:07:36 2381 148000 9134 20899
Average 2381 148000 9134 20899
> There is still about 192000 blocks of swap free and about 20000 units of =
> real memory free.
Despite all that, it's possible for availsmem to be approaching zero.
That will mess up virtual memory handling. All it takes is a process to
malloc a lot of memory and then do nothing with it.
> Apart from vhand consuming the system everything else seems okay.=20
> During the early stages of the symptom you can't even ping the unit.
> I have changed the processor, memory and motherboard.
>
> I've been working on this problem for 3 months now and I am at wits end.
>
> Customers are complaining and everything I try seems to have no effect.
Install scodb (in 5.0.4, that means change "N" to "Y" in
/etc/conf/sdevice.d/scodb, relink, reboot). Now when it gets into this
state, hit ^X on the console. "stack" to look at the kernel stack. "q"
to quit, continue system operation. Do ^X, stack, q repeatedly while
it's thrashing. This gets you a crude profile of kernel activity. What
is it doing?
>Bela<
--
Tales of our recent world trip, http://www.armory.com/~alexia/trip/trip.html
Jeff Liebermann
On Sat, 4 Apr 1998 17:11:16 GMT, Bela Lubkin <be...@sco.com>
wrote:
>Con Torrisi wrote:
>> I administer several SCO 5.0.4 Enterprise servers with 3 of them running =
>> as Web Servers. Two of them are run almost identical software and were =
>> set up at about the same time. One system has displayed a problem I have =
>> never encountered before.
Pardon my suspicious paranoid tendencies, but are these web
servers exposed to the Intenet? If so, your symptoms sounds like
some kind of DOS (denial of service) attack from the net. I
won't speculate which one due to lack of info. I suggest you:
1. Install the various security enhancements from the SCO web
site. See:
http://www.sco.com/security/
2. Install some kind of packet sniffer or minitor to see what's
happening.
>> First the system seems to hang. It could be hours or days after a =
>> reboot.
"Seems to hang" is a bit different than "hung". If it's hung,
you will not have keyboard control, cannot switch multiscreens on
the console, and will not see any activity on either the network
card LED or the drive light. "Seems to hang" sounds like some
kind of exessively busy condition. A good clue is that a runaway
process will may (not guaranteed) consume enough ram to start the
system swapping. You will see this as exessive disk activity on
the drive lights. However, a network attack will simply hog
processor cycles or network resources. You probably won't see
any exessive disk activity.
>> Then after about 5 minutes it will respond but it is obviously under =
>> stress.
Then the attack from the internet is continueing. The 5 minute
delay is indicative of a SYN flood. Run:
netstat
and see if you have a zillion sockets open, but not acknowledged.
Jeff Liebermann 150 Felker St. #D Santa Cruz CA 95060
(408)699-0483 pager (408)426-1240 fax (408)336-2558 home
http://www.cruzio.com/~jeffl WB6SSY
je...@comix.santa-cruz.ca.us je...@cruzio.com
Con Torrisi
-----Original Message-----
From: Bela Lubkin <be...@sco.com>
Newsgroups: comp.unix.sco.misc
To: post article to comp.unix.sco.misc; please followup there
<sco...@xenitec.on.ca>
Cc: Con Torrisi <c...@bretts.com.au>
Date: Sunday, 5 April 1998 3:15
Subject: Re: SCO 5.0.4 Slows and vhand spins wildly.
Bela,
Thanks for the advice. But I can't find scosar on the system. Is there
another way of getting availsmem?
I can detect the moment the problem ocurrs with a script I have written and
get any system stats. Unfortunately I am never at the console so activating
scodb won't do much good. The system in question is telehoused and I can
only get at it during working hours. Problem usually occurs evenings or
early morning.
Thanks
Con
>Con Torrisi wrote:
>
>> This is a multi-part message in MIME format.
>
>and for that reason, I almost didn't respond. Fix your news poster.
>
>> I administer several SCO 5.0.4 Enterprise servers with 3 of them running
=
>> as Web Servers. Two of them are run almost identical software and were =
>> set up at about the same time. One system has displayed a problem I have
=
>> never encountered before.
>> First the system seems to hang. It could be hours or days after a =
>> reboot.
>> Then after about 5 minutes it will respond but it is obviously under =
>> stress.
Con Torrisi
-----Original Message-----
From: Jeff Liebermann <je...@comix.santa-cruz.ca.us>
Newsgroups: comp.unix.sco.misc
To: distri...@xenitec.on.ca <distri...@xenitec.on.ca>
Date: Sunday, 5 April 1998 8:04
Subject: Re: SCO 5.0.4 Slows and vhand spins wildly.
Jeff,
Thanks for the advice. Yes I suspected a DOS especially seeing the problem
usually occurs evenings or early morning. But careful analysis of netstat
shows no sign of the SYN attack. I have installed all the latest security
patches from SCO.
When I say it "seems to hang". Its just that. First 5 minutes the system
will not respond to anything. No keyboard control and can't switch
multiscreens. There is little to no disk activity. I have never been at the
console during this first 5 minutes but I have had others there and they say
it does not respond at all. From the internet side its dead. Can't telnet
and can't even ping to it. Other units on the same network are fine.
Internally the system has some life and will continue some running
processes. I have a script running looking for the problem and recording
systems stats the moment vhand cpu time moves from 00:00:00.
So of course you are correct. The condition is an extreemely busy state. But
what I can't work out is why vhand is so busy when there is so much free
real and swap, and no other process is growing in size.
Its happening every day or so now.
Thanks for your interest
Con
>On Sat, 4 Apr 1998 17:11:16 GMT, Bela Lubkin <be...@sco.com>
>wrote:
>
>>Con Torrisi wrote:
>>> I administer several SCO 5.0.4 Enterprise servers with 3 of them running
=
>>> as Web Servers. Two of them are run almost identical software and were =
>>> set up at about the same time. One system has displayed a problem I have
=
>>> never encountered before.
>
>Pardon my suspicious paranoid tendencies, but are these web
>servers exposed to the Intenet? If so, your symptoms sounds like
>some kind of DOS (denial of service) attack from the net. I
>won't speculate which one due to lack of info. I suggest you:
>
>1. Install the various security enhancements from the SCO web
>site. See:
> http://www.sco.com/security/
>
>2. Install some kind of packet sniffer or minitor to see what's
>happening.
>
>>> First the system seems to hang. It could be hours or days after a =
>>> reboot.
>
>"Seems to hang" is a bit different than "hung". If it's hung,
>you will not have keyboard control, cannot switch multiscreens on
>the console, and will not see any activity on either the network
>card LED or the drive light. "Seems to hang" sounds like some
>kind of exessively busy condition. A good clue is that a runaway
>process will may (not guaranteed) consume enough ram to start the
>system swapping. You will see this as exessive disk activity on
>the drive lights. However, a network attack will simply hog
>processor cycles or network resources. You probably won't see
>any exessive disk activity.
>
>>> Then after about 5 minutes it will respond but it is obviously under =
>>> stress.
>
Lawrence Kirby
In article <01bd6070$c81a1c80$e50f3fcb@conathome>
c...@bretts.com.au "Con Torrisi" writes:
...
>Thanks for the advice. But I can't find scosar on the system. Is there
>another way of getting availsmem?
I refer you back to Bela's response:
>>Easy way to track this (OSR5.0.4 only): <<<< link /usr/bin/sar to
>>/usr/bin/scosar. >>>> Then run `scosar -E 1 100` or whatever. You can run
>>this on already-collected (by 5.0.4) sar data files.
<<<<>>>> added for emphasis.
--
-----------------------------------------
Lawrence Kirby | fr...@genesis.demon.co.uk
Wilts, England | 7073...@compuserve.com
-----------------------------------------
Bela Lubkin
Con Torrisi wrote:
> Thanks for the advice. But I can't find scosar on the system. Is there
> another way of getting availsmem?
Like I said, link /usr/bin/sar to /usr/bin/scosar. The sar binary acts
differently depending on its name. For reasons that I consider invalid,
some of the OSR5 sar functionality is hidden in a link that isn't
shipped, "scosar".
Two other ways:
echo o -d availsmem | crash
scodb> d &availsmem
(using user-level scodb, /etc/scodb, which becomes operable after you
have linked in the kernel-level scodb).
> I can detect the moment the problem ocurrs with a script I have written and
> get any system stats. Unfortunately I am never at the console so activating
> scodb won't do much good. The system in question is telehoused and I can
> only get at it during working hours. Problem usually occurs evenings or
> early morning.
When I need to use scodb on a remote system, here's what I do. It
involves two systems: "victim" is the OSR5 system to be debugged, and
"control" is another system physically co-located with it.
Steps 1 & 2 have to be done by someone on-site:
1. Set up a second system, "control", beside "victim". The second
system needs to have a serial port and be able to sustain an
inbound telnet or rlogin, i.e. just about anything that supports
TCP/IP *except* any sort of Windows box.
2. Physically cable the two machines' COM1 ports together.
Once these steps have been done, you can do everything else remotely:
3. Add "SYSTTY=1" as a separate line in /etc/default/boot on "victim".
4. Telnet into "control" and start a serial connection to COM1 (e.g.
`cu -ltty1a dir`).
5. Reboot "victim" (by logging in as root either by direct telnet, or
over the serial port from "control")
6. After a while, you should get the boot: prompt, serially, from
"victim" to "control", and thus over telnet to your remote site.
7. Hit return, allow the system to boot normally. Now you can get
into scodb by hitting ^X:
^X -> your telnet program -> telnetd on "control" -> cu -> serial
console on "victim"
If you have a lot of remote co-located OSR5 systems, you can set up a
"buddy system" where every OSR5 system has another OSR5 system as its
serial console. Then you can remotely debug any problematic system from
its "buddy".
Con Torrisi
Thanks Bela (and Lawrence Kirby). Got scosar showing me the info I need now. But
wouldn't you know it, the fault has not occurred for 4 days now. This is the
longest it has gone for months.
Luckily I have recorded every action and change I have made so I will start
backtracking my log book to see if anything is significant.
regards
Con Torrisi
Con Torrisi
Problem still here.
Okay, so after 4 days of clear running, vhand strikes again. Thanks for the
scosar hint Bela, it reported:
SCO_SV ppc 3.2v5.0.4 Pentium 04/08/98
00:00:00 availrmem availsmem (-E)
01:00:00 28099 40934
11:40:00 28006 40158
12:40:00 27692 39793
13:00:01 26875 36014 ------> vhand spins
13:20:01 26580 34499 System usable again but slow.
13:40:00 26734 34650
14:00:00 26925 34937
14:20:00 26881 34893
14:40:00 26873 34806
15:00:00 26910 34888
15:20:01 26941 39262
15:40:01 26910 39179
16:00:01 26889 39133
16:20:01 26898 39292
16:40:00 26887 39327
17:00:04 26853 39159 System goes crunch..very slow.
17:25:00 unix restarts Reboot clears it
17:40:01 28660 46224
18:00:04 28612 44469
19:00:00 28587 43518
20:00:01 28562 43481
21:00:01 28549 43203
This looks okay to me. Still plenty of resource left.
Hav'nt tried turning on scodb because the server is isolated and noone is
generally near the console when it happens.
One thing I did last Saturday morning was replace Squid 1.1.15 with the
1.1.14 binaries from skunkware. These are much smaller in size than the ones
I compiled. It was starting to look like the problem was gone. Squid after
about 4 hours of running has grown to 12600 blocks.(as reported in ps -elf).
Any chance squid is my culprit???
Any help is greatly appreciated.
Con Torrisi
>Con Torrisi wrote:
>
>> I administer several SCO 5.0.4 Enterprise servers with 3 of them running
=
>> as Web Servers. Two of them are run almost identical software and were =
>> set up at about the same time. One system has displayed a problem I have
=
>> never encountered before.
>> First the system seems to hang. It could be hours or days after a =
>> reboot.
>> Then after about 5 minutes it will respond but it is obviously under =
>> stress.
>> It may or may not recover completely after about 30 minutes.
>> During this time the "vhand" system program runs wildly using up all =
>> possible CPU time.
>> No other process is growing in size or using up very much CPU resource.
>> It doesn't matter if there is a telnet session, so not the telnet death =
>> problem.
>> Sar reports only a small amount of swap is used up and real memory also =
>> drops by a small amount.
>
>What's happening with availrmem and availsmem while this goes on?
>
>Easy way to track this (OSR5.0.4 only): link /usr/bin/sar to
>/usr/bin/scosar. Then run `scosar -E 1 100` or whatever. You can run
Bela Lubkin
Con Torrisi wrote:
> Problem still here.
>
> Okay, so after 4 days of clear running, vhand strikes again. Thanks for the
> scosar hint Bela, it reported:
>
> SCO_SV ppc 3.2v5.0.4 Pentium 04/08/98
> 00:00:00 availrmem availsmem (-E)
> 01:00:00 28099 40934
> 11:40:00 28006 40158
> 12:40:00 27692 39793
> 13:00:01 26875 36014 ------> vhand spins
> 13:20:01 26580 34499 System usable again but slow.
> 13:40:00 26734 34650
You're getting samples every 20 minutes. You said the problem happens
for about 5 minutes at a time, so this isn't necessarily going to pick
it up at all.
Run:
scosar -E 2 1000000
on one of the console multiscreens, an X window, or a character
terminal. When the system bogs down, look at that output -- has
availsmem gone near zero?
> Hav'nt tried turning on scodb because the server is isolated and noone is
> generally near the console when it happens.
No other machine nearby, to use as a "buddy system" serial console?
> One thing I did last Saturday morning was replace Squid 1.1.15 with the
> 1.1.14 binaries from skunkware. These are much smaller in size than the ones
> I compiled. It was starting to look like the problem was gone. Squid after
> about 4 hours of running has grown to 12600 blocks.(as reported in ps -elf).
> Any chance squid is my culprit???
Not if there's no memory event...
The difference in binary size is probably that you built COFF and the
Skunkworkers built ELF.
>Bela<
Con Torrisi
>Con Torrisi wrote:
>
>> Problem still here.
>>
>> Okay, so after 4 days of clear running, vhand strikes again. Thanks for
the
>> scosar hint Bela, it reported:
>>
>> SCO_SV ppc 3.2v5.0.4 Pentium 04/08/98
>> 00:00:00 availrmem availsmem (-E)
>> 01:00:00 28099 40934
>> 11:40:00 28006 40158
>> 12:40:00 27692 39793
>> 13:00:01 26875 36014 ------> vhand spins
>> 13:20:01 26580 34499 System usable again but slow.
>> 13:40:00 26734 34650
>
>You're getting samples every 20 minutes. You said the problem happens
>for about 5 minutes at a time, so this isn't necessarily going to pick
>it up at all.
>
>Run:
>
> scosar -E 2 1000000
At the moment of vhand going off I sampled the system and availsmem was
still about 37000, and dropped only a little.
>
>on one of the console multiscreens, an X window, or a character
>terminal. When the system bogs down, look at that output -- has
>availsmem gone near zero?
>
>> Hav'nt tried turning on scodb because the server is isolated and noone is
>> generally near the console when it happens.
>
>No other machine nearby, to use as a "buddy system" serial console?
System is in a locked secured room a long way away, and the problem mostly
occurs at night.
>
>> One thing I did last Saturday morning was replace Squid 1.1.15 with the
>> 1.1.14 binaries from skunkware. These are much smaller in size than the
ones
>> I compiled. It was starting to look like the problem was gone. Squid
after
>> about 4 hours of running has grown to 12600 blocks.(as reported in
ps -elf).
>> Any chance squid is my culprit???
>
>Not if there's no memory event...
Correct. But I think the growing squid process is setting the system off. I
went back through my logs and records and found the squid process was at
about 21Mb when the problem hit. Is there a process limit somewhere that
might trigger the problem when squid (or any process) hits about 20Mb
process size? I have reconfigured squid and I am watching it very closely.
Process size has stabilised at about 14Mb (with lots of tuning) and the
system has been fine now for almost 3 days.
After further investigation I discovered that squid has logged 134511432
Page faults with physical i/o. And that figure has not changed in 3 days.
One of my other systems reports 306179 Page Faults and has been running for
90 days. If you know squid this means that there was a request for a page
that didn't exist, a normal condition which says I should:
- Buy more memory for your system. -----IT HAS PLENTY
- Try a different malloc library, such as GNU malloc. -----DRASTIC I
THINK
- Reduce the cache_mem parameter in the config file. -----ALREADY SET AT
THE DEFAULT OF 8MB
- Turn the memory_pools off in the config file. -----I HAVE JUST DONE THIS
- Reduce the cache_swap parameter in your config file. This will reduce the
number of objects Squid keeps. --- DOWN TO 100MB NOW ANY LOWER AND I
COULD FORGET THE WHO IDEA OF A PROXY
- Reduce the maximum_object_size parameter. You won't be able to cache the
larger objects, and your byte volume hit ratio may go down, but Squid will
perform better overall. --- DROPPED IT FROM 4MB TO 2MB
- Try the ``NOVM'' version of Squid. ----???WHATS THIS???
>
>The difference in binary size is probably that you built COFF and the
>Skunkworkers built ELF.
Oh I see.
>
>>Bela<
What do you think. Is squid triggering the memory event?
Regards
Con
stephen f. lancour
Con,
From http://squid.nlanr.net/Squid description of SQUID-1.NOVM
"This is a parallel version to Squid-1.1. This ``NOVM'' version uses
less virtual memory, but more file descriptors."
This version builds and runs fine here on 5.0.4, but is never stressed.
Hope this helps.
Steve Lancour
Bela Lubkin
Con Torrisi wrote:
> > scosar -E 2 1000000
>
> At the moment of vhand going off I sampled the system and availsmem was
> still about 37000, and dropped only a little.
Ok, then I think we can rule out virtual memory exhaustion.
> >on one of the console multiscreens, an X window, or a character
> >terminal. When the system bogs down, look at that output -- has
> >availsmem gone near zero?
> >
> >> Hav'nt tried turning on scodb because the server is isolated and noone is
> >> generally near the console when it happens.
> >
> >No other machine nearby, to use as a "buddy system" serial console?
> System is in a locked secured room a long way away, and the problem mostly
> occurs at night.
You're not answering my question; you must not have understood my
original description of the "buddy system". It allows you to debug one
system under a kernel debugger such as scodb, over the network, from
halfway across the planet. You do not need to get into a locked room.
What's needed is for there to be *two* systems in the locked room. The
other one could be running any multiuser operating system, as long as it
has a free serial port that can be cabled to your problem systems' COM1
port. Yes, someone needs to go into the locked room *once*, at any
convenient time, to set up the cables. That's it.
> >> One thing I did last Saturday morning was replace Squid 1.1.15 with the
> >> 1.1.14 binaries from skunkware. These are much smaller in size than the ones
> >> I compiled. It was starting to look like the problem was gone. Squid after
> >> about 4 hours of running has grown to 12600 blocks.(as reported in ps -elf).
> >> Any chance squid is my culprit???
> >
> >Not if there's no memory event...
> Correct. But I think the growing squid process is setting the system off. I
> went back through my logs and records and found the squid process was at
> about 21Mb when the problem hit. Is there a process limit somewhere that
> might trigger the problem when squid (or any process) hits about 20Mb
> process size? I have reconfigured squid and I am watching it very closely.
There are a whole set of limits which *can* be set, but which normally
are not. That doesn't mean they aren't set on your system. For a brief
check, login and (if your login shell isn't `ksh`) run ksh. Then run
`ulimit -a`. On my system, which *does* have intentional restrictions
on various limits, I get:
time(seconds) unlimited
file(blocks) 4194303
data(kbytes) unlimited
stack(kbytes) 8192
coredump(blocks) 32768
nofiles(descriptors) 110
vmemory(kbytes) 65536
> Process size has stabilised at about 14Mb (with lots of tuning) and the
> system has been fine now for almost 3 days.
> After further investigation I discovered that squid has logged 134511432
> Page faults with physical i/o. And that figure has not changed in 3 days.
Where does the figure come from? Internal to squid? If so, how is it
determining the number?
> One of my other systems reports 306179 Page Faults and has been running for
> 90 days. If you know squid this means that there was a request for a page
> that didn't exist, a normal condition which says I should:
> - Buy more memory for your system. -----IT HAS PLENTY
> - Try a different malloc library, such as GNU malloc. -----DRASTIC I
> THINK
> - Reduce the cache_mem parameter in the config file. -----ALREADY SET AT
> THE DEFAULT OF 8MB
> - Turn the memory_pools off in the config file. -----I HAVE JUST DONE THIS
> - Reduce the cache_swap parameter in your config file. This will reduce the
> number of objects Squid keeps. --- DOWN TO 100MB NOW ANY LOWER AND I
> COULD FORGET THE WHO IDEA OF A PROXY
From your comment, it sounds like this controls the overall size of the
(on-disk) cache. The *name* doesn't seem to imply that. I don't know
squid, so I can't judge one way or the other.
> - Reduce the maximum_object_size parameter. You won't be able to cache the
> larger objects, and your byte volume hit ratio may go down, but Squid will
> perform better overall. --- DROPPED IT FROM 4MB TO 2MB
> - Try the ``NOVM'' version of Squid. ----???WHATS THIS???
> What do you think. Is squid triggering the memory event?
Again, I don't know. During the event, you say that vhand is consuming
lots of CPU. Is anything else consuming any CPU at all? e.g. is squid?
If so, maybe it's the cause. I would want to get a system call trace on
squid during the event. trace(CP) comes with the OpenServer devsys;
`scotruss` is at ftp://ftp.dgii.com/users/erics/scotruss.tar.Z.
scotruss works better most of the time, but sometimes trace is useful
too.
vhand would probably act fairly crazy if you set the kernel parameter
GPGSHI really high (on the same order as your availsmem low-water mark
of 37000).
>Bela<
Con Torrisi
Bela,
I will look into the Buddy system and activating scodb. The only other server in
the room is an NT box (yuk!).
>
>There are a whole set of limits which *can* be set, but which normally
>are not. That doesn't mean they aren't set on your system. For a brief
>check, login and (if your login shell isn't `ksh`) run ksh. Then run
>`ulimit -a`. On my system, which *does* have intentional restrictions
>on various limits, I get:
>
> time(seconds) unlimited
> file(blocks) 4194303
> data(kbytes) unlimited
> stack(kbytes) 8192
> coredump(blocks) 32768
> nofiles(descriptors) 110
> vmemory(kbytes) 65536
Here's mine:
time(seconds) unlimited
file(blocks) 4194303
data(kbytes) unlimited
stack(kbytes) unlimited
coredump(blocks) 4194303
nofiles(descriptors) 300
vmemory(kbytes) unlimited
>> Process size has stabilised at about 14Mb (with lots of tuning) and the
>> system has been fine now for almost 3 days.
>> After further investigation I discovered that squid has logged 134511432
>> Page faults with physical i/o. And that figure has not changed in 3 days.
>
>Where does the figure come from? Internal to squid? If so, how is it
>determining the number?
Can't give you much more info just now. I need to look at squids internals for
that.
>> - Reduce the cache_swap parameter in your config file. This will reduce the
>> number of objects Squid keeps. --- DOWN TO 100MB NOW ANY LOWER AND I
>> COULD FORGET THE WHO IDEA OF A PROXY
>
>>From your comment, it sounds like this controls the overall size of the
>(on-disk) cache. The *name* doesn't seem to imply that. I don't know
>squid, so I can't judge one way or the other.
Yes, this is the overall size of the on-disk cache. It "swaps" data in and out
of this.
>> What do you think. Is squid triggering the memory event?
>
>Again, I don't know. During the event, you say that vhand is consuming
>lots of CPU. Is anything else consuming any CPU at all? e.g. is squid?
Nothing else significantly uses any CPU time. Squid uses only a very small
amount. From the moment of the event I record "ps -elf" output at 1 minute
intervals for 10 minutes. Nothing is growing in CPU usage or process size during
that time. Squid manages to clock up about 00:00:03 whereas vhand gets a massive
00:09:30 clock cycles.
>If so, maybe it's the cause. I would want to get a system call trace on
>squid during the event. trace(CP) comes with the OpenServer devsys;
>`scotruss` is at ftp://ftp.dgii.com/users/erics/scotruss.tar.Z.
>scotruss works better most of the time, but sometimes trace is useful
>too.
I'll check out trace.
>
>vhand would probably act fairly crazy if you set the kernel parameter
>GPGSHI really high (on the same order as your availsmem low-water mark
>of 37000).
Nope. GPGSHI is set to 40
I can send you the whole config.h file if you would like to check it out.
The system has been fine now for 5 days. This is the longest it has been up for
many months. It's looking more and more like squid is the trigger. If so then I
have 3 other systems configured identically waiting for squid to get loaded
sufficiently to set them off. The other systems are only lightly loaded whereas
the problem system is a growing ISP with heavy proxy traffic.
I'll study up on trace and scodb and post a messge to the squid users group as
well.
Meanwhile, I'm enjoying the uptime and if it continues for a few more days I'll
tickle squid to see if it reacts again.
Cheers
Con
Bela Lubkin
Con Torrisi wrote:
> I will look into the Buddy system and activating scodb. The only other server in
> the room is an NT box (yuk!).
I hear you can find a telnetd for NT. The hard part will be finding a
program that lets you run an outbound serial session from an inbound
telnet session. Maybe an old DOS version of Kermit or some other comm
program.
Your ulimits look fine, not the cause of this problem
> >> What do you think. Is squid triggering the memory event?
> >
> >Again, I don't know. During the event, you say that vhand is consuming
> >lots of CPU. Is anything else consuming any CPU at all? e.g. is squid?
> Nothing else significantly uses any CPU time. Squid uses only a very small
> amount. From the moment of the event I record "ps -elf" output at 1 minute
> intervals for 10 minutes. Nothing is growing in CPU usage or process size during
> that time. Squid manages to clock up about 00:00:03 whereas vhand gets a massive
> 00:09:30 clock cycles.
Still, if squid got 3 seconds it was doing a fair amount of activity
(that's on the order of a billion clock cycles on current hardware!) So
it is probably something that squid's doing that's driving vhand mad.
> >If so, maybe it's the cause. I would want to get a system call trace on
> >squid during the event. trace(CP) comes with the OpenServer devsys;
> >`scotruss` is at ftp://ftp.dgii.com/users/erics/scotruss.tar.Z.
> >scotruss works better most of the time, but sometimes trace is useful
> >too.
> I'll check out trace.
Get scotruss as well. You wouldn't walk 5 miles to help a friend patch
a tire, leaving your tire patching kit in the garage and bringing only a
piece of chewing gum.
> >vhand would probably act fairly crazy if you set the kernel parameter
> >GPGSHI really high (on the same order as your availsmem low-water mark
> >of 37000).
> Nope. GPGSHI is set to 40
>
> I can send you the whole config.h file if you would like to check it out.
Send me /etc/conf/cf.d/stune, it's smaller and just as informative.
> The system has been fine now for 5 days. This is the longest it has been up for
> many months. It's looking more and more like squid is the trigger. If so then I
> have 3 other systems configured identically waiting for squid to get loaded
> sufficiently to set them off. The other systems are only lightly loaded whereas
> the problem system is a growing ISP with heavy proxy traffic.
>
> I'll study up on trace and scodb and post a messge to the squid users group as
> well.
>
> Meanwhile, I'm enjoying the uptime and if it continues for a few more days I'll
> tickle squid to see if it reacts again.
Ok. If I remember right, you said you'd stabilized the system by tuning
squid down almost to uselessness? That's not a good solution. I know
others on this group are using squid and not treading on eggshells like
that. Did you build it yourself or get binaries from somewhere? Maybe
some build switch was set wrong (though the kernel still shouldn't be
reacting like you're experiencing...)
>Bela<
Con Torrisi
>Con Torrisi wrote:
>
>> I will look into the Buddy system and activating scodb. The only other
server in
>> the room is an NT box (yuk!).
Sorry Bela the NT box is in the next room. Cabling will be a problem.
Security is tight. This is a telehouse situation. Remote admin where the
equipment is so hard to get at is a very difficult thing.
>I hear you can find a telnetd for NT. The hard part will be finding a
>program that lets you run an outbound serial session from an inbound
>telnet session. Maybe an old DOS version of Kermit or some other comm
>program.
I'll look into it.
>
>Your ulimits look fine, not the cause of this problem
>
>Still, if squid got 3 seconds it was doing a fair amount of activity
>(that's on the order of a billion clock cycles on current hardware!) So
>it is probably something that squid's doing that's driving vhand mad.
Sure looks like it.
>> >If so, maybe it's the cause. I would want to get a system call trace on
>> >squid during the event. trace(CP) comes with the OpenServer devsys;
>> >`scotruss` is at ftp://ftp.dgii.com/users/erics/scotruss.tar.Z.
>> >scotruss works better most of the time, but sometimes trace is useful
>> >too.
>> I'll check out trace.
>
>Get scotruss as well. You wouldn't walk 5 miles to help a friend patch
>a tire, leaving your tire patching kit in the garage and bringing only a
>piece of chewing gum.
Okay.
>
>> >vhand would probably act fairly crazy if you set the kernel parameter
>> >GPGSHI really high (on the same order as your availsmem low-water mark
>> >of 37000).
>> Nope. GPGSHI is set to 40
>>
>> I can send you the whole config.h file if you would like to check it out.
>
>Send me /etc/conf/cf.d/stune, it's smaller and just as informative.
Contents of stune:
NODE "ppc"
EVDEVS 96
EVQUEUES 88
NSPTTYS 64
NUMSP 256
NSTREAM 8064
NHINODE 1024
TTHOG 4096
NCLIST 512
NSTREVENT 8192
NUMTIM 1040
NUMTRW 1040
SECLUID 0
SECSTOPIO 0
SECCLEARID 1
MAXUP 200
NOFILES 300
NSTRPAGES 800
MAXSLICE 50
>
>Ok. If I remember right, you said you'd stabilized the system by tuning
>squid down almost to uselessness? That's not a good solution. I know
>others on this group are using squid and not treading on eggshells like
>that. Did you build it yourself or get binaries from somewhere? Maybe
>some build switch was set wrong (though the kernel still shouldn't be
>reacting like you're experiencing...)
I compile the original squid using a SCO development set. But the last one
I installed I got the binaries from Skunkware pre-compiled.
Until last night the system was fine for 7 days. Yesterday I increased the
disk swap (cache) size of squid from 100 to 200Mb. I traced the size of
squid with "ps -p(process_id_of_squid) -o size -o vsz". It grew to about
size=15048 and vsz=15848 and stabilised. When vhand went nuts last night
this size did not alter.
During the event I could not ping the SCO box. But I could not ping the NT
box either!!! But the router and terminal server (Bay 5399) responded fine
to ping. I telneted to the router and it couldn't reach the SCO box either.
Strange. After 5 minutes the SCO box responded to ping and I could telnet in
and vhand was still going and the NT box came back to life as well.
I know this sounds like a network problem but how could something cause
this? Maybe the network card?? Could tcp/ip on the SCO box drown the NT box
or vice-versa??
Thanks for your persistence Bela.
Con
Bela Lubkin
Con Torrisi wrote:
> >> I can send you the whole config.h file if you would like to check it out.
> >
> >Send me /etc/conf/cf.d/stune, it's smaller and just as informative.
> Contents of stune:
>
> NODE "ppc"
> EVDEVS 96
> EVQUEUES 88
> NSPTTYS 64
> NUMSP 256
> NSTREAM 8064
> NHINODE 1024
> TTHOG 4096
> NCLIST 512
> NSTREVENT 8192
> NUMTIM 1040
> NUMTRW 1040
> SECLUID 0
> SECSTOPIO 0
> SECCLEARID 1
> MAXUP 200
> NOFILES 300
> NSTRPAGES 800
> MAXSLICE 50
Ok, I don't see anything wrong there. (I assume `netstat -m` is clean
of failures.)
> >Ok. If I remember right, you said you'd stabilized the system by tuning
> >squid down almost to uselessness? That's not a good solution. I know
> >others on this group are using squid and not treading on eggshells like
> >that. Did you build it yourself or get binaries from somewhere? Maybe
> >some build switch was set wrong (though the kernel still shouldn't be
> >reacting like you're experiencing...)
>
> I compile the original squid using a SCO development set. But the last one
> I installed I got the binaries from Skunkware pre-compiled.
>
> Until last night the system was fine for 7 days. Yesterday I increased the
> disk swap (cache) size of squid from 100 to 200Mb. I traced the size of
> squid with "ps -p(process_id_of_squid) -o size -o vsz". It grew to about
> size=15048 and vsz=15848 and stabilised. When vhand went nuts last night
> this size did not alter.
>
> During the event I could not ping the SCO box. But I could not ping the NT
> box either!!! But the router and terminal server (Bay 5399) responded fine
> to ping. I telneted to the router and it couldn't reach the SCO box either.
> Strange. After 5 minutes the SCO box responded to ping and I could telnet in
> and vhand was still going and the NT box came back to life as well.
Interesting.
Does the NT box depend on the OSR5 box at all? (e.g. for DNS). If you
could ping neither box, that certainly suggests an external (to either
system) network problem. A sniffer on the cable would be interesting,
probably not possible in this high security scenario, though...
> I know this sounds like a network problem but how could something cause
> this? Maybe the network card?? Could tcp/ip on the SCO box drown the NT box
> or vice-versa??
Some sort of broadcast storm, maybe?
Monitoring `netstat -m` during the problem would probably be
interesting, even if there are no failures. If the current usage
statistics change rapidly, you know there's a lot of net activity.
`ndstat -l` and `netstat -in` would also be interesting. Can you
arrange for those three commands to be run periodically, all the time?
I'm assuming you won't be able to start a script to run them *during*
the event. The period should be chosen so that you'll probably get a
number of samples during an event; but not so frequent that the
monitoring is a burden in itself. Maybe every 15 seconds.
>Bela<
Jeff Liebermann
On Mon, 20 Apr 1998 10:25:29 GMT, Bela Lubkin <be...@sco.com>
wrote:
>Monitoring `netstat -m` during the problem would probably be
>interesting, even if there are no failures. If the current usage
>statistics change rapidly, you know there's a lot of net activity.
>`ndstat -l` and `netstat -in` would also be interesting. Can you
>arrange for those three commands to be run periodically, all the time?
>I'm assuming you won't be able to start a script to run them *during*
>the event. The period should be chosen so that you'll probably get a
>number of samples during an event; but not so frequent that the
>monitoring is a burden in itself. Maybe every 15 seconds.
I've been reading the blow-by-blow troubleshooting exercise for
about a week and trying not to interfere. Methinks this exercise
is chaseing the symptoms and not the cause. Some suggestions:
1. If you think that SQUID is the culprit, try disarming it
completely. Yes, the web server will be less responsive, but at
least it will *ELIMINATE* an obvious cause.
2. Having two servers hickup at the same time implies a network
originated problem. My original theory that the server(s) are
being attacked from the outside still stands as possible.
3. Get and use some network diagnostic and monitoring tools. I
use MRTG found at:
http://www.ee.ethz.ch/~oetiker/webtools/mrtg/mrtg.html
It will monitor and log anything that belches integers, runs on
any Unix or NT box, and displays on any web browser exept NS4.04.
For example, this is the idle percentage on one of my customers
servers.
http://www.cruzio.com/~jeffl/mrtg/idle.html
For examples of what people who know what they're doing:
http://ee-staff.ethz.ch/~oetiker/webtools/mrtg/users.html
If you have problems with SNMP on SCO, see:
http://gate.cruzio.com/~jeffl/sco/snmp_install.txt
Knowing when things happen and how long they last is important
for isolating the cause and will be very useful for future
troubleshooting exercises.
4. Borrow a protocol analyzer. MS Netmon from SMS 1.1 on NT4 is
what I use. Set traps and filters so that it captures and saves
when anything alarming appears. Be sure to capture traffic while
the SCO box is down.
5. If you have a firewall, check it to make sure that commonly
available hacker DOS attacks will be properly trapped. Each one
has some characteristic port number or "fingerprint" that will
identify the culprit.
6. Check some really simple things like:
Duplicate IP addresses with: arp -a
Streams failure with: netstat -m
(Gotta run. Customers...)
Con Torrisi
Bela and Jeff
Thanks for the attention.
I apologise in advance for the verbose nature of this post. Problem getting
progressively worse.
I am chasing the hardware aspect of this as well. Putting together a totally new
machine to eliminate the hardware as a source. But even so if its hardware why
would the OS behave like this.
I think squid may have been a wild goose chase. I monitored its memory usage
before, during and after the event and there is no change. Maybe coincidence the
problem abated after the change of version/settings?
I got scotruss working and ran it against squid but it poured info out. Too much
and if I restrict it to a specific system call then which one?? Also it doesn't
show the time of the call to match to the event.
I can't disable squid totally because all the user settings are pointing through
it to get access to the net and it would effectively disable all users.
I will get a copy of MRTG and study up on it. I'll also ask around for a
protocol analyser.
Checked arp -a and found no dupes
No failures reports from netstat -m
Could be a DOS attack but I have all the latest security patches and I can't see
any of the usual signs.
Here is a result of log and stats collected immediately after the event before
the system was rebooted.
This time I was actually in the room.
Here's what happened.
vhand goes wild at 10:43 and clocks up 00:47:56 CPU time before coming to rest.
Took about 10 minutes before I could log in and about 50 minutes for vhand to
settle down.
Could not ping the NT box on the network from any other box but it was still
working and I could not ping anyone from the NT box during that time. So both
Unix and NT systems were isolated. The router and the terminal server were okay.
No other units are on the network.
Looking at the lights there seemed to be normal network traffic, very little
disk activity and the router traffic was lower than usual. (propably because the
20 or so logged on users could not get out to the internet because the DNS and
PROXY in the Unix box were not responding.)
scosar -E reports there is lots of memory available. Swap and real.
sar -w showed
00:00:00 swpin/s bswin/s swpot/s bswot/s pswch/s (-w)
09:20:00 0.00 0.0 0.00 0.0 25
09:40:00 0.00 0.0 0.00 0.0 34
10:00:01 0.00 0.0 0.00 0.0 42
10:20:00 0.00 0.0 0.00 0.0 34
10:40:00 0.03 0.2 0.04 2.6 29
11:00:02 0.17 1.3 0.36 4.9 65
11:20:02 0.02 0.2 0.00 0.0 84
11:40:02 0.03 0.2 0.00 0.0 87
sar -p showed
00:00:00 vflt/s pflt/s pgfil/s rclm/s (-p)
08:40:00 18.41 55.94 0.00 0.00
09:00:00 18.43 55.74 0.03 0.00
09:20:00 18.59 56.28 0.01 0.00
09:40:00 18.83 56.77 0.02 0.00
10:00:01 18.49 55.99 0.00 0.00
10:20:00 18.49 56.01 0.00 0.00
10:40:00 18.83 56.65 0.18 0.42
11:00:02 6.48 10.80 0.58 1.87
11:20:02 1.34 1.96 0.49 0.00
11:40:02 1.92 3.30 0.71 0.00
12:00:00 1.86 2.49 0.50 0.00
12:20:00 0.87 2.25 0.03 0.00
12:40:00 0.78 1.98 0.02 0.00
sar -q showed
00:00:00 runq-sz %runocc swpq-sz %swpocc (-q)
08:20:01 1.9 11
08:40:00 2.0 11
09:00:00 1.9 10
09:20:00 2.0 11
09:40:00 1.8 10
10:00:01 2.1 9
10:20:00 2.1 9
10:40:00 2.3 8
11:00:02 1.1 70
11:20:02 1.0 84
11:40:02 1.1 100
12:00:00 1.0 44
--------------------------------------------------------------------------------
------
netstat -m
streams allocation:
config alloc free total max
fail
stream 8064 149 7915 50623 212 0
queues 908 305 603 101255 431 0
mblks 7098 374 6724 4371114 6966 0
buffer headers 7098 744 6354 605264 7037 0
class 1, 64 bytes 128 16 112 1606504 133 0
class 2, 128 bytes 64 0 64 653390 3247 0
class 3, 256 bytes 128 10 118 616118 113 0
class 4, 512 bytes 16 6 10 24245 43 0
class 5, 1024 bytes 40 4 36 25149 77 0
class 6, 2048 bytes 124 101 23 106352 205 0
class 7, 4096 bytes 118 100 18 26517 146 0
class 8, 8192 bytes 1 0 1 135 2 0
class 9, 16384 bytes 2 0 2 210 3 0
class 10, 32768 bytes 2 0 2 112 4 0
class 11, 65536 bytes 0 0 0 0 0 0
class 12, 131072 bytes 0 0 0 0 0 0
class 13, 262144 bytes 0 0 0 0 0 0
class 14, 524288 bytes 0 0 0 0 0 0
total configured streams memory: 3200.00KB
streams memory in use: 661.90KB
maximum streams memory used: 2041.23KB
ndstat -l showed
Device MAC address in use Factory MAC Address
------ ------------------ -------------------
/dev/net1 00:60:08:58:d0:12 00:60:08:58:d0:12
Multicast address table
-----------------------
01:00:5e:00:00:01
FRAMES
Unicast Multicast Broadcast Error Octets Queue Length
---------- --------- --------- ------ ----------- ------------
In: 591001 0 22009 0 254384191 0
Out: 554317 0 9766 3951 234427010 0
DLPI Module Info: 2 SAPs open, 18 SAPs maximum
MAC Driver Info: Media_type: Ethernet
Min_SDU: 14, Max_SDU: 1514, Address length: 6
Interface speed: 10 Mbits/sec
DLPI Restarts Info: Last queue size: 0
Last send time: 7101103
Restart in progress: 0
Number of restarts: 0
Interface Version: MDI 100
ETHERNET SPECIFIC STATISTICS
Collision Table - The number of frames successfully transmitted,
but involved in at least one collision:
Frames Frames
------- -------
1 collision 2567 9 collisions 0
2 collisions 2339 10 collisions 0
3 collisions 0 11 collisions 0
4 collisions 0 12 collisions 0
5 collisions 0 13 collisions 0
6 collisions 0 14 collisions 0
7 collisions 0 15 collisions 0
8 collisions 0 16 collisions 0
Bad Alignment 0 Number of frames received that were
not an integral number of octets
FCS Errors 0 Number of frames received that did
not pass the Frame Check Sequence
SQE Test Errors 0 Number of Signal Quality Error Test
signals that were detected by the adapter
Deferred Transmissions 10736 Number of frames delayed on the
first transmission attempt because
the media was busy
Late Collisions 0 Number of times a collision was
detected later than 512 bits into
the transmitted frame
Excessive Collisions 0 Number of frames dropped on transmission
because of excessive collisions
Internal MAC Transmit 0 Number of frames dropped on transmission
Errors because of errors not covered above
Carrier Sense Errors 0 Number of times that the carrier sense
condition was lost when attempting to
send a frame that was deferred for an
Carrier Sense Errors 0 Number of times that the carrier sense
condition was lost when attempting to
send a frame that was deferred for an
excessive amount of time
Frame Too Long 0 Number of frames dropped on reception
because they were larger than the
maximum Ethernet frame size
Internal MAC Receive 0 Number of frames dropped on reception
Errors because of errors not covered above
Spurious Interrupts 0 Number of times the adapter interrupted
the system for an unknown reason
No STREAMS Buffers 3951 Number of frames dropped on reception
because no STREAMS buffers were available
Underruns/Overruns 0 Number of times the transfer of
data to or from the frame buffer
did not complete successfully
Device Timeouts 0 Number of times the adapter failed to
respond to a request from the driver
About 1 hour later the collisions had increased to:
1 collision 2569 9 collisions 0
2 collisions 2342 10 collisions 0
and the No STREAMS was the same.
!!! No STREAMS !!! netstat -m showed no failures???? or were the streams buffers
dropped because the system wasn't responding.
Cheers
Con
Bela Lubkin
Con Torrisi wrote:
> vhand goes wild at 10:43 and clocks up 00:47:56 CPU time before coming to rest.
> Took about 10 minutes before I could log in and about 50 minutes for vhand to
> settle down.
> Could not ping the NT box on the network from any other box but it was still
> working and I could not ping anyone from the NT box during that time. So both
> Unix and NT systems were isolated. The router and the terminal server were okay.
> No other units are on the network.
I wonder if the network segment was going insane (bad termination or
something like that). The stats from `ndstat -l` come out of the NIC
driver, and it's up to that driver to actually maintain them. You could
have had a billion bad packet interrupts and the driver didn't bother to
log them. Can you swap in a different NIC (one that uses a completely
different driver) -- just to see if it logs anything? I'm thinking of
things like electrical noise on the line, that wouldn't show up as
packets per se, so might not have shown on the "lights".
> Looking at the lights there seemed to be normal network traffic, very little
> disk activity and the router traffic was lower than usual. (propably because the
> 20 or so logged on users could not get out to the internet because the DNS and
> PROXY in the Unix box were not responding.)
> No STREAMS Buffers 3951 Number of frames dropped on reception
> because no STREAMS buffers were available
> !!! No STREAMS !!! netstat -m showed no failures???? or were the streams buffers
> dropped because the system wasn't responding.
There's a sub-pool of network buffers used at interrupt time. If that's
exhausted it'll rack up failures, but I'm not sure you'll see anything
in `netstat -m`. Running out at interrupt time suggests a heavy
interrupt load -- noise on the network media.
>Bela<
Michael Wojcik
In article <980421050...@vagabond.armory.com> Bela Lubkin <be...@sco.com> writes:
>Con Torrisi wrote:
>> No STREAMS Buffers 3951 Number of frames dropped on reception
>> because no STREAMS buffers were available
>
>> buffers dropped because the system wasn't responding.
>
>exhausted it'll rack up failures, but I'm not sure you'll see anything
>in `netstat -m`. Running out at interrupt time suggests a heavy
>interrupt load -- noise on the network media.
Out of curiosity - the number of "No STREAMS Buffers" errors (3951)
matches the "FRAMES / Out / Error" number from ndstat -l:
[From Con Torrisi's note 01bd6ce1$42eb4c00$5b0f...@conwork.harpbbt]
> ndstat -l showed
> Device MAC address in use Factory MAC Address
> ------ ------------------ -------------------
> /dev/net1 00:60:08:58:d0:12 00:60:08:58:d0:12
> Multicast address table
> -----------------------
> 01:00:5e:00:00:01
> FRAMES
> Unicast Multicast Broadcast Error Octets Queue Length
> ---------- --------- --------- ------ ----------- ------------
> In: 591001 0 22009 0 254384191 0
> Out: 554317 0 9766 3951 234427010 0
If the "No STREAMS Buffers" errors were input interrupt-servicing
errors, shouldn't they be listed under In? Is something being
misreported, or is this actually an output failure? Perhaps
something's trying to write to the network, getting a temporary
no-buffers condition, and spinning in retries? Is it possible
there's a narrow window for a race between vhand and some low-
level part of the network code - eg. a transient output buffer
allocation failure triggers a page steal but then either tries to
claim the page before it's ready (and fails, triggering another
steal) or fails to lock the region (allowing vhand to steal the
page being swapped in, thereby requiring another page-in and
potentially another steal)? It's been a long time since I looked
in any detail at the design of the "classic" vhand, and I don't
know how different OSR4's is from the one Bach describes.
Michael Wojcik m...@microfocus.com
AAI Development, Micro Focus Inc.
Department of English, Miami University
Love thine and hamster and live with the hamster.
< From the hamster and the sutra >
-- roro
Kevin Smith
In article <980421050...@vagabond.armory.com> Bela Lubkin <be...@sco.com> writes:
>Con Torrisi wrote:
>
>> Took about 10 minutes before I could log in and about 50 minutes for vhand to
>> settle down.
>> Could not ping the NT box on the network from any other box but it was still
>> working and I could not ping anyone from the NT box during that time. So both
>> Unix and NT systems were isolated. The router and the terminal server were okay.
>> No other units are on the network.
>
>something like that). The stats from `ndstat -l` come out of the NIC
>driver, and it's up to that driver to actually maintain them. You could
>have had a billion bad packet interrupts and the driver didn't bother to
>log them. Can you swap in a different NIC (one that uses a completely
>different driver) -- just to see if it logs anything? I'm thinking of
>things like electrical noise on the line, that wouldn't show up as
>packets per se, so might not have shown on the "lights".
I had a recent problem (HPUX) where the machine would totally freeze for
random periods (several seconds to many minutes). We tracked it down
to network noise eminating from a couple of old pc's and sub-standard
wiring (10baseT wiring with the flat 8-wire gray satin phone cord).
The HP would totally freeze to the point of losing echo on serial
terminals. We could also see high utilization on one of the hubs and
the HP's NIC stats would show a high number of frame errors once the
noise stopped. The network storm could be stopped by unplugging one
of the pc's from the hub for a couple of seconds.
Obviously this requires someone on site, but it might be usefull to
disconnect the machine from the network when vhand starts spinning
and see what happens. Also if the machine is connected to a hub
with any kind of utilization indicator, what does it say?
--
Do two rights make | Kevin Smith, ShadeTree Software, Philadelpha, PA, USA
a libertarian | 001-215-487-3811 shady.com,kevin bbs.cpcn.com,sysop
Con Torrisi
Kevin,
Thanks for sharing that experience. I am looking very closely at the network
side of this problem.
Your description is close to what we are experiencing. The cables were all made
by an "amateur" and I have been concerned about them for some time.
I have been chasing the virtual memory management of the system but that may
just be a symptom. Still very strange that 'vhand' reacts in such a way.
I am getting someone to check and certify all the cables and wiring.
Con
Con Torrisi
Bela
>
>I wonder if the network segment was going insane (bad termination or
>something like that). The stats from `ndstat -l` come out of the NIC
>driver, and it's up to that driver to actually maintain them. You could
>have had a billion bad packet interrupts and the driver didn't bother to
>log them. Can you swap in a different NIC (one that uses a completely
>different driver) -- just to see if it logs anything? I'm thinking of
>things like electrical noise on the line, that wouldn't show up as
>packets per se, so might not have shown on the "lights".
>
I am chassing the network side of this as a potential source and I am getting
some interesting stats. Can you make any sense out of this.
I am getting a steady count of "No STREAMS Buffers" out of ndstat. While
netstat -m shows no failures. (Still haven't replaced the network card and
driver yet)
At the moment of the event I have a script that pings various machines at 10
second intervals.
If I ping the NT box I sometimes get the message "ping: socket: Out of stream
resources"
If I ping the SCO box itself I sometimes get "ping: unknown host ......" (It
doesn't even ping itself)
Later (maybe 20 seconds) the pings work but drop packets.
Even later (minutes) all is okay again but the system is still stressed vhand
hogging the cpu.
Is this "Out of stream resources" a symptom or a cause?
After 3 hours uptime we have 329 no STREAMS Buffers errors.
Others have suggested cable problems and we are systematically changing all the
cables. I am organising a cable tester and talking to someone about installing a
network analyser.
Con
Jeff Liebermann
>If I ping the SCO box itself I sometimes get "ping: unknown host ......" (It
>doesn't even ping itself)
Let's seperate the sockets and streams buffer issues here.
If you're running out of sockets, you are being attacked with a
SYN flood. If you are running out of streams buffers on a
machine that allegedly has dynamic streams buffer allocation, you
have either a major traffic overload, or the upper limit of
memory available for such occassions is depleted. SCO just
posted OSS469A for 3.2v5.0.4 which includes fixes for various
denial of service attacks. I quote the attached docs at:
ftp://ftp.sco.com/SLS/oss469a.ltr
6. Incoming udp broadcasts can bring down a system by using up
all its streams buffers. This may be caused because a program,
such as scologin, has stopped reading incoming packets, or by a
flood of incoming packets which are arriving too fast for a slow
system.
You should see this kind of traffic on an external packet
sniffer. I use Netmon borrowed from MS SMS 1.1 on NT4 for the
purpose.
>Later (maybe 20 seconds) the pings work but drop packets.
Pinging itself drops packets? It takes a while for all the
paritally open socket connections to time out and recover. Since
the attack built up gradually, it will timeout and recover at the
same rate.
>Is this "Out of stream resources" a symptom or a cause?
According to my conspiracy theory, it's a symptom.
>After 3 hours uptime we have 329 no STREAMS Buffers errors.
netstat -m statistics are cummulative and do not reset until you
either reboot or restart tcp/ip.
>Others have suggested cable problems and we are systematically changing all the
>cables. I am organising a cable tester and talking to someone about installing a
>network analyser.
It's possible. I've had flakey cables cause some rather bizarre
symptoms. I solve such problems by disconnecting sections of the
LAN and isolating the problem area. It gets to be really
entertaining when there are multiple cable problems. About 2
years ago, Asante was shipping silver coloured 10baseT cables
with their Mac ethernet products. This silver cable wasn't even
twisted pair, had no markings at to the category level (i.e.
CAT3, CAT5) and only worked because it was relatively short. I
think it was some kind of surplus telephone wire. So far, I've
fixed two High Schools, on hospital, and several large lans with
problems ranging from rotten performance, high error rates,
disappearing machines, and exessively busy servers. I have seen
an exessively busy system slow down when faced with heavy retries
due to these crummy silver coloured cables, but not to the
extreme that you're seeing.
Your cable testing orgy may not be successful if you are simply
testing the cable. I've also seen mangled RJ45 sockets where the
wire pins that form the contacts and bent or out of place. This
requires only a visual inspection to detect, but if you have
hundreds of them, it's no fun. If you have CAT5 wiring in the
wall, the wall jacks and interior wiring is another point of
failure.
I once partially crimped an RJ45 connector on a patch cable. The
gold pins were sticking out about 2mm higher than acceptable. I
destroyed two cheap hubs and one ethernet card before I realized
what had happened. Once the RJ45 socket (receptacle) pins get
mashed, it's almost impossible to get a decent connection.
Intermittant wiring follows a pattern. In general, a loose
connection will be evident when things are moved. One of those
things is the building. When people are present or temperature
changes are occurring, the intermittant will be evident. When
things are quiet and stable, then the loose coonections are also
stable. Slam a few doors and see if you can create the problem.
At this point, I have to sift through the "evidence" and
intentionally ignore the irrelivent to reach an adequate wild
guess. The fact that BOTH the NT box and SCO box are affected
points to a network originated problem. Most (not all) of the
symptoms points to a SYN flood DOS attack (out of sockets).
Crummy wiring is possible, but my experience is that such things
follow a different pattern of unreliability. I don't believe the
wiring theory. Ethernet cards have been known to go insane and
cause similar problems. However, my experience is that they
affect the entire network and not just two machines. I vote for
a SYN flood DOS attack.
[x]email [x]news [ ]mailing list
--
Jeff Liebermann 150 Felker St #D Santa Cruz CA 95060
(408)699-0483 pgr (408)426-1240 fax (408)336-2558 home
Con Torrisi
Jeff Liebermann replied with:
> I vote for a SYN flood DOS attack.
Early in the piece I suspected a SYN flood so I installed SSE010. It made no
difference.
Also at the moment of the event I run "netstat -a -f inet" which I am led to
believe will show a large number of unacknowldged SYN requests if its a SYN
flood. This looks normal to me, only the odd SYN_RCVD and always from a local
dial-in.
> If you're running out of sockets, you are being attacked with a SYN flood.
I don't think I have seen a "No Sockets" error yet. The message I got was:
"ping: socket: Out of stream resources". socket is complaining that there is no
stream resource.
>If you are running out of streams buffers on a machine
>that allegedly has dynamic streams buffer allocation, you
>have either a major traffic overload, or the upper limit of
>memory available for such occassions is depleted.
'netstat -m' shows we never reach the upper limit of memory available to
streams. Traffic overload fits the scenario but where is it coming from, and so
sudden and so overwhelming.
The other network devices i.e. router and terminal server respond to ping and
behave normally durring the event. But these devices are "less intelegent".
I chased the DoS aspect for a long time but can't find and solid evidence its
coming from the Internet. (Not to say its not)
Seeing I have sse010 do you think I should replace it with oss469a.??
Con
Con Torrisi
Bela Lubkin
Con Torrisi wrote:
> Jeff Liebermann replied with:
> > I vote for a SYN flood DOS attack.
>
> Early in the piece I suspected a SYN flood so I installed SSE010. It made no
> difference.
Neither sse010 nor oss469a (or oss468a) address SYN floods.
> Also at the moment of the event I run "netstat -a -f inet" which I am led to
> believe will show a large number of unacknowldged SYN requests if its a SYN
> flood. This looks normal to me, only the odd SYN_RCVD and always from a local
> dial-in.
So you're not being SYN flooded.
> >If you are running out of streams buffers on a machine
> >that allegedly has dynamic streams buffer allocation, you
> >have either a major traffic overload, or the upper limit of
> >memory available for such occassions is depleted.
> 'netstat -m' shows we never reach the upper limit of memory available to
> streams. Traffic overload fits the scenario but where is it coming from, and so
> sudden and so overwhelming.
`netstat -m` shows that you aren't pushing its limits. So a network
buffer allocation will not fail because it's hitting the STREAMS-imposed
limit (NSTRPAGES). However, since it's dynamically allocated, a STREAMS
allocation can fail because the *kernel* doesn't have any memory to
give. Yet `scosar -E` shows that there's plenty of availsmem (~37000,
right?)
Very strange.
> The other network devices i.e. router and terminal server respond to ping and
> behave normally durring the event. But these devices are "less intelegent".
>
> I chased the DoS aspect for a long time but can't find and solid evidence its
> coming from the Internet. (Not to say its not)
I wonder if you could have the subnet isolated while it is acting weird.
When you notice it's acting weird, start up various monitoring (sar
collecting data to a file, etc.), then cut the cable. I understand that
you aren't at the site, so this will disconnect you. If you can have
some local person just simply disconnect the hub from the rest of the
network, leave it alone for 30 minutes, then reconnect it, you might
learn a lot. For the disconnected period, I'm assuming that the
OSR5.0.4 system, the NT system, terminal server and router would still
all be connected together, but the router's port to the rest of the
world would be disconnected.
> Seeing I have sse010 do you think I should replace it with oss469a.??
sse010 fixes only the various weird-IP-packets DoS attacks. oss469a
fixes dozens of other problems, including what sse010 fixes.
>Bela<
Con Torrisi
-----Original Message-----
From: Bela Lubkin <be...@sco.com>
Newsgroups: comp.unix.sco.misc
To: post article to comp.unix.sco.misc; please followup there
<sco...@xenitec.on.ca>
Cc: Con Torrisi <c...@bretts.com.au>
Date: Tuesday, 28 April 1998 20:48
Subject: Re: SCO 5.0.4 Slows and vhand spins wildly.
Bela,
An update for you:
>`netstat -m` shows that you aren't pushing its limits. So a network
>buffer allocation will not fail because it's hitting the STREAMS-imposed
>limit (NSTRPAGES). However, since it's dynamically allocated, a STREAMS
>allocation can fail because the *kernel* doesn't have any memory to
>give. Yet `scosar -E` shows that there's plenty of availsmem (~37000,
>right?)
Right
>
>Very strange.
Strange indeed
>I wonder if you could have the subnet isolated while it is acting weird.
>When you notice it's acting weird, start up various monitoring (sar
>collecting data to a file, etc.), then cut the cable. I understand that
>you aren't at the site, so this will disconnect you. If you can have
>some local person just simply disconnect the hub from the rest of the
>network, leave it alone for 30 minutes, then reconnect it, you might
>learn a lot. For the disconnected period, I'm assuming that the
>OSR5.0.4 system, the NT system, terminal server and router would still
>all be connected together, but the router's port to the rest of the
>world would be disconnected.
Good idea. I'm trying to organise this.
I mentioned the NT box on the network was not responding to a ping during
the event. It turns out the someone had set its gateway as the SCO box, so
when it's down the NT box had no gateway. I set it to point at the router
and now the NT box is always up, even during the event.
I have also replaced all the LAN cables and installed NetXray which is not
showing any network problems apart from the SCO system going "off the air"
for a time.
So, it doesn't seem to be a network generated problem.
I am still trying to get the owner to organise a replacement Network card. I
have asked for an Intel EtherExpress 100B. The current card is a 3Com.
Grasping at straws here but because "ndstat -l" is reporting thousands of
"No STREAMS Buffers" and netstat -m says no problem then maybe the Card or
Driver is a dud.
Odd thing. I reboot the system and ndstat shows a steady increase in the
number of " No STREAMS Buffers". The event occurs and the count remains
stable. How does this figure?
Like you say Bela, "Very Strange"
Regards
Con
Paul Fischer of BTG Inc. Office Automation
I have seen something similar in my network. How long does it take
for the SCO box to die? Try killing snmpd and syslogd the next time
the sco box locks up. See if it opens up again.
Con Torrisi" "Bela Lubkin (tor...@bretts.com.aubelal@sco.com) wrote:
: -----Original Message-----
: Bela,
: Strange indeed
: Regards
: Con
--
Paul Fischer Internet Administrator BTG, Inc.
(703) 761-6644 (703) 556-9290 (fax) 1945 Old Gallows Rd
"I have a notebook, therefore I will succeed!" Vienna, VA 22182
Con Torrisi
-----Original Message-----
From: Paul Fischer of BTG Inc. Office Automation <pa...@pfischer.btg.com>
Newsgroups: comp.unix.sco.misc
To: distri...@xenitec.on.ca <distri...@xenitec.on.ca>
Date: Saturday, 23 May 1998 12:48
Subject: Re: SCO 5.0.4 Slows and vhand spins wildly.
>I have seen something similar in my network. How long does it take
>for the SCO box to die? Try killing snmpd and syslogd the next time
>the sco box locks up. See if it opens up again.
>
>Con Torrisi" "Bela Lubkin (tor...@bretts.com.aubelal@sco.com)
wrote:
>
-----Snip------
...........
-----Snip------
Paul,
Problem has dissappeared. Have not had this problem since replacing the
network card and driver. However, I now get a runaway streams memory usage
at about the same interval as the vhand problem. It sits at about 1800Kb
usage then for no apparent reason it will blow out. Some time it goes to the
max memory available and locks up the network, sometimes it goes to about
5000Kb used and stops and retreats back to the normal setting.
So, I think the original problem was network related as well. I have
overcome the streams problem by setting the limit to 9000Kb. Seems a waste
but it keeps the system alive. I feel sure there is a bug in streams memory
management because if I set cron to run "netstat >/dev/null" 5 times every
few minutes the memory usages stays very low. Stop this and it will start
climbing.
When streams memory usage is high "netstat -m" doesn't seem to show where
its being used.
Cheers
Paul Fischer
Please try something for me. Stop the cron job for a while and let the
streams usage climb. Use netstat -m to determine when it has climbed
quite a bit. Then, do a "kill -9" on syslogd and snmpd (and hostmib if
you are running it). Then immediately do a "netstat -m" and see if your
buffers have been freed up.
I am seeing a streams memory leak problem under OSR 5.0.2, 5.0.4, and UW7.
If I don't run syslog or snmpd, I don't have a problem. This all started
for me when I put a Sparc with virtual intefaces on the network. So far I
don't know if anyone has been able to reproduce it.
I will also try to run "netstat > /dev/null" on a few of my systems and
turn syslog and snmpd back on. Maybe your fix will help me. I would
still like to know if mine helps you.
On Sat, 23 May 1998, Con Torrisi wrote:
> Date: Sat, 23 May 1998 16:32:51 +1000
> From: Con Torrisi <c...@bretts.com.au>
> To: SCO Support Newsgroup <sco...@xenitec.on.ca>,
> "Paul Fischer of BTG Inc. Office Automation" <pa...@btg.com>
=-=- =-=- =-=- =-=- =-=- =-=- =-=- =-=- =-=- =-=- =-=- =-=- =-=- =-=- =-=- =-=-
|"I do not know when I shall depart on my last journey, and still less do I |
|know what will be my destination, but you may be sure that if St. Peter |
|shows any reluctance to open the door I shall use your name without scruple."|
| - Joseph Conrad |=-=- =-=- =-=- =-=- =-=- =-=- =-=- -=-=- -=-=- =|
=-=- =-=- =-=- =-=- =-=- =-=-| HTTP://www.btg.com/~paul |
|Paul Fischer (703) 383-6428 BTG, Inc. 3877 Fairfax Ridge Rd. |
|Internet Admin (703) 383-4065 (fax) pa...@btg.com Fairfax, VA 22030 |
=-=- =-=- =-=- =-=- =-=- =-=- =-=- =-=- =-=- =-=- =-=- =-=- =-=- =-=- =-=- =-=-
Graham Broadbridge
Paul Fischer <pa...@btg.com> wrote:
: I am seeing a streams memory leak problem under OSR 5.0.2, 5.0.4, and UW7.
: If I don't run syslog or snmpd, I don't have a problem. This all started
: for me when I put a Sparc with virtual intefaces on the network. So far I
: don't know if anyone has been able to reproduce it.
I've seen something similar under 5.0.2 and 5.0.4.
In my case it was the PC interface server that was installed that was
causing the problem.
Just another data point :-)
Graham.
--
-------------------------------------------------------------------------------
Graham Broadbridge Home <gra...@peachy.apana.org.au>
Marsfield NSW Work <gra...@nsw.unilink.oz.au>
Australia AmprNet <vk2...@gw.vk2yui.ampr.org>
<vk2...@amsat.org>
Netmeeting: peachpitt.apana.org.au
-------------------------------------------------------------------------------