problem with top, reboot, and kill, etc..
Andrew Gower
I have 2 identical redhat 7.1 linux servers which both have the same
reocurring problem.
After the server has been running for a few days all of a sudden it develops
a fault, wherby none of the server software on the machine will accept an
external IP connection. I can only log into the computer from another
machine on the same subnet
When I log in I find that
a) top won't work, (it just sits there doing nothing, until I press ctrl+d
to kill it)
b) reboot won't work (it says it's going to reboot, but doesn't, I have to
kill the power to reset!)
c) kill won't work properly, some things will die, but my server software
either stays there, or the process is renamed to 'defunct' (what does this
mean?), I can kill it with a kill -9 though
d) even after I have killed every single running process, both uptime and
proc/meminfo still report that the server is under the same load that it was
when the problem occured.
e) downing the eth0 interface, reloading my network card module, and upping
eth0, does NOT fix the problem, it has no effect
f) sometimes (but not always) there is an extra symptom, where every
keypress I type into SSH is delayed by one, e.g if I type 'reboot', it will
display 'reboo', until I press some more keys.
The only way I have to found to fix this is with a hard reset of the power!
It is getting incredibly annoying having to do this every few days because
these are remote servers which I have no direct access to! I'm absolutely
desperate to fix this problem. It looks to me like something has gone wrong
with /proc and/or the tcp/ip stack but I have no idea how what is causing
it.
My server:
Redhat 7.1, 2.4.9 kernel, Ai7cxxx SCSI controller, 2x1ghz P3, Intel E100 net
card (I am using the e100.o driver not the eepro100.o which had even more
problems). 512meg ECC ram
hope someone can help, or point me in the right direction :-)
Thanks
Andrew
Kosh Vader
>a fault, wherby none of the server software on the machine will accept an
>external IP connection. I can only log into the computer from another
>machine on the same subnet
>c) kill won't work properly, some things will die, but my server software
>either stays there, or the process is renamed to 'defunct' (what does this
>mean?), I can kill it with a kill -9 though
>d) even after I have killed every single running process, both uptime and
>proc/meminfo still report that the server is under the same load that it was
>when the problem occured.
Almost sounds to me like there are some processes that are badly
chewing up system resources (i.e. memory). Have you seen a message
similar to "unable to load interpreter: /lib/ld-linux.so.2"? This is
indicative of memory--physical and virtual--being nearly exhausted.
Definition:
A "defunct", or "zombie", process is child process that, when it dies,
fails to be cleaned up by its parent process. This is usually due to
the fact the parent process has failed to receive notification of its
child's termination. [Parent, child, zombie... I didn't come up with
these terms! :) ] Normally, when a process finishes, it calls the
exit system call. It is the kernel's job at that point to release all
system resources that have been previously allocated to the process,
save its exit status, and leave the process ID in the process table.
It's up to the parent process to check the exit status of its child
process by invoking the wait system call, and to tell the kernel to
completely delete the process.
As init (process ID) is parent to all first generation processes, a
command like the following should clean up any zombie processes
(first-generation only):
kill -s SIGHUP 1 or killall -HUP init
The killing off of the parent process cleans up any of its zombie
children. Occasionally, however, when the parent process dies
abnormally, child processes that are still running become "orphan"
processes, and are then owned by init. Orphan processes, unlike
zombie processes, do not release there system resources.
Chris
On Tue, 6 Nov 2001 09:42:12 -0000, "Andrew Gower" <ne...@jagex.com>
wrote:
>Hi,
>I have 2 identical redhat 7.1 linux servers which both have the same
>reocurring problem.
>
>
>When I log in I find that
>a) top won't work, (it just sits there doing nothing, until I press ctrl+d
>to kill it)
>b) reboot won't work (it says it's going to reboot, but doesn't, I have to
>kill the power to reset!)
Andrew Gower
ok it just happened again and so I've done some more tests, and I've noticed
something verrry interesting
I have noticed that after the system breaks the clock stops working until I
reboot!
I can type 'date' repeatedly and get the same time repeatedly. What's more
typing 'sleep 1' will cause it to sleep forever (presumably because of the
broken clock), which I strongly suspect is why reboot and many other things
have stopped working
however the hardware clock IS still working, so it looks like a software
fault
[admin@localhost admin]$ /sbin/hwclock
Wed 07 Nov 2001 09:39:18 AM EST 0.585391 seconds
[admin@localhost admin]$ /sbin/hwclock
Wed 07 Nov 2001 09:39:54 AM EST 0.380249 seconds
I notice that uptime has locked up at the same constant load, and top won't
work, but I suspect that is because both these things also rely on the
clock. I think all the symptoms I am seeing could easily be a result of this
failure.
Soo... any idea a) why my clock is stopping, and b) how I can fix it? :-)
Thanks
Andrew
"Kosh Vader" <kosh...@vorlonempire.org> wrote in message
news:q5gfut4atmkun92pl...@4ax.com...
Kosh Vader
In the meantime, can you just a short list of some the daemons that
are running at any given time, say at boot up?
Secondly, what daemon process is leaving zombies processes laying
aroung? Just the name.
Lastly, can you give any vitals, both at boot up time and "down" time?
For example,
cat /proc/meminfo
cat /proc/interrupts
cat /proc/loadavg
vmstat
Chris
On Wed, 7 Nov 2001 00:50:44 -0000, "Andrew Gower" <ne...@jagex.com>
Andrew Gower
thanks for you help :-)
Here are my systems vitals when everything is running normally (e.g BEFORE
it breaks)
----
[admin@localhost admin]$ ps -ax
PID TTY STAT TIME COMMAND
1 ? S 0:05 init [3]
2 ? SW 0:00 [keventd]
3 ? SWN 0:00 [ksoftirqd_CPU0]
4 ? SWN 0:00 [ksoftirqd_CPU1]
5 ? SW 0:04 [kswapd]
6 ? SW 0:00 [kreclaimd]
7 ? SW 0:00 [bdflush]
8 ? SW 0:00 [kupdated]
10 ? SW 0:00 [scsi_eh_0]
11 ? SW 0:00 [scsi_eh_1]
444 ? S 0:00 syslogd -m 0
449 ? S 0:00 klogd -2
610 ? S 0:00 /usr/sbin/atd
625 ? S 0:01 /usr/sbin/sshd
723 ? S 0:00 crond
784 tty1 S 0:00 /sbin/mingetty tty1
785 tty2 S 0:00 /sbin/mingetty tty2
786 tty3 S 0:00 /sbin/mingetty tty3
787 tty4 S 0:00 /sbin/mingetty tty4
788 tty5 S 0:00 /sbin/mingetty tty5
789 tty6 S 0:00 /sbin/mingetty tty6
976 ? S 17:04 /home/admin/thttpd-2.20b/thttpd -t
/home/admin/throttle -d /home/admin/web
980 ? S 0:00 /bin/bash
/home/admin/runescape/server/startserver.unix
983 ? S 43:00
/home/admin/IBMJava2-13/jre/bin/exe/java -mx100663296 mudserver 4 43594
992 ? S 0:00
/home/admin/IBMJava2-13/jre/bin/exe/java -mx100663296 mudserver 4 43594
993 ? S 0:00
/home/admin/IBMJava2-13/jre/bin/exe/java -mx100663296 mudserver 4 43594
994 ? S 0:00
/home/admin/IBMJava2-13/jre/bin/exe/java -mx100663296 mudserver 4 43594
995 ? S 0:00
/home/admin/IBMJava2-13/jre/bin/exe/java -mx100663296 mudserver 4 43594
996 ? S 0:02
/home/admin/IBMJava2-13/jre/bin/exe/java -mx100663296 mudserver 4 43594
998 ? S 0:00
/home/admin/IBMJava2-13/jre/bin/exe/java -mx100663296 mudserver 4 43594
999 ? S 0:09
/home/admin/IBMJava2-13/jre/bin/exe/java -mx100663296 mudserver 4 43594
1002 ? S 0:00 /bin/bash
/home/admin/runescape2/server/startserver.unix
1005 ? S 43:19
/home/admin/IBMJava2-13/jre/bin/exe/java -mx100663296 mudserver 3 43596
1014 ? S 0:00
/home/admin/IBMJava2-13/jre/bin/exe/java -mx100663296 mudserver 3 43596
1015 ? S 0:00
/home/admin/IBMJava2-13/jre/bin/exe/java -mx100663296 mudserver 3 43596
1016 ? S 0:00
/home/admin/IBMJava2-13/jre/bin/exe/java -mx100663296 mudserver 3 43596
1017 ? S 0:00
/home/admin/IBMJava2-13/jre/bin/exe/java -mx100663296 mudserver 3 43596
1018 ? S 0:02
/home/admin/IBMJava2-13/jre/bin/exe/java -mx100663296 mudserver 3 43596
1022 ? S 0:08
/home/admin/IBMJava2-13/jre/bin/exe/java -mx100663296 mudserver 3 43596
1297 ? S 0:00 /usr/sbin/sshd
1298 pts/1 S 0:00 -bash
1333 pts/1 R 0:00 ps -ax
I am running 2 java programs which are my main servers, they show up several
times because java is using native threads. The java processes are running
as user 'admin', thttpd is running as user 'nobody', pretty much everything
else is 'root'.
After the box breaks these java processes stall (I suspect they are stuck on
a sleep command due to the stopped clock), and won't die properly. I kill
the main processes with 'kill 983' and 'kill 1005' respectively, which would
normally tidy up the children too, but once they are stalled it doesn't,
(the parent dies, but everything else sticks around), and I have to kill the
other threads with a 'kill -9'.
[admin@localhost admin]$ cat /proc/meminfo
total: used: free: shared: buffers: cached:
Mem: 524820480 399667200 125153280 0 108023808 39903232
Swap: 1077469184 0 1077469184
MemTotal: 512520 kB
MemFree: 122220 kB
MemShared: 0 kB
Buffers: 105492 kB
Cached: 38968 kB
SwapCached: 0 kB
Active: 8380 kB
Inact_dirty: 136080 kB
Inact_clean: 0 kB
Inact_target: 4 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 512520 kB
LowFree: 122220 kB
SwapTotal: 1052216 kB
SwapFree: 1052216 kB
[admin@localhost admin]$ cat /proc/interrupts
CPU0 CPU1
0: 1418892 1427230 IO-APIC-edge timer
1: 2 2 IO-APIC-edge keyboard
2: 0 0 XT-PIC cascade
8: 0 1 IO-APIC-edge rtc
16: 11882 11942 IO-APIC-level aic7xxx
17: 6 10 IO-APIC-level aic7xxx
18: 79968754 79966137 IO-APIC-level e100
NMI: 0 0
LOC: 2846441 2846400
ERR: 0
MIS: 0
[admin@localhost admin]$ cat /proc/loadavg
0.13 0.11 0.07 1/42 1338
[admin@localhost admin]$ vmstat
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us sy
id
0 0 0 0 122200 105492 38980 0 0 4 1 598 33 8 6
86
-----
and here are some results AFTER it has broken, I don't have all the info
yet, because it hasn't broken again since you asked, but I do have some
stuff
[admin@localhost admin]$ uptime
9:19am up 21:33, 4 users, load average: 1.21, 0.90, 0.77
[admin@localhost server]$ more /proc/meminfo
total: used: free: shared: buffers: cached:
Mem: 524820480 293588992 231231488 0 89731072 89399296
Swap: 1077469184 6459392 1071009792
MemTotal: 512520 kB
MemFree: 225812 kB
MemShared: 0 kB
Buffers: 87628 kB
Cached: 80996 kB
SwapCached: 6308 kB
Active: 76108 kB
Inact_dirty: 97572 kB
Inact_clean: 1252 kB
Inact_target: 16 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 512520 kB
LowFree: 225812 kB
SwapTotal: 1052216 kB
SwapFree: 1045908 kB
Thanks
Andrew
"Kosh Vader" <kosh...@vorlonempire.org> wrote in message
news:u23hut4u95bimqnkk...@4ax.com...
Andrew Gower
information which was missing before.
[admin@localhost admin]$ cat /proc/interrupts
CPU0 CPU1
0: 4227334 4237028 IO-APIC-edge timer
1: 2 2 IO-APIC-edge keyboard
2: 0 0 XT-PIC cascade
8: 0 1 IO-APIC-edge rtc
16: 22968 23012 IO-APIC-level aic7xxx
17: 6 10 IO-APIC-level aic7xxx
18: 246695388 246684802 IO-APIC-level e100
NMI: 0 0
LOC: 11350129 11350118
ERR: 0
MIS: 0
[admin@localhost admin]$ cat /proc/loadavg
0.89 0.82 0.74 2/42 1831
[admin@localhost admin]$ vmstat
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us sy
id
1 0 0 5908 9884 93480 67784 0 0 1 1 174 40 25 6
69
Kosh Vader
Chris
On Thu, 8 Nov 2001 08:37:54 -0000, "Andrew Gower" <ne...@jagex.com>
wrote:
>It just crashed again, once again the system clock has stopped. Here is the