Dying processes (inetd, cron, syslogd, sshd)
ke...@actual-systems.com
We are having problems on various SMP machines (5.0.6a + rs506a
installed) where at times of large load most of the running processes
just seem to stop (e.g. inetd, cron, syslogd, sshd,....) This always
seems to occur at times of large stress to the disks, but we have never
managed to put our fingers on exactly what is causing it. When it does
happen not only does the inetd process die, but also cron and syslog
which makes it very tricky for us to put anything in place to try and
catch what is happening.
We are able to ping the machine when it does happed and also login at
the console and over a modem but not over a telnet of ssh connection.
We have had an issue open with SCO before who advised us to install
scodb and set it to trigger when the inetd process stops - and when it
does to get a sysdump. We have tried this, but the sysdump created was
too big for swap - do you know of any way from within scodb to reduce
the size of the sysdump created?
This machine (which has had the problem once a day for the last three,)
is used as a backup server in our office. All that runs on it is two
rsync's of our main machine - one for mail/uucp spools, and one for the
main data. The problem always has occured during these rsyncs, normally
when transferring a large file.
Any help on this would be much appreciated - or even advice as to which
way to look to find the answer to what is happening would be much
appreciated.
Again, thanks,
Keith
Simon Hobson
(in message <1122912804.9...@g44g2000cwa.googlegroups.com>):
> We are having problems on various SMP machines (5.0.6a + rs506a
> installed) where at times of large load most of the running processes
> just seem to stop (e.g. inetd, cron, syslogd, sshd,....) This always
> seems to occur at times of large stress to the disks, but we have never
> managed to put our fingers on exactly what is causing it. When it does
> happen not only does the inetd process die, but also cron and syslog
> which makes it very tricky for us to put anything in place to try and
> catch what is happening.
>
> We are able to ping the machine when it does happed and also login at
> the console and over a modem but not over a telnet of ssh connection.
When you say stop, do you mean that the process dies (quits) or is still
there but no longer doing anything ?
Can you kill a process which has stopped ?
It sounds similar to a problem we had at work when we first got our IBM
server (X250) with ServRaid controller. There was a known bug in the firmware
which would result in disk writes failing to complete in a random and
intermittent manner - it was fixed by a firmware update.
The symptoms were that every now and then (fortunately it came to light
during pre-changover testing) there would be an error in the raid controller
log, NOT the OS logs, and the write would never complete. So the system would
be in a position where the raid controller thinks it's had an error and never
completes the operation, and the OS is waiting for the controller to either
complete or report an error (which it never did). The result of this is that
whatever process was trying to write would halt waiting for the IO to
complete, and since it was waiting at kernel level (I assume) the process was
unkillable.
What's more, any other process that tried to access the block in question
would also halt waiting for the IO request to complete. One day, the fault
was somewhere important and over a period of a few hours, every process 'just
stopped' and the only thing left was to hit the power switch. When the thing
rebooted and we looked at the data, it became apparent that the system had
stopped writing data the day before (I guess the buffer flushing process got
stuck) !
Simon
ke...@actual-systems.com
disappear.
The machine is still running and we can login to the console and work
so it sounds different to your issue.
Keith
ke...@actual-systems.com
What would be the outcome if you had one process that kept on wanting
more and more resource?
Do other processes hold onto the resource they have or will they
eventually get 'bullied' out of the resource they are using and
essentially stop (which theoretically would give the results I am
seeing.)
Any ideas? or does anyone have any idea's as to how I would track down
what was causing this to happen.
Thanks again,
Keith
Bela Lubkin
> Anyone have any idea's on this problem?
I posted on August 1st, but never saw it come back to me. This time I'm
Bcc'ing you so you'll see it even if USENET swallows it again...
ke...@actual-systems.com wrote:
> We are having problems on various SMP machines (5.0.6a + rs506a
> installed) where at times of large load most of the running processes
> just seem to stop (e.g. inetd, cron, syslogd, sshd,....) This always
> seems to occur at times of large stress to the disks, but we have never
> managed to put our fingers on exactly what is causing it. When it does
> happen not only does the inetd process die, but also cron and syslog
> which makes it very tricky for us to put anything in place to try and
> catch what is happening.
>
> We are able to ping the machine when it does happed and also login at
> the console and over a modem but not over a telnet of ssh connection.
>
> We have had an issue open with SCO before who advised us to install
> scodb and set it to trigger when the inetd process stops - and when it
> does to get a sysdump. We have tried this, but the sysdump created was
> too big for swap - do you know of any way from within scodb to reduce
> the size of the sysdump created?
>
> This machine (which has had the problem once a day for the last three,)
> is used as a backup server in our office. All that runs on it is two
> rsync's of our main machine - one for mail/uucp spools, and one for the
> main data. The problem always has occured during these rsyncs, normally
> when transferring a large file.
scodb can't reduce the size of a crash dump, but you can force the dumps
to fit by limiting the amount of memory seen by the kernel. To do this,
append " mem=1m-100m" to DEFBOOTSTR in /etc/default/boot (substituting a
bit less than actual size of your dump area in place of "100m").
The load you describe would probably run in 12MB of RAM, but don't limit
memory more than you have to. The problem might be memory size-related.
You want to keep as much as you can of the machine's normal memory size.
[new material begins]
> What would be the outcome if you had one process that kept on wanting
> more and more resource?
There are some problem scenarios like that. A common one is a process
spinning out of control, allocating more and more memory. It will
eventually use all available memory; its next allocation attempt will
fail, and in most cases it will then die. Unless you have changed the
defaults, such a process usually writes a core dump. On OSR5, during
the dumping of a process's core, the process continues to own all of its
memory until the dump is complete. This means that the machine remains
critically out of memory for a long time. The process may have grown
nearly as large as your combined RAM + swap. To dump it, not only does
the kernel have to write that much data, it also may have to page a
large portion of it in from swap. This can take many minutes with large
memory and a slow disk...
During that period, other processes that try to allocate memory will
usually fail. Their subsequent behavior depends on their error
handling. Some will dump core, some will exit gracefully, some may even
stay up. And some may get into weird catatonic states.
> Do other processes hold onto the resource they have or will they
> eventually get 'bullied' out of the resource they are using and
> essentially stop (which theoretically would give the results I am
> seeing.)
For memory, a "hog" process will cause others to get written out to
swap, but those processes still "own" their memory (it will get paged
back in if they need to access it). The troubles happen when a process
tries to allocate more memory while the system is strapped.
There are probably other resources where similar things could happen.
> Any ideas? or does anyone have any idea's as to how I would track down
> what was causing this to happen.
If you had a process spin out and dump, it would leave a huge core file
that you would be able to find. If a process spins out and dies
_without_ leaving a dump, a more subtle trace is left. Normally, OSR5
doesn't use any swap at all; `swap -l` will have identical values in the
"blocks" and "free" columns. ("Normal" modern systems have enough RAM
that they never need to invoke the tremendous performance loss of
swapping.) After such an incident, `swap -l` will show quite a bit of
swap in use. This represents pages that got pushed out, and whose
processes have never actually needed to access them since the incident.
What does `crash` "p" show in the "EVENT" column for the hung processes?
>Bela<
ke...@actual-systems.com
I will give those things a try and get back to you on the results.
Hope all is going well for you now that you have left SCO.
Keith
ke...@actual-systems.com
Just a couple of updates on the suggestions Bela gave.
I've done a 'find' for any core files and didn't find any new ones, so
they are not being created - I have yet to have the problem happen
again so I havn't done a 'swap -l' to see if any of it is being used
when the problem occurs.
While I've not used 'crash' to see what the EVENT column shows - I
don't actually think it will show anything as a ps done after the
problem has occured doesn't show the processes - they have just
disappeared.
I've set 'mem=1m-100m' on both my machines here and will try to force
the problem to happen tonight and hopefully I will have more questions
tommorow.
Thanks again,
Keith
Bela Lubkin
> I've done a 'find' for any core files and didn't find any new ones, so
> they are not being created - I have yet to have the problem happen
> again so I havn't done a 'swap -l' to see if any of it is being used
> when the problem occurs.
Do at least one before the system goes weird, so you'll know what's
normal.
> While I've not used 'crash' to see what the EVENT column shows - I
> don't actually think it will show anything as a ps done after the
> problem has occured doesn't show the processes - they have just
> disappeared.
Ah, I don't think I had understood properly. Disappearing processes
could have been killed by a low-memory situation, but also by another
process deliberately killing them, or all sorts of other things.
> I've set 'mem=1m-100m' on both my machines here and will try to force
> the problem to happen tonight and hopefully I will have more questions
> tommorow.
Is that the actual size of swap on the machines? Like I said, you
shouldn't restrain memory tighter than necessary: use "mem=1m-%dm",
filling in the actual size of swap on each machine (short it by 1MB to
be sure).
>Bela<
ke...@actual-systems.com
within scodb (I was triggered into debug mode when the inetd process
stopped.)
I have a few questions now.
1. From looking at this sysdump how can I tell what has caused this
process to stop running?
2. Bela, in your previous answer you mentioned that 'Disappearing
processes could have been killed by a low-memory situation' - is there
any way to tell if this is what has happened (or does the answer to the
first question give me the answer to the second....)
Again thanks for all your help,
Keith
Bela Lubkin
> I have now had the problem happen again and have a sysdump created from
> within scodb (I was triggered into debug mode when the inetd process
> stopped.)
>
> I have a few questions now.
> 1. From looking at this sysdump how can I tell what has caused this
> process to stop running?
Exactly how did you trigger scodb on death of the process? If you had
it catch the right thing, you should be able to see what signal caused
the death. If that signal was raised by the process's own action (e.g.
`kill(getpid(), SIGTERM)', or by trying to allocate memory when none was
available) -- the evidence will be on its own stack. If it was killed
by another process (`kill(1234, SIGTERM)'), that process's stack will
tell the story.
Start by doing a "stack -p 1234", where 1234 is the PID of inetd.
> 2. Bela, in your previous answer you mentioned that 'Disappearing
> processes could have been killed by a low-memory situation' - is there
> any way to tell if this is what has happened (or does the answer to the
> first question give me the answer to the second....)
It might. But you can also examine the system's memory state. The 4
most interesting variables are the ones shown by `sar -r`, although one
is spelled differently: freemem, freeswap, availrmem and availsmem. You
can display them by just entering their names:
scodb:1> freemem
48E4B
scodb:2> freeswap
100000
scodb:3> availrmem
5DBE6
scodb:4> availsmem
146949
That corresponds to:
$ sar -r 1 1
SCO_SV deeptht 3.2v5.0.6 i80386 08/12/2005
15:40:42 freemem freeswp availrmem availsmem (-r)
15:40:43 299572 8388608 383979 1337833
`scodb` output is in hexadecimal. All of the variables are in 4K
pages; `sar -r` output is in 4K pages except "freeswp", which is in
512-byte sectors, so we need to multiply it by 8:
$ bc
ibase=16
48E4B; 100000*8; 5DBE6; 146949
298571
8388608
383974
1337673
The numbers are slightly different because I'm looking at a live system.
If your system is deeply out of memory, `availsmem' will be near 0.
Anything under about 1000 is desperate. `freemem' and `freeswap' may or
may not be near 0, depending on exactly what the memory-eating process
is doing. `availrmem' is unlikely to be low.
Lack of memory is just one possible cause...
Show me the "stack -p ..." output for inetd. That'll give me some
idea...
>Bela<
ke...@actual-systems.com
We were tripped into scodb as we had setup a watch point on the inetd
process.
We work out the pid of inetd using crash (p | grep inetd) - then we
convert it to hex and go into scodb.
-------------
# crash
dumpfile = /dev/mem, namelist = /unix, outfile = stdout
> p | grep inetd
29 s 256 1 256 0 76 0 selwait inetd load nxec
> q
(PID 29=1D in hex)
Ctrl+alt+D
debug0:1> bp wl &proc[1d].p_utime
debug0:2> q
--------------------------------
Then once this problem occured we were in debug mode and I did the
following to get the dumpfile I have.
-------------------
debug0:4> bp dis *
debug0:5> sysdump()
debug0:6> bc *
debug0:7> q
# sysdump -i /dev/swap -fbhm -o /usr/tmp/dump
---------------------
Now the following is the output when I used scodb to look at the
dumpfile I have - I would have followed your advice about getting the
PID of inetd, but I think the process must have already stopped by the
time I had the dump so I just did 'stack' to give you everything and
then 'p' from within crash to get al the running processes.
----------------------------------------------------------------
back:/usr/tmp # scodb -d dump
dumpfile = dump
namelist = dump
stunfile = dump
varifile = dump
PID 0147: /etc/inetd
scodb:1>
scodb:1> stack
E0000DC0 exit2(0, 9) <- psig+18F
E0000DE8 psig(? 8057644, 0, 0, F01D4D98) <- systrap+39F
E0000E10 systrap(E0000E1C) <-
scall_nokentry+14
scodb:2> q
# crash -d dump
dumpfile = dump, namelist = dump, outfile = stdout
> p
PROC TABLE SIZE = 83
SLOT ST PID PPID PGRP UID PRI CPU EVENT NAME
FLAGS
0 s 0 0 0 0 95 0 runout sched
load sys lock nwak
1 r 1 0 0 0 80 0 init
load nwak
2 s 2 0 0 0 95 0 vhand vhand
load sys lock nwak nxec
3 r 3 0 0 0 95 0 bdflush
load sys lock nwak nxec
4 r 4 0 0 0 36 0 CPU1 idle proc
load sys lock nxec 2000000
5 p 5 0 0 0 36 0 CPU2 idle proc
load sys lock nxec 2000000
6 s 6 0 0 0 95 0 kmd_id kmdaemon
load sys lock nwak nxec
7 r 7 1 0 0 95 0 htepi_daemon
load sys lock nwak
8 s 8 0 0 0 95 0 pbintrpool strd
load sys lock nwak nxec
10 r 55 1 55 0 73 0 ifor_pmd
load nxec exit
11 z 57 55 55 0 76 0 zombie
nou nxec exit
12 r 52 1 50 0 80 0 syslogd
load nwak nxec exit
13 s 43 1 0 0 95 0 0xc1031150 htepi_daemon
load sys lock nwak
14 r 524 1 524 0 75 0 getty
load
15 r 85 1 71 0 75 0 strerr
load exit
18 r 525 1 525 0 75 0 getty
load
20 r 526 1 526 0 75 0 getty
load
21 r 276 1 276 0 81 0 cron
load nwak nxec exit
25 s 206 1 0 0 95 0 0xc1035150 htepi_daemon
load sys lock nwak
26 r 210 1 0 0 95 0 htepi_daemon
load sys lock nwak
29 z 568 1 0 204 76 0 zombie
nou exit
31 z 384 1 384 0 76 0 zombie
nou nxec exit
33 z 382 1 382 0 76 0 zombie
nou nxec exit
34 p 327 1 327 0 76 0 inetd
load nxec exit
35 r 17987 1 17987 0 80 0 getty
load nwak
41 r 392 1 384 0 75 0 lockd
load nxec exit
42 r 393 392 384 0 75 0 lockd
load nxec
43 r 394 392 384 0 75 0 lockd
load nxec
44 r 395 392 384 0 75 0 lockd
load nxec
45 r 396 392 384 0 75 0 lockd
load nxec
47 r 434 1 432 0 80 0 calserver
load nwak nxec exit
48 r 17869 1 276 0 76 0 rsync
load exit
49 r 442 1 442 0 81 0 caldaemon
load nwak exit
50 r 449 1 449 0 66 0 prngd
load nxec exit
53 r 527 1 527 0 75 0 getty
load
54 r 507 1 507 0 76 0 sshd
load nxec exit
55 r 528 1 528 0 75 0 getty
load
56 r 529 1 529 0 75 0 getty
load
57 r 530 1 530 0 75 0 getty
load
58 r 531 1 531 0 75 0 getty
load
59 r 532 1 532 0 75 0 getty
load
60 r 533 1 533 0 75 0 getty
load
61 r 534 1 534 0 75 0 getty
load
62 r 535 1 535 0 80 0 sdd
load nwak exit
63 z 17871 17869 276 0 88 0 zombie
nou exit
64 z 17240 276 276 0 73 0 zombie
nou exit
65 r 17872 1 276 0 75 0 rcmd
load nxec exit
68 s 17900 17869 276 0 95 0 0xfc2e3b90 rsync
load nwak nxec
> q
------------------------
And also looking at the memory as suggested shows it wasn't critically
close to it's limits (unless this is showing the freemem etc after the
inetd process has stopped.)
------------------------
scodb:2> freemem
6EEA0
scodb:3> freeswap
100000
scodb:4> availrmem
6FB0C
scodb:5> availsmem
16E941
------------------------
As always your help here is very much appreciated.
Thanks,
Keith
Bela Lubkin
> We were tripped into scodb as we had setup a watch point on the inetd
> process.
>
> We work out the pid of inetd using crash (p | grep inetd) - then we
> convert it to hex and go into scodb.
>
> -------------
> # crash
> dumpfile = /dev/mem, namelist = /unix, outfile = stdout
> > p | grep inetd
> 29 s 256 1 256 0 76 0 selwait inetd load nxec
> > q
>
> (PID 29=1D in hex)
>
> Ctrl+alt+D
> debug0:1> bp wl &proc[1d].p_utime
> debug0:2> q
Ok, that should work. You can skip the `crash` part, use "ps()" in
scodb and pick out inetd's PID there (already printed in hex). But
you can't pipe it to "grep inetd", so that's a bit of a negative...
> Then once this problem occured we were in debug mode and I did the
> following to get the dumpfile I have.
>
> -------------------
> debug0:4> bp dis *
> debug0:5> sysdump()
> debug0:6> bc *
> debug0:7> q
> # sysdump -i /dev/swap -fbhm -o /usr/tmp/dump
> ---------------------
>
> Now the following is the output when I used scodb to look at the
> dumpfile I have - I would have followed your advice about getting the
> PID of inetd, but I think the process must have already stopped by the
> time I had the dump so I just did 'stack' to give you everything and
> then 'p' from within crash to get al the running processes.
Again, "ps()" in scodb will do that.
The process was dying when the dump was taken.
Your three pieces of evidence here are inconsistent with each other.
You've set up to break when PID 0x1D is dying. The `scodb -d dump`
output from after the breakpoint says it's showing PID 0x147. And the
`crash -d dump` output has an inetd PID 0x34. So at best I have to
assume these are from three different events.
> back:/usr/tmp # scodb -d dump
> dumpfile = dump
> namelist = dump
> stunfile = dump
> varifile = dump
> PID 0147: /etc/inetd
> scodb:1>
> scodb:1> stack
> E0000DC0 exit2(0, 9) <- psig+18F
> E0000DE8 psig(? 8057644, 0, 0, F01D4D98) <- systrap+39F
> E0000E10 systrap(E0000E1C) <-
> scall_nokentry+14
> scodb:2> q
`exit2(0, 9)' is, I believe, delivering a signal 9 (SIGKILL) to the
process. There are few things inside the kernel that generate SIGKILL.
This suggests a user process is deliberately killing the dying
processes.
> # crash -d dump
> dumpfile = dump, namelist = dump, outfile = stdout
> > p
> PROC TABLE SIZE = 83
> SLOT ST PID PPID PRI EVENT NAME FLAGS
> 0 s 0 0 95 runout sched load sys lock nwak
> 1 r 1 0 80 init load nwak
> 2 s 2 0 95 vhand vhand load sys lock nwak nxec
> 3 r 3 0 95 bdflush load sys lock nwak nxec
> 4 r 4 0 36 CPU1 idle proc load sys lock nxec 2000000
> 5 p 5 0 36 CPU2 idle proc load sys lock nxec 2000000
> 6 s 6 0 95 kmd_id kmdaemon load sys lock nwak nxec
> 7 r 7 1 95 htepi_daemon load sys lock nwak
> 8 s 8 0 95 pbintrpool strd load sys lock nwak nxec
> 10 r 55 1 73 ifor_pmd <-- load nxec exit
> 11 z 57 55 76 zombie <-- nou nxec exit
> 12 r 52 1 80 syslogd <-- load nwak nxec exit
> 13 s 43 1 95 0xc1031150 htepi_daemon load sys lock nwak
> 14 r 524 1 75 getty load
> 15 r 85 1 75 strerr <-- load exit
> 18 r 525 1 75 getty load
> 20 r 526 1 75 getty load
> 21 r 276 1 81 cron <-- load nwak nxec exit
> 25 s 206 1 95 0xc1035150 htepi_daemon load sys lock nwak
> 26 r 210 1 95 htepi_daemon load sys lock nwak
> 29 z 568 1 76 zombie <-- nou exit
> 31 z 384 1 76 zombie <-- nou nxec exit
> 33 z 382 1 76 zombie <-- nou nxec exit
> 34 p 327 1 76 inetd <-- load nxec exit
> 35 r 17987 1 80 getty load nwak
> 41 r 392 1 75 lockd <-- load nxec exit
> 42 r 393 392 75 lockd load nxec
> 43 r 394 392 75 lockd load nxec
> 44 r 395 392 75 lockd load nxec
> 45 r 396 392 75 lockd load nxec
> 47 r 434 1 80 calserver <-- load nwak nxec exit
> 48 r 17869 1 76 rsync <-- load exit
> 49 r 442 1 81 caldaemon <-- load nwak exit
> 50 r 449 1 66 prngd <-- load nxec exit
> 53 r 527 1 75 getty load
> 54 r 507 1 76 sshd <-- load nxec exit
> 55 r 528 1 75 getty load
> 56 r 529 1 75 getty load
> 57 r 530 1 75 getty load
> 58 r 531 1 75 getty load
> 59 r 532 1 75 getty load
> 60 r 533 1 75 getty load
> 61 r 534 1 75 getty load
> 62 r 535 1 80 sdd <-- load nwak exit
> 63 z 17871 17869 88 zombie <-- nou exit
> 64 z 17240 276 73 zombie <-- nou exit
> 65 r 17872 1 75 rcmd <-- load nxec exit
> 68 s 17900 17869 95 0xfc2e3b90 rsync load nwak nxec
I deleted some less interesting columns from the table.
Notice the large number of processes with the "exit" flag. I marked
them with "<--" to make them visually obvious. These are all dying.
Does this list more or less match your experience of what dies in one of
these events? (Your mental list should be a little longer since there
are a few here that have gotten far enough along in dying that we can't
see what they were running, just "zombie").
To capture a better dump next time, I suggest rigging the system up so
that it breaks into scodb when _any_ of these processes die. That means
you'll catch it as early as possible -- possibly while the "attacker"
process is still issuing the SIGKILLs.
> And also looking at the memory as suggested shows it wasn't critically
> close to it's limits (unless this is showing the freemem etc after the
> inetd process has stopped.)
>
> scodb:2> freemem
> 6EEA0
> scodb:3> freeswap
> 100000
> scodb:4> availrmem
> 6FB0C
> scodb:5> availsmem
> 16E941
Right, that makes it look like it has nothing to do with memory.
> As always your help here is very much appreciated.
I'm wondering if we should take it offline -- I feel like I should be
looking directly at one of these dumps (hopefully one collected as early
as possible in an event).
Instead of setting up individual breakpoints for all the processes you
expect to die, you can use the fact that it's SIGKILL to your advantage.
Take a look at the code for kill():
scodb> u kill+175
kill+175 movl %eax,4(%edi)
kill+178 cmpl %eax,9
kill+17D jne kill+18C
kill+17F movb %al,-1(%ebp)
If your kernel matches the one I'm looking at here, kill+17F is an
address only reached during a `kill(pid, 9)' call. (If your kernel is
different: do "u kill", then type "/,9". It will scroll forward to the
right point in the disassembly. Hit <Return> a few times after that,
then "q" to exit the disassembler.)
Set a breakpoint on that ("bp kill+17F").
Test it by:
$ sleep 100 &
[1] 27388
$ kill -9 27388
... and you should drop into the debugger. Quit.
There shouldn't be a lot of SIGKILLing going on in a normal working
system. Hopefully this won't trigger until the problem event. If it
does, you might have to find another way.
But if it does false trigger, you should also try to understand who is
killing -9 whom -- it might be an important clue. Suppose a particular
daemon has a habit of creating a child, letting it run for a while, then
killing it. Suppose its memory of the child PID is corrupted somehow.
e.g. what if it does:
do_something &
childpid=$!
...
ps -ef | grep $childpid | awk '{ print "kill -9 " $2 }' | sh
This is horribly bad code, do not use it as an example of anything
besides trouble...
So this "works fine" for a while, when $childpid is 12345. But one day
it runs its child right when the system PID counter has rolled over,
gets PID 23. `ps -ef | grep 23` is likely to match several processes
other than the intended victim.
Anyway, if you trap on delivery of signal 9, you're likely to catch the
problem happening.
And if it happens without hitting that trigger, you learn that it
_isn't_ a process-to-process kill, it's actually coming directly out of
the kernel -- also very interesting news.
>Bela<
ke...@actual-systems.com
output from crash and scodb - they were both using the same dump file
(in the example of how it was setup was a different example.)
You are correct about the 'exiting' processes - they all seem to be the
processes that are the ones that are killed when this happens - so it
does look like the dump was triggered at the correct time.
I did a bit of testing on a machine here doing 'kill -9' to various
processes and the result was the 'exit2(0, 9)' as you predicted -
however when I setup the breakpoint to be 'bp kill+175' I can't seem to
get it to trigger no matter what processes I kill. I also tried to
trigger with lines above and below this but they either triggered at
every kill or not at all. Is there another line that we can setup as a
trigger?
On these servers the problem seems to happen when they are running
rsync's - so I'm going to grab the source for the version that we are
running and see if there is anything in there doing kill's similar to
your example.
Again thanks for your help.
Keith
Bela Lubkin
> I'm very confused by the difference in PID's showing for inetd from the
> output from crash and scodb - they were both using the same dump file
> (in the example of how it was setup was a different example.)
Hmmm. It's possible that inetd had forked at that time; but they really
look like they should be the same.
> You are correct about the 'exiting' processes - they all seem to be the
> processes that are the ones that are killed when this happens - so it
> does look like the dump was triggered at the correct time.
I'm sure it was triggered at the right time at a gross level, now we're
looking for finer precision.
> I did a bit of testing on a machine here doing 'kill -9' to various
> processes and the result was the 'exit2(0, 9)' as you predicted -
> however when I setup the breakpoint to be 'bp kill+175' I can't seem to
> get it to trigger no matter what processes I kill. I also tried to
> trigger with lines above and below this but they either triggered at
> every kill or not at all. Is there another line that we can setup as a
> trigger?
`kill+175' was the start of the disassembly I looked at:
scodb> u kill+175
kill+175 movl %eax,4(%edi)
kill+178 cmpl %eax,9
kill+17D jne kill+18C
kill+17F movb %al,-1(%ebp)
The correct breakpoint here is "bp kill+17F" -- immediately after it has
compared the signal number to 9 (and branched elsewhere if it wasn't 9).
> On these servers the problem seems to happen when they are running
> rsync's - so I'm going to grab the source for the version that we are
> running and see if there is anything in there doing kill's similar to
> your example.
Hmmm... what version is it? This appears in the OLDNEWS file in rsync
source:
" NEWS for rsync 2.5.5, aka Snowy River (2 Apr 2002)
" Protocol: 26 (unchanged)
" Changes since 2.5.4:
"
" BUG FIXES:
"
" * Fix situation where failure to fork (e.g. because out of process
" slots) would cause rsync to kill all processes owned by the
" current user. Yes, really! (Paul Haas, Martin Pool)
That seems like it could possibly have something to do with the
problem... ;-}
On the other hand, it looks like it would have killed even more
processes, but would have done so with SIGUSR1. So unless this is a
hacked version of rsync, it probably isn't precisely that problem.
>Bela<
ke...@actual-systems.com
I'm still having problems with the kill+17F line. If I set 'bp
kill+17F' I can kill -9 any process I want and it doesn't trigger.
I've put the output from 'u kill' below so you can make sure that it is
the same version you are running on - but from looking at the list of
lines 175-17F from your example before It looks to be the same.
scodb:1> u kill
kill pushl %ebp
kill+1 movl %eax,pid_stamp
kill+6 movl %ebp,%esp
kill+8 subb %esp,1C
kill+B pushl %edi
kill+C movl -14(%ebp),%eax
kill+F pushl %esi
kill+10 movl %edi,u.u_ap
kill+16 pushl %ebx
kill+17 movl %eax,4(%edi)
kill+1A testl %eax,%eax
kill+1C jl kill+25
kill+1E cmpl %eax,20
kill+23 jl kill+44
kill+25 popl %ebx
kill+26 popl %esi
kill+27 movb u.u_error,16
kill+2E popl %edi
kill+2F movl %esp,%ebp
kill+31 popl %ebp
kill+32 ret
kill+33 nop
kill+34 movb u.u_error,3
kill+3B popl %ebx
kill+3C popl %esi
kill+3D popl %edi
kill+3E movl %esp,%ebp
kill+40 popl %ebp
kill+41 ret
kill+42 nop
kill+43 nop
kill+44 movl %eax,u.u_renv
kill+49 testl %eax,3000000
kill+4E jne kill+61
kill+50 movl %eax,4(%edi)
kill+53 cmpl %eax,14
kill+58 jne kill+61
kill+5A movl 4(%edi),16
kill+61 movw %si,(%edi)
kill+64 testw %si,%si
kill+67 jle kill+70
kill+69 movl %ebx,&proc[1]
kill+6E jmp kill+75
kill+70 movl %ebx,&proc[2]
kill+75 movl %eax,u.u_procp
kill+7A testw %si,%si
kill+7D movl -C(%ebp),%eax
kill+80 jne kill+8B
kill+82 movw %ax,10(%eax)
kill+86 testw %ax,%ax
kill+89 je kill+34
kill+8B movb -1(%ebp),3
kill+8F movl %edx,v.ve_proc
kill+95 cmpl %ebx,%edx
kill+97 jnb kill+1EE
kill+9D movb %al,(%ebx)
kill+9F testb %al,%al
kill+A1 je kill+1DB
kill+A7 movw %ax,12(%ebx)
kill+AB testw %si,%si
kill+AE movw -6(%ebp),%ax
kill+B2 jle kill+BD
kill+B4 cmpw %ax,%si
kill+B7 jne kill+1DB
kill+BD testw %si,%si
kill+C0 jne kill+D3
kill+C2 movl %eax,-C(%ebp)
kill+C5 movw %ax,10(%eax)
kill+C9 cmpw 10(%ebx),%ax
kill+CD jne kill+1DB
kill+D3 cmpb %si,FFFFFFFF
kill+D7 jnl kill+EA
kill+D9 movsx %edx,%si
kill+DC negl %edx
kill+DE movsx %eax,10(%ebx)
kill+E2 cmpl %eax,%edx
kill+E4 jne kill+1DB
kill+EA movl %eax,50(%ebx)
kill+ED movl %ecx,-14(%ebp)
kill+F0 movl %eax,4C(%eax)
kill+F3 subl %eax,%ecx
kill+F5 jnle kill+1DB
kill+FB movl %eax,4(%edi)
kill+FE cmpl %eax,19
kill+103 jne kill+112
kill+105 movl %eax,-C(%ebp)
kill+108 movl %edx,C(%ebx)
kill+10B movl %ecx,C(%eax)
kill+10E cmpl %ecx,%edx
kill+110 je kill+18C
kill+112 movw %ax,A(%ebx)
kill+116 xorl %edx,%edx
kill+118 movw %dx,u.u_uid
kill+11F movw -4(%ebp),%ax
kill+123 testl %edx,%edx
kill+125 je kill+148
kill+127 xorl %eax,%eax
kill+129 movw %ax,8(%ebx)
kill+12D cmpl %edx,%eax
kill+12F je kill+148
kill+131 movl %eax,-4(%ebp)
kill+134 andl %eax,FFFF
kill+139 cmpl %edx,%eax
kill+13B movl %eax,0
kill+140 setnz %al
kill+143 jmp kill+14A
kill+145 nop
kill+146 nop
kill+147 nop
kill+148 xorl %eax,%eax
kill+14A testl %eax,%eax
kill+14C je kill+16D
kill+14E xorl %eax,%eax
kill+150 xorl %edx,%edx
kill+152 movw %ax,u.u_ruid
kill+158 movw %dx,8(%ebx)
kill+15C cmpl %eax,%edx
kill+15E je kill+16D
kill+160 movl %edx,-4(%ebp)
kill+163 andl %edx,FFFF
kill+169 cmpl %eax,%edx
kill+16B jne kill+17F
kill+16D cmpl %ebx,&proc[1]
kill+173 jne kill+18C
kill+175 movl %eax,4(%edi)
kill+178 cmpl %eax,9
kill+17D jne kill+18C
kill+17F int3
kill+180 incl %ebp
kill+181 decl %esp
kill+183 addl %esi,50(%ebp)
kill+186 movb -1(%ebp),1
kill+18A jmp kill+1D6
kill+18C movb -1(%ebp),0
kill+190 movl %eax,4(%edi)
kill+193 testl %eax,%eax
kill+195 je kill+1D6
kill+197 pushl %eax
kill+198 pushl %ebx
kill+199 call sigqueue_zalloc
kill+19E addb %esp,8
kill+1A1 movl %ecx,%eax
kill+1A3 testl %ecx,%ecx
kill+1A5 je kill+1C3
kill+1A7 movl %edx,4(%edi)
kill+1AA movl 4(%ecx),%edx
kill+1AD movl %edx,-C(%ebp)
kill+1B0 movw %dx,12(%edx)
kill+1B4 movw 10(%ecx),%dx
kill+1B8 movl %edx,-C(%ebp)
kill+1BB movw %dx,A(%edx)
kill+1BF movw 14(%ecx),%dx
kill+1C3 movsx %eax,-6(%ebp)
kill+1C7 pushl %eax
kill+1C8 pushl %ecx
kill+1C9 movl %ecx,4(%edi)
kill+1CC pushl %ecx
kill+1CD pushl %ebx
kill+1CE call psignalinfo
kill+1D3 addb %esp,10
kill+1D6 testw %si,%si
kill+1D9 jnle kill+1EE
kill+1DB movl %eax,v.ve_proc
kill+1E0 addl %ebx,158
kill+1E6 cmpl %ebx,%eax
kill+1E8 jb kill+9D
kill+1EE movb %al,-1(%ebp)
kill+1F1 testb %al,%al
kill+1F3 je kill+3B
kill+1F9 popl %ebx
kill+1FA popl %esi
kill+1FB popl %edi
kill+1FC movb u.u_error,%al
kill+201 movl %esp,%ebp
kill+203 popl %ebp
kill+204 ret
kill+205 nop
kill+206 nop
kill+207 nop
kill+208 nop
kill+209 nop
kill+20A nop
kill+20B nop
kill+20C nop
kill+20D nop
kill+20E nop
kill+20F nop
Unfortunately (or maybe fortunately,) the version of rsync we are using
is newer than 2.5.5, and it was built from the directly from the source
on the rsync site with no hacks.
back:/ # rsync --version
rsync version 2.6.3pre1 protocol version 28
Copyright (C) 1996-2004 by Andrew Tridgell and others
<http://rsync.samba.org/>
Capabilities: 32-bit files, socketpairs, hard links, symlinks,
batchfiles,
inplace, no IPv6, 32-bit system inums, 64-bit internal
inums
Thanks again.
Keith
P.S. I've also replied to your mail, but it was bounced back to me - so
if you still havn't received it in a few hours let me know.
Bela Lubkin
> I'm still having problems with the kill+17F line. If I set 'bp
> kill+17F' I can kill -9 any process I want and it doesn't trigger.
>
> I've put the output from 'u kill' below so you can make sure that it is
> the same version you are running on - but from looking at the list of
> lines 175-17F from your example before It looks to be the same.
>
> scodb:1> u kill
> kill pushl %ebp
> kill+1 movl %eax,pid_stamp
> kill+6 movl %ebp,%esp
...
> kill+175 movl %eax,4(%edi)
> kill+178 cmpl %eax,9
> kill+17D jne kill+18C
> kill+17F int3
Yeah, that looks like the right code. (Though it seems a little odd
that scodb disassembled the breakpoint as "int3". That _is_ the
instruction it uses to mark a breakpoint, but it normally hides
breakpoints in a disassembly.)
I need to try it live on my own system, which I am not near at the
moment. I'll try later today.
[later] Yeah, that doesn't work. So do this:
scodb:1> u kill+10
kill+10 movl %edi,u.u_ap
kill+16 pushl %ebx
kill+17 movl %eax,4(%edi)
kill+1A testl %eax,%eax [hit q]
scodb> bp mod kill+1a
[1] quitif %eax != 9 [hit ESC]
:q
This makes a breakpoint that is hit _every_ time the kill() system call
is invoked. "quitif %eax != 9" causes scodb to quit (i.e. return the
system to normal operation) if it wasn't a kill -9. This kind of
breakpoint sometimes has side effects; you should only do this on one of
your test systems.
When this breakpoint hits, you will be in the context of the process
issuing the kill -9, not the process being killed. This is OK. We know
that whatever process is doing it is killing a whole bunch of other
processes. You can see the PID to which the kill is being sent by
doing:
scodb> d %edi
If it's 0 or negative, it's a single call that kills multiple processes
(see kill(S) or kill(C)). Investigate that immediately.
If it's positive, it's a single-process kill. Quit and see if the
debugger pops up again immediately. Look at several of the PIDs,
quitting after each one. Then start investigating. If you're looking
at the problem event, you know it's going to issue 10-20 kills. None of
the processes it's killing will disappear while you quit the debugger
and cycle to the next kill -- the calling process is going to be able to
fire off a bunch of kill() system calls before any of the receiving
processes actually receive the signals and start dying. Remember the
kernel isn't running when you're sitting at the scodb prompt...
You might get some false hits (processes doing kill -9's that aren't
part of the problem). You should be able to recognize the real problem
case pretty easily.
Once you know what process is doing it, the rest will probably fall into
place pretty quickly...
>Bela<
ke...@actual-systems.com
Keith
ke...@actual-systems.com
I've had this machine now tripinto debug mode, but unfortunately I am
remote from there so I had to have someone else run the commands you
wanted and give me the response.
>From what I can see it was triggered by an rsync - but I don't know how
to read the output from 'd %edi' to tell you if it was 0 or negative or
positive.
----------------------
BreakPoint: PID 000005BF /usr/bin/rcmd asa /usr/local/bin/rsync
--server--
F01D1676 kill+1A : kill+1a
---------------------
Then the output from 'd %edi' was as follows
u+1148 000005BF 00000009 00001000 000000000
Then when scodb was exited it popped straight back in and the output
for the first part was the same (what triggered it,) but the output
from 'd %edi' was as follows.
u+1148 0000052A 00000009 00001000 000000000
u+1148 00000536 00000009 00001000 000000000
u+1148 00000542 00000009 00001000 000000000
u+1148 00000536 00000009 00001000 000000000
u+1148 00000552 00000009 00001000 000000000
u+1148 000005BF 00000009 00001000 000000000
Does this mean anything to you?
Thanks,
Keith
Bela Lubkin
Yes, plenty. Each of those "u+1148" lines represents another trip into
kill(). The first time, we can see that the caller is PID 0x5BF (1471),
running `rcmd asa rsync`. %edi points to the arguments, here "5BF" and
"9": it's sending signal 9 to PID 0x5BF -- which is itself. That seems
odd... is there any chance that your guy at the keyboard mistyped that
one?
The next 6 calls show signal 9 being sent to PIDs 1322, 1334, 1346,
1334, 1362, 1471 (the last one again being the process doing the kills).
If we assume that the first one was a typo, we can say that PID 1471
(rcmd) is killing off a series of other PIDs, ending with itself.
I experimented with `rcmd`, learning that it does finish by killing
itself with signal 9. I didn't figure out how to trigger it into a
killing spree, but there obviously is a way. Since it's killing
unrelated processes, it's definitely a bug of some sort.
I'm curious about what's causing `rcmd` to malfunction that way, but
the main thing is to fix the problem on your system. To do that, let's
just eliminate the use of `rcmd`.
Find the script that's doing `rcmd asa rsync`; change it to use a
different remote command executor. For instance, change it to use `ssh
asa rsync`. If `rcmd` is adequate then you obviously don't feel the
need for the level of security offered by `ssh`, but the point here is
just to use a different tool that doesn't have this bug. You can
configure `ssh` in ways that allow passwordless remote execution. (It's
possible that `rsync` itself is directly invoking `rcmd`; in which case
you can use `rsync --rsh=/usr/bin/ssh` or something like that.)
>Bela<
ke...@actual-systems.com
I started using rcmd as using ssh seemed to cause this problem to
happen more often! (and as both machines are on the same LAN I wasn't
worried about security.) As you said, it just a case of changing the
rsync command that is used.
Where you think there is a typo that is probably the case as the
details were transferred from screen to paper to email.
Thanks again,
Keith
Brian K. White
You can also skip shh (a cpu eater) and rcmd both and install rsync as a
permanent service.
if you used to have a command like this running on boxaa:
rsync -avz --delete /u/asa boxbb:/u
which replicates /u/asa from boxaa to boxbb
You can do the following on either boxaa or boxbb, depending on which box
you want to initiate the transfer.
Doesn't matter which one is receiving or sending, it matters which one you
want to initiate the job. Who's cron job calls the shots?
On the box that is NOT initiating the job, lets say this is boxaa, do the
following:
# create 2 config files in /etc
vi /etc/rsyncd.conf:
-----top----
uid = root
gid = sys
secrets file = /etc/rsyncd.secrets
read only = false
[root]
path = /
auth users = root
----end----
vi /etc/rsyncd.secrets:
----top----
root:somepassword
----end----
# create an rc start script
vi /etc/init.d/rsync:
----top----
#!/bin/ksh
case $1 in
start) /usr/local/bin/rsync --daemon ;;
esac
----end----
# link it into rc2.d so it runs at boot
ln -s /etc/init.d/rsync /etc/rc2.d/S99rsync
# use it manually to start it up now without rebooting
/etc/init.d/rsync start
now, on boxbb, do the following instead of your previous rsync command:
RSYNC_PASSWD=somepasswd rsync -avz /u/asa otherbox::root/u
"somepasswd" has nothing to do with the real root password on either box,
and probably should intentionally not be the same as the root password,
because it's written in plain text in a file on both machines. somepasswd
just needs to be the same between the rsync command on boxbb and the
rsyncd.secrets file on boxaa
This is a really insecure set up because it allows the client root privs to
write anywhere it wants on boxaa, and the password allowing this to happen
is written in plain text in the script that has the rsync command. Obviously
just for starters, rsyncd.secrets on boxaa, and the script on boxbb should
be chmod a-r so that only root can read them. te cron job running the script
on boxbb will be root and read it fine. The rc script on boxaa is also root.
But this setup is a direct analog to the original rcmd-as-root or
ssh-as-root you were using.
A safer set up is this, create a module in rsync that only sees a certain
directory (and all subdirectories), and write to it instead of /:
BOXAA:
/etc/rsyncd.conf:
-----top----
uid = root
gid = sys
secrets file = /etc/rsyncd.secrets
read only = false
list = false
[asa]
path = /u/asa
auth users = asa
----end----
/etc/rsyncd.secrets:
----top----
asa:asapassword
----end----
BOXBB:
RSYNC_PASSWD=asapasswd rsync -avz --delete /u/asa/* asa@otherbox::asa
That says to connect to otherbox as user "asa" , to module asa, and put
things into the current directory (which we know is already /u/asa)
If /u/asa/* expands to too large of a list, then you can make a less safe
arrangement that allows writing to all of /u , so that you can specify
/u/asa as the source.
like this:
BOXAA:
/etc/rsyncd.conf:
-----top----
uid = root
gid = sys
secrets file = /etc/rsyncd.secrets
read only = false
list = false
[u]
path = /u
auth users = u
----end----
/etc/rsyncd.secrets:
----top----
u:upassword
----end----
BOXBB:
RSYNC_PASSWD=upasswd rsync -avz --delete /u/asa u@otherbox::u
Note: you do not have to create a unix user "asa" or "u" anywhere.
You did not have to use the "user@" part of the syntax in my first example
because I'm assuming the script is running as root in a cron job.
Although the rsync user is not related to the unix user anywhere, if you
don't specify a user to use with the "::" syntax, then it defaults to using
your current unix login name.
In all cases shown here, the rsync daemon on boxaa is still being started by
root and the config file tells it to continue operating as root, which means
that it has permission to create, destroy, overwrite any files anywhere, and
it has permission to set any ownerships and permissions. So any files that
get sent from boxbb get their permissions and ownership duplicated as well
as their contents.
But the module "asa" only "sees" /u/asa, so a client using that module
cannot write anywhere outside of /u/asa. And in my examples I also made it
so that only the rsync user "asa" has permission to use module asa. Not even
root can get around that since rsync is only concerned with the contents of
rsyncd.secrets and rsyncd.conf, and no "root" exists in my later examples,
and even if it did, root is not listed on the "auth users" for the asa
module. You could make a unix-like arrangement where root can do everything
and asa can do a subset by having both root and asa modules in rsyncd.conf .
BOXAA:
/etc/rsyncd.conf:
-----top----
uid = root
gid = sys
secrets file = /etc/rsyncd.secrets
read only = false
list = false
[root]
path = /
auth users = root
[asa]
path = /u/asa
auth users = asa
----end----
/etc/rsyncd.secrets:
----top----
root:somepassword
asa:asapassword
----end----
A typical use of this type of arrangement might be:
BOXBB:
RSYNC_PASSWD=asapasswd rsync -avz --delete /u/asa/* asa@otherbox::asa
RSYNC_PASSWD=somepasswd rsync -avz /etc/default/asa
otherbox::root/etc/default
your cron job uses the asa module, and yet you can still use the root module
manually to do anything you want
Actually, for manual, transient, rootly tasks, you can still use your
existing familiar rcmd-based commands. None of the above affects the rcmd or
ssh based functionality in the slightest.
So you can try setting it up and testing it while any cron jobs continue to
do what they're already doing.
For the record, I use rsync a lot and for quite a while, both in twice daily
mirror scripts between numerous hosts, and for random manual admin tasks as
a more convenient alternative to ftp/sftp.
And I have not had any problem with killing sprees like that, so you might
want to try using the version of rsync I'm using. The actual binaries are
here:
http://www.aljex.com/bkw/sco/#rsync
rsync.tar.bz2 is the latest version, there is also the next previous version
I used. That previous version I used a _lot_ all over the place. It's
received a lot more testing than the new one but I have been using the new
one heavily too, but not on many hosts yet, since the creation date shown on
the page.
The previous version is the full source after compile so you have to
manually copy out the binaries out, or run "make install" if you have
"make".
The unversioned tar is always the latest verion (that I've built anyway)
with just the binries and man pages already in their installed absolute
paths, just untar it and it already works as a client. It's compiled-in
defaults match it's own install path and default to using rcmd, so if you
put the same version on two boxes, you don't have to specify --rsync-path=
or --rsh=
just "rsync -avz --delete source target"
If "rsync --version" crashes, install oss646c.
Then try the recipe above.
Replacing the rsync binary is the only thing that might affect your existing
cron jobs depending on how the cron job is written.
The cron job might or might not start using the new binary, and only if that
happens, the options used might or might not need simple adjusting due to
differences in the compiled in defaults for --rsync-path and --rsh.
Doing it this way:
* doesn't require user equivalency. (/.rhosts, /etc/hosts, /etc/hosts.equiv)
all you need to set up is rsyncd.conf & .secrets
* the tcp traffic is handled directly by rsync itself, meaning:
- no ssh encryption/decryption eating cpu on both boxes
- you can use or not-use compression to better suit the job at hand.
(use -av on a dir full of jpg/tiff/png, use -avz on a database)
- if rcmd or ssh is doing anything bad like dropping connections,
hanging/stalling, your killing spree issue, well they are not used any more.
- can play with rsync options like --blocking-io to see if it helps
connection reliability problems, that might not have as much effect if the
traffic is really going over some other client & server like rcmd or ssh if
they don't have equivalent options themselves.
Brian K. White -- br...@aljex.com -- http://www.aljex.com/bkw/
+++++[>+++[>+++++>+++++++<<-]<-]>>+.>.+++++.+++++++.-.[>+<---]>++.
filePro BBx Linux SCO FreeBSD #callahans Satriani Filk!
Bela Lubkin
> ----- Original Message -----
> From: <ke...@actual-systems.com>
> > I started using rcmd as using ssh seemed to cause this problem to
> > happen more often! (and as both machines are on the same LAN I wasn't
> > worried about security.) As you said, it just a case of changing the
> > rsync command that is used.
>
> You can also skip shh (a cpu eater) and rcmd both and install rsync as a
> permanent service.
>
> if you used to have a command like this running on boxaa:
>
> rsync -avz --delete /u/asa boxbb:/u
>
> which replicates /u/asa from boxaa to boxbb
>
[...]
> now, on boxbb, do the following instead of your previous rsync command:
> RSYNC_PASSWD=somepasswd rsync -avz /u/asa otherbox::root/u
[heavily chopped]
I just wanted to comment that this does sound like a good idea. My
previous quick glance at the `rsync` man page left me with the
impression that it always used `rsh` unless overridden to e.g. `ssh`.
With Brian's examples in mind, I went back and read about the "::" and
"rsync://" naming schemes and the fact that this doesn't involve a
separate remote execution protocol. Switching from remote execution to
remote copying is a strength reduction -- it slightly improves the
security of the process. Plus, of course, `rsync` is 100% designed to
talk to itself; eliminating the oblique complexities of a remote command
executor should be helpful. And finally it _will_ avoid the killing
spree problem since `rcmd` (rsh) will be out of the picture.
>Bela<
ke...@actual-systems.com
It also sounds like a good idea to me - however it may take a little
time to setup etc, so for the moment I have just changed to use ssh
rather than rcmd. This method is something that may be very helpful as
I am relying more and more on rsync and a method to do so without as
much overhead is definately a good thing.
Keith
Jean-Pierre Radley
|
| You can do the following on either boxaa or boxbb, depending on which box
| you want to initiate the transfer.
| Doesn't matter which one is receiving or sending, it matters which one you
| want to initiate the job. Who's cron job calls the shots?
| On the box that is NOT initiating the job, lets say this is boxaa, do the
| following:
|
| # create 2 config files in /etc
The rsync configure script, like many others out there, defaults to
using /usr/local as a Prefix to /bin, thus installing its binaries in
/usr/local/bin.
This is "a good thing"(TM); I, for one, always see to it that /usr/local
is NOT in the root filesystem, thus saving lots of grief when updating
the Operating System.
Inexplicably, the configure script defaults to having rsyncd look for
rsyncd.conf in /etc (if it's not in the current working directory).
It's more logical to me that this file should reside in /usr/local/etc.
So I put it to you that you will increase happiness (and, mirabile
dictu, even decrease entropy?) if you would run rsync's configure script
with the option '--with-rsyncd-conf=/usr/local/etc'.
:-)
--
JP
Brian K. White
----- Original Message -----
From: "Jean-Pierre Radley" <j...@jpr.com>
Newsgroups: comp.unix.sco.misc
To: <dis...@jpr.com>
My opinion is it's philosophical issue with no best answer.
For example, your points are perfectly logical and valid.
However I feel there is more than one equally logical organization, just
with different reasons behind the logic.
For my part, I prefer not to have config files scattered in multiple places
around the filesystem any more than necessary.
Given that I don't think any particular location is more right than any
other in this case, I feel the least surprise is produced by not adding any
uneccessary ./configure options.
So the resulting binary is more likely to agree with more documents and
examples "out there".
It would seem that this argument should apply doubly to the binary, which I
do have to force into /usr/local. More, I also have to hack the
default --rsync-path
because when using rcmd and ssh, the path seen by the remote exec agent does
not include /usr/local/bin, so it can't run "rsync", and must be
"/usr/local/bin/rsync"
If I didn't touch anything rsync would install in /usr/bin or /bin
and --rsync-path would be "rsync", and that would work.
What can I say, I detest foreign substances in [/usr]/[s]bin but not in /etc
Maybe there is a logical reason for this actually. There may be both OS and
3rd party versions of the same program, you don't want your add-on binary to
be in danger of being replaced or removed by OS updates or rolling back OS
updates, yet you only want one config file to worry about.
(wget, curl, ssh, several other things that you used to always have to add
in /usr/local, are now shipped with the OS or in vendor supplied add-ons
like gnutools and gwxlibs)
Bill Vermillion
And followed to the letter by many of the free Unix like
implementations - such as FreeBSD.
>However I feel there is more than one equally logical
>organization, just with different reasons behind the logic. For
>my part, I prefer not to have config files scattered in multiple
>places around the filesystem any more than necessary.
In the xBSD land all programs added after the OS are part in
/usr/local which has a complete hierarch just as the regular OS
with directories under /usr/local being the typical etc, bin,
sbin, include, libdata, libexec, man, share, and var.
One nice thing about this is you can completely replace the OS
and never touch your locally installed files. So often in systems
with add-on programs sharing the same place with the OS it can
be a chinese-fire-drill [to use an old-phrase] trying to make sure
you keep what you want, and don't remove things that require a
re-install.
To my way of thinking, keeping add-ons and their ancilliary files
under /usr/local is the only logical choice - but then again I'm
not Spock or known for being completely logical.
>Given that I don't think any particular location is more right
>than any other in this case, I feel the least surprise is
>produced by not adding any uneccessary ./configure options.
Agree on that point.
...
>What can I say, I detest foreign substances in [/usr]/[s]bin but
>not in /etc
And having weaned myself from the Sys5 POV and back to my earliest
Unix learning, I've grown to really dislike systems that REQUIRE
/usr to have OS REQUIRED files. I like having /usr on a separate
partition. Which goes with the above comment about reinstalling,
where I can remake the / filesystem, and re-install, without ever
touching any add-on application.
Part of this POV is I'm lazy and hate to re-do something I've
already done.
>Maybe there is a logical reason for this actually. There may be
>both OS and 3rd party versions of the same program, you don't
>want your add-on binary to be in danger of being replaced or
>removed by OS updates or rolling back OS updates, yet you only
>want one config file to worry about. (wget, curl, ssh, several
>other things that you used to always have to add in /usr/local,
>are now shipped with the OS or in vendor supplied add-ons like
>gnutools and gwxlibs)
My points persactly :-)
Bill
--
Bill Vermillion - bv @ wjv . com
Brian K. White
> /usr/local which has a complete hierarch just as the regular OS
> with directories under /usr/local being the typical etc, bin,
> sbin, include, libdata, libexec, man, share, and var.
>
> One nice thing about this is you can completely replace the OS
> and never touch your locally installed files. So often in systems
> with add-on programs sharing the same place with the OS it can
> be a chinese-fire-drill [to use an old-phrase] trying to make sure
> you keep what you want, and don't remove things that require a
> re-install.
>
> To my way of thinking, keeping add-ons and their ancilliary files
> under /usr/local is the only logical choice - but then again I'm
> not Spock or known for being completely logical.
/etc gets hand customized in freebsd just like any other.
The biggest difference is freebsd has mergemaster.
Bill Vermillion
>> /usr/local which has a complete hierarch just as the regular OS
>> with directories under /usr/local being the typical etc, bin,
>> sbin, include, libdata, libexec, man, share, and var.
>>
>> One nice thing about this is you can completely replace the OS
>> and never touch your locally installed files. So often in systems
>> with add-on programs sharing the same place with the OS it can
>> be a chinese-fire-drill [to use an old-phrase] trying to make sure
>> you keep what you want, and don't remove things that require a
>> re-install.
>> To my way of thinking, keeping add-ons and their ancilliary files
>> under /usr/local is the only logical choice - but then again I'm
>> not Spock or known for being completely logical.
>/etc gets hand customized in freebsd just like any other.
>The biggest difference is freebsd has mergemaster.
The ports tree uses variables PREFIX and LOCALBASE that is prepended
to the places that the application files go - eg etc, libexec,
lib. So they wind up going to /usr/local/etc and others.
You'll see this heavily used in /usr/ports/Mk
See 'man ports' and also see 'man make.conf'
It's not 'hand customized' but it is the default method
unless you override it. You can change these in /etc/make.conf
mergemaster has to do with taking the configuration files you
have in /etc and giving you the ability to merge new changes with
your old files so you don't have to re-enter a lot of data if you
did an upgrade. It really doesn't have a lot to do with
putting all the add-ons into /usr/local.
Bill
Brian K. White
----- Original Message -----
From: "Bill Vermillion" <b...@wjv.com>
Newsgroups: comp.unix.sco.misc
To: <dis...@jpr.com>
Sent: Monday, August 29, 2005 7:45 PM
Subject: Re: rsync configuration [Was: Re: Dying processes (inetd, cron,
syslogd, sshd)]
I know what mergemaster does and the sense behind /usr/local and all that.
My point was not that mergemaster had anything to do with /usr/local. It was
that, even on freebsd, there are things you customize per system in the main
/etc
which is a part of the base os, which means that at every update you face
the same 2 possible choices as any other OS:
1) lose* your changes during the update as the update overwrites them.
*OK it does usually back them up first such as the way sco's custom renames
things by appending pound signs and redhat's rpm appends .rpmorig
2) don't lose your changes, but break the apps whose configs syntax changed
but you didn't get a new matching default config during the update.
Except, the difference from any other os is, freebsd has mergemaster which
helps with the task of updating configs in-place _instead_ of having to
choose 1 or 2 above.
I forgot another thing too, some of the things you customize per box even in
/etc and /boot, the file you customize is actually one you add yourself or
that the update process specifically knows to leave alone, that a similar OS
supplied & updated file reads-in. So many things you edit in /etc and /boot
are actually safe in a update. This is just even more reason to feel that
/etc is not a bad place to put config files, including 3rd party add-ons.
ke...@actual-systems.com
I sent you an email a couple of days ago - I was wondering if you
received it or if you want me to resend it.
It was to discuss what you had proposed in a previous email to me on
this subject.
Thanks,
Keith
Bela Lubkin
Multiple systems running SCO OpenServer Release 5.0.6 + rs506a were
having an intermittent problem. The symptom was that most of the
processes on the system would suddenly die without warning.
After some public discussion, Keith and I agreed that the most likely
way to solve the problem was for me to access the systems directly. We
arranged to do this under the auspices of the company I am now working
for. We also agreed that when the problem was solved, I would post the
story so that others might avoid the problem in the future.
So...
In brief, the problem was being caused by an old, buggy version of the
pseudo-random number generator, `prngd`. The eventual solution was
simply to upgrade the machines to a more recent version of `prngd`.
Now for some details. I will describe the discovery process, the actual
cause of the problem, the important details of the symptoms, and the
solution.
We arranged for me to have console access to two live backup machines
which were experiencing the problem. Keith configured his firewall to
allow me to `ssh` in to a master machine. From that system I could `cu`
to COM1 of each of the test machines. Keith configured scodb into their
kernels, and booted them with COM1 as their consoles. Now I had remote
live kernel debugger access.
I installed GNU `screen` on the master machine so that I could connect
and disconnect at will without losing console output from the test
machines.
I established kernel breakpoints to enter the debugger whenever certain
system processes (like `cron`, `inetd` etc.) died. Then it was a matter
of waiting for an "event".
In the first event, the process was being killed by signal 9 (SIGKILL).
Many other processes had pending SIGKILL signals. The timing made me
think that the problem was being caused by a single multi-process kill
rather than a series of individual kills. The kill(S) system call
accepts special arguments of "-PGID" (negative process group ID) to kill
all processes in a particular group; or -1 to kill all processes in the
system. Unfortunately, the process responsible for the call had
finished and gone on to other business; I didn't catch it in the act.
This might have been different on a single-CPU system, but Keith's test
systems were MP. While I was examining the dying process on one CPU
(probably in the few milliseconds after the debugger prompt came up),
the culprit was finishing his dirty work on the other CPU.
Now that I thought it was a multi-process SIGKILL, I put a breakpoint
into kill(S) itself, essentially:
if (signal is SIGKILL and process to kill is < 0 [multiple procs])
breakpoint
This eventually triggered as well. Previous evidence had lead me to
suspect the remote-command program, `rcmd` (`rsh` on other *ix systems).
`rcmd` was already implicated because, according to Keith, the events
always happened while large files were being copied across the network.
Also, I had run `truss` on it and observed that it used SIGKILL to kill
off its own child process. I suspected a race condition where `rcmd`
was getting confused about its child process's ID and mistakenly doing
`kill(-1, 9)'.
So I was fairly surprised when the actual caller was `prngd`! Sure
enough, it was calling `kill(-1, 9)'. But why??
Keith's systems were running `prngd` version 0.9.6. I found the
development site for `prngd`,
http://www.aet.tu-cottbus.de/personen/jaenicke/postfix_tls/prngd.html
and its FTP repository,
ftp://ftp.aet.tu-cottbus.de/pub/postfix_tls/related/prngd/
Fortunately there was an "old" subdirectory with 40 previous versions of
the `prngd` source. This allowed me to see how this bug appeared and
later disappeared.
`prngd` is the pseudo-random number generator; on OpenServer, this is
primarily used by `ssh`. (Interestingly, Keith had earlier mentioned
that the problem seemed to get worse when they used `ssh` instead of
`rcmd` for their large file copies. Now that we've identified `prngd`
as the culprit, this makes sense...) Cryptographical protocols tend to
use random numbers to make it harder to crack the encryption. `prngd`
creates "pseudo" random numbers by running what it calls "entropy
gathering commands". On OSR5, these are commands like `ps -efl`,
`netstat -in`, `df`, and `tail -200 /var/adm/syslog`. The command list
is stored in /etc/prngd.conf.
Since `prngd` is portable to many operating systems, it has to deal with
all sorts of unusual conditions. One of these is this: some time in the
past, it had run into entropy gathering commands which would sometimes
hang. You can imagine that a command like `netstat -in`, which dives
into kernel networking structures, might have some obscure bugs. In
order to protect itself from possible hangs, `prngd` monitors the
entropy gathering commands and kills them off if they run for too long.
(Since this appears in the first public release of `prngd`, it looks
like the author anticipated the hanging problem without necessarily
having seen it.)
The code for this in `prngd` 0.9.6 looked sort of like this:
pid = (start the entropy gathering command, report its PID)
...
if (too much time has gone by: entropy gathering command seems hung)
if (pid != -1)
kill(pid, SIGKILL)
It also set up a signal handler to receive SIGCHILD, which notifies a
parent process when its subprocess dies. Some of the code in that
handler looked like this:
pid = -1 /* note no entropy gathering command currently running */
This lead to a race condition. The fatal sequence went like this:
pid = (start the entropy gathering command, report its PID)
...
if (too much time has gone by: entropy gathering command seems hung)
if (pid != -1)
/* at this moment, pid isn't -1 */
/* also at this moment, the entropy gathering command
finishes, so we enter the SIGCHILD handler */
...
pid = -1
...
/* back in the main code */
kill(pid, SIGKILL)
The main code intended to kill off the one entropy gathering process,
but it got tricked into sending SIGKILL to PID -1. Which means "kill
every process on the system". Whoops.
The timing window between `if (pid != -1)' and `kill(pid, SIGKILL)' is
small. You might think "that will never happen!". People who program
in multithreaded / multitasking environments soon learn that this sort
of "race condition" _always_ eventually shows up. The CPU is running
millions or billions of instructions per second and this window is only
a few instructions wide, but you can be sure it will eventually get hit.
If the consequences were less catastrophic, it might never be noticed.
For instance if the only consequence was that, once every few million
runs, an "entropy gathering process" would hang and never finish (but
`prngd` kept running), it might never have been fixed.
I started checking other versions of `prngd`. The first public release
of `prngd` was 0.1.0, on 2000-07-03, and it already had this bug. 0.9.6
was published on 2001-02-19. One short week later, 2001-02-26, version
0.9.8 was released -- including the fix for this bug. (The current
version is 0.9.29, released 2004-07-12...) The bug lasted about 7 1/2
months.
So this was an already known bug in a system daemon. The tricky part
was tracking back from the symptoms to their cause.
Now, a word about the symptoms. Keith had reported that "most of the
running processes just seem to stop". Here's what was actually
happening. When `prngd` called `kill(-1, 9)', this sent SIGKILL to
every _eligible_ process. The kill(C) man page says:
" If the effective user ID of the sender is root, send the
" signal to all processes (except processes 0 and 1).
This is an imperfect description, because certain other processes are
also exempt. The signal is not sent to any kernel processes. On OSR5,
these include "sched", "vhand", "bdflush", "CPUn idle process",
"kmdaemon", "vddaemon", "strd", "htepi_daemon", "dtdaemon", and possibly
others.
Perhaps even more importantly, process 1 -- `init` -- is exempt. One of
init's jobs is to restart certain processes if they die. After an
"event" on one of Keith's systems, `init` restarted all the processes it
was responsible for: about a dozen `getty` processes (for the console
multiscreens), and the daemon starting daemon, `sdd`.
By the time a human got a look at the system, after an event which
theoretically killed "every" process on the system, there were about 20
processes running.
Now you too can recognize the symptoms of a "kill(-1, 9)" call...
The solution, of course, was very simple. Keith installed a newer
version of `prngd`, and the problem hasn't happened once in the last
month. It used to happen several times a week.
I recommend checking the `prngd` (sometimes called `in.prngd`) binaries
on all of your systems running any sort of *ix OS -- OpenServer,
UnixWare, Linux, Mac OS/X, BSD, whatever. If you find a version earlier
than 0.9.8, upgrade to a more recent version. (`prngd` version 0.9.10
came after 0.9.9.) All *ix systems are vulnerable to some amount of
trouble from this bug. Even if `prngd` is run under its own separate
user ID, it would still succeed in killing _itself_, thus shutting down
the pseudo random number service.
Given the age of the problematic versions, you won't find many of them
running around out there. But it's still worth checking.
Likewise, if you ever have "nearly all" processes on a system die,
consider whether it might have been a global kill. As we saw here, this
doesn't necessarily leave the process table completely blank.
=============================================================================
I am available for Unix problem solving and security consultation
through IS-Data, LLC, a Santa Cruz consultancy specializing in network
security. www.is-data.net.
>Bela<