kern/72979: unkillable process(es) stuck in `STOP' state
Mikhail Teterin
>Number: 72979
>Category: kern
>Synopsis: unkillable process(es) stuck in `STOP' state
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Thu Oct 21 21:00:44 GMT 2004
>Closed-Date:
>Last-Modified:
>Originator: Mikhail Teterin
>Release: FreeBSD 6.0-CURRENT i386
>Organization:
Virtual Estates, Inc.
>Environment:
System: FreeBSD mi 6.0-CURRENT FreeBSD 6.0-CURRENT #1: Wed Oct 20 12:08:24 EDT 2004 mteterin@mi:/meow/obj/misha/src/sys/Gigabyte i386
debug.mpsafenet and safevm both set to 0
>Description:
Somehow, it is possible to get a process into a state, where it can not be
killed. The state is reported as `T' by ps and as `STOP' by top. There seems
to be a zomie-child of the process, when this happens:
UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND
1042 1096 1 57 8 0 68044 50228 - T ?? 0:27,88 kontact
1042 4903 1096 0 -84 0 0 0 - Z ?? 0:03,07 <defunct>
But neither -CONT nor -KILL can get rid of the process. Attempts to ktrace
it result in empty ktrace.out.
>How-To-Repeat:
It is unclear, what exactly is causing this. So far, I have seen it
twice -- both time with KDE-programs (KMail and Kontact), which attach
a debugger to themselves, when crashing (a frequent occurence).
Whatever mistake KDE may be making in their error-handling, it does
not explain an unkillable process.
May be, this has something to do with the threading library (libpthread)?
>Fix:
Don't know...
>Release-Note:
>Audit-Trail:
>Unformatted:
_______________________________________________
freebs...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs...@freebsd.org"
Stefan Walter
From: Stefan Walter <s...@gegenunendlich.de>
To: freebsd-gn...@FreeBSD.org, m...@aldan.algebra.com
Cc:
Subject: Re: kern/72979: unkillable process(es) stuck in `STOP' state
Date: Mon, 25 Oct 2004 14:46:20 +0200
I have just seen the same thing on a RELENG_5 system as of Oct 24th. In
this case it was Umbrello, which comes with devel/kdesdk3; Umbrello
crashed (application bug, AFAICS), and when I closed the window that
showed the backtrace, the process remained in state 'STOP', according to
top(1).
I don't know if the application's backtrace can be of any help here, but
just in case:
[Switching to LWP 100139]
0x296474d7 in wait4 () from /lib/libc.so.5
#0 0x296474d7 in wait4 () from /lib/libc.so.5
#1 0x296388df in waitpid () from /lib/libc.so.5
#2 0x294b4089 in waitpid () from /usr/lib/libpthread.so.1
#3 0x28b2c7ed in KCrash::defaultCrashHandler ()
from /usr/local/lib/libkdecore.so.6
#4 0x294ba6ef in sigaction () from /usr/lib/libpthread.so.1
#5 0xbfbfff94 in ?? ()
#6 0x0000000b in ?? ()
#7 0xbfbfdf40 in ?? ()
#8 0xbfbfdc80 in ?? ()
#9 0x00000000 in ?? ()
#10 0x294ba41c in sigaction () from /usr/lib/libpthread.so.1
#11 0x0828fd78 in JavaCodeClassField::~JavaCodeClassField ()
#12 0x28f5fd0b in QObject::~QObject () from /usr/X11R6/lib/libqt-mt.so.3
#13 0x0818698d in UMLObject::~UMLObject ()
#14 0x0818a8cd in UMLRole::~UMLRole ()
#15 0x28f5fd0b in QObject::~QObject () from /usr/X11R6/lib/libqt-mt.so.3
#16 0x0818698d in UMLObject::~UMLObject ()
#17 0x080fa535 in UMLAssociation::~UMLAssociation ()
#18 0x28f606a5 in QObject::event () from /usr/X11R6/lib/libqt-mt.so.3
#19 0x28f0a29d in QApplication::internalNotify ()
from /usr/X11R6/lib/libqt-mt.so.3
#20 0x28f096ca in QApplication::notify () from /usr/X11R6/lib/libqt-mt.so.3
#21 0x28aa1945 in KApplication::notify () from /usr/local/lib/libkdecore.so.6
#22 0x28f0afd5 in QApplication::sendPostedEvents ()
from /usr/X11R6/lib/libqt-mt.so.3
#23 0x28f0adec in QApplication::sendPostedEvents ()
from /usr/X11R6/lib/libqt-mt.so.3
#24 0x28eba528 in QEventLoop::processEvents ()
from /usr/X11R6/lib/libqt-mt.so.3
#25 0x28f1b44f in QEventLoop::enterLoop () from /usr/X11R6/lib/libqt-mt.so.3
#26 0x28f1b3a8 in QEventLoop::exec () from /usr/X11R6/lib/libqt-mt.so.3
#27 0x28f0a3f4 in QApplication::exec () from /usr/X11R6/lib/libqt-mt.so.3
#28 0x08144ee3 in main ()
Mikhail Teterin
From: Mikhail Teterin <mi...@aldan.algebra.com>
To: dav...@freebsd.org
Cc: Michael Nottebrock <michaeln...@gmx.net>, r...@freebsd.org,
dav...@t2t2.com, dav...@viatech.com.cn,
freebsd-gn...@freebsd.org
Subject: Re: kern/72979: unkillable process(es) stuck in `STOP' state
Date: Mon, 25 Oct 2004 22:38:56 -0400
David, would you take a look at this? Some apps sometimes end up unkillable:
http://www.freebsd.org/cgi/query-pr.cgi?pr=72979
If you provide an encouraging answer quick enough, we may even be able to
persuade re@ to hold the release long enough for you to fix it. Thanks!
-mi
Ken Smith
From: Ken Smith <kens...@cse.Buffalo.EDU>
To: Mikhail Teterin <mi...@aldan.algebra.com>
Cc: dav...@freebsd.org,
Michael Nottebrock <michaeln...@gmx.net>, r...@freebsd.org,
dav...@t2t2.com, dav...@viatech.com.cn,
freebsd-gn...@freebsd.org
Subject: Re: kern/72979: unkillable process(es) stuck in `STOP' state
Date: Mon, 25 Oct 2004 23:21:29 -0400
--lrZ03NoBR/3+SXJZ
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Mon, Oct 25, 2004 at 10:38:56PM -0400, Mikhail Teterin wrote:
> David, would you take a look at this? Some apps sometimes end up unkillab=
le:
>=20
> http://www.freebsd.org/cgi/query-pr.cgi?pr=3D72979
>=20
> If you provide an encouraging answer quick enough, we may even be able to=
=20
> persuade re@ to hold the release long enough for you to fix it. Thanks!
>=20
> -mi
David,
Scott thought you might have had some work done on this already, and
perhaps even some testable code. If that's true can you let us know
please? Scott needs to head to Europe for the conference but I can
try to push this stuff along while he's occupied with the conference.
If you do have some time invested in this can you let us know what
state it's in please? We'll take it from there to decide what to
do with it.
Thanks.
--=20
Ken Smith
- From there to here, from here to | kens...@cse.buffalo.edu
there, funny things are everywhere. |
- Theodore Geisel |
--lrZ03NoBR/3+SXJZ
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (SunOS)
iD8DBQFBfcK3/G14VSmup/YRAkicAKCDimmHXdtFX3/jaU5g5C+DBD4euwCfbKf/
xJe3pYk6xs9mPvTHAR1eDBc=
=1mOr
-----END PGP SIGNATURE-----
--lrZ03NoBR/3+SXJZ--
David Xu
From: David Xu <dav...@freebsd.org>
To: Mikhail Teterin <mi...@aldan.algebra.com>
Cc: Michael Nottebrock <michaeln...@gmx.net>, r...@freebsd.org,
dav...@t2t2.com, dav...@viatech.com.cn,
freebsd-gn...@freebsd.org
Subject: Re: kern/72979: unkillable process(es) stuck in `STOP' state
Date: Tue, 26 Oct 2004 19:11:54 +0800
Mikhail Teterin wrote:
>David, would you take a look at this? Some apps sometimes end up unkillable:
>
> http://www.freebsd.org/cgi/query-pr.cgi?pr=72979
>
>If you provide an encouraging answer quick enough, we may even be able to
>persuade re@ to hold the release long enough for you to fix it. Thanks!
>
> -mi
>
>
>
>
I had committed a change about 3 days ago, this might fix the problem,
can you
try ?
http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/kern_exit.c.diff?r1=1.250&r2=1.251&f=h
David Xu
Ken Smith
From: Ken Smith <kens...@cse.Buffalo.EDU>
To: David Xu <dav...@freebsd.org>
Cc: Mikhail Teterin <mi...@aldan.algebra.com>,
Michael Nottebrock <michaeln...@gmx.net>, r...@freebsd.org,
dav...@t2t2.com, dav...@viatech.com.cn,
freebsd-gn...@freebsd.org
Subject: Re: kern/72979: unkillable process(es) stuck in `STOP' state
Date: Tue, 26 Oct 2004 10:00:32 -0400
On Tue, Oct 26, 2004 at 07:11:54PM +0800, David Xu wrote:
> Mikhail Teterin wrote:
>
> >David, would you take a look at this? Some apps sometimes end up
> >unkillable:
> >
> >http://www.freebsd.org/cgi/query-pr.cgi?pr=72979
> >
> >If you provide an encouraging answer quick enough, we may even be able to
> >persuade re@ to hold the release long enough for you to fix it. Thanks!
> >
> >-mi
> >
> I had committed a change about 3 days ago, this might fix the problem,
> can you
> try ?
>
> http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/kern_exit.c.diff?r1=1.250&r2=1.251&f=h
>
That looks both promising (given symptoms) and low-risk.
Mikhail et. al., will you have any problems testing that?
--
Ken Smith
- From there to here, from here to | kens...@cse.buffalo.edu
there, funny things are everywhere. |
- Theodore Geisel |
Mikhail Teterin
From: Mikhail Teterin <mi...@aldan.algebra.com>
To: Ken Smith <kens...@cse.buffalo.edu>
Cc: David Xu <dav...@freebsd.org>,
Michael Nottebrock <michaeln...@gmx.net>, r...@freebsd.org,
dav...@t2t2.com, dav...@viatech.com.cn,
freebsd-gn...@freebsd.org
Subject: Re: kern/72979: unkillable process(es) stuck in `STOP' state
Date: Tue, 26 Oct 2004 12:07:47 -0400
=On Tue, Oct 26, 2004 at 07:11:54PM +0800, David Xu wrote:
=That looks both promising (given symptoms) and low-risk.
=
=Mikhail et. al., will you have any problems testing that?
This problem was not easy to reproduce to begin with :-(
I don't even know, how it can be positively claimed gone. Shall we merge
David's fix into RELENG_5 and hope for the best?
-mi
Scott Long
From: Scott Long <sco...@freebsd.org>
To: Mikhail Teterin <mi...@aldan.algebra.com>
Cc: Ken Smith <kens...@cse.buffalo.edu>,
David Xu <dav...@freebsd.org>,
Michael Nottebrock <michaeln...@gmx.net>, r...@freebsd.org,
dav...@t2t2.com, dav...@viatech.com.cn,
freebsd-gn...@freebsd.org
Subject: Re: kern/72979: unkillable process(es) stuck in `STOP' state
Date: Tue, 26 Oct 2004 10:14:48 -0600
Mikhail Teterin wrote:
> =On Tue, Oct 26, 2004 at 07:11:54PM +0800, David Xu wrote:
> =That looks both promising (given symptoms) and low-risk.
> =
> =Mikhail et. al., will you have any problems testing that?
>
> This problem was not easy to reproduce to begin with :-(
>
> I don't even know, how it can be positively claimed gone. Shall we merge
> David's fix into RELENG_5 and hope for the best?
>
> -mi
One of the emails in the thread was by Sam Leffler and he seemed to have
a very easy procedure for producing the problem. It only required
attaching GDB to any running threaded process. Can you look that up and
see if you get similar results?
Scott
Ken Smith
From: Ken Smith <kens...@cse.Buffalo.EDU>
To: David Xu <dav...@freebsd.org>
Cc: Mikhail Teterin <mi...@aldan.algebra.com>,
Michael Nottebrock <michaeln...@gmx.net>, r...@freebsd.org,
dav...@t2t2.com, dav...@viatech.com.cn,
freebsd-gn...@freebsd.org
Subject: Re: kern/72979: unkillable process(es) stuck in `STOP' state
Date: Tue, 26 Oct 2004 22:53:05 -0400
--5mCyUwZo2JvN/JJP
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Tue, Oct 26, 2004 at 07:11:54PM +0800, David Xu wrote:
> Mikhail Teterin wrote:
>=20
> >David, would you take a look at this? Some apps sometimes end up=20
> >unkillable:
> >
> >http://www.freebsd.org/cgi/query-pr.cgi?pr=3D72979
> >
> >If you provide an encouraging answer quick enough, we may even be able t=
o=20
> >persuade re@ to hold the release long enough for you to fix it. Thanks!
> >
> >-mi
> >
> I had committed a change about 3 days ago, this might fix the problem,=20
> can you
> try ?
>=20
> http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/kern_exit.c.diff?r1=3D=
1.250&r2=3D1.251&f=3Dh
>=20
> David Xu
David,
I think I have successfully tested your fix. Even if it doesn't wind
up fixing these problems it does fix something that shouldn't happen.
Please commit to RELENG_5 and RELENG_5_3. If you are too busy to do
it can you let me know please? I'll do it if you can't. We're making
progress on the other big show-stopper problem and might have a resolution
to that as early as tomorrow (if a patch doesn't work at least we know
the root cause of a hang and can tell people how to avoid it; but a
patch that might fix the cause of the hang is being tested). We're
thinking about doing a mini-RC2 just to test out these things and
hopefully be ready to re-roll the release when the conference is over
and people return home.
Knowing the fix it's trivially easy to do something that should not
happen:
% gdb prog # can be anything
gdb> break main
gdb> run
gdb> ^Z
% ps
% kill <pid of gdb>
Process for prog will be left behind and can't be killed. With David's
patch process for prog dies along with gdb.
Thanks...
--=20
Ken Smith
- From there to here, from here to | kens...@cse.buffalo.edu
there, funny things are everywhere. |
- Theodore Geisel |
--5mCyUwZo2JvN/JJP
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (SunOS)
iD8DBQFBfw2O/G14VSmup/YRAm65AJwLdHfdSQJF5PN82MMyfa01nTknzQCffcGW
EdNmGeogzX6PaOT2e/vJBfc=
=GcZV
-----END PGP SIGNATURE-----
--5mCyUwZo2JvN/JJP--
Ken Smith
From: Ken Smith <kens...@cse.Buffalo.EDU>
To: Ken Smith <kens...@cse.Buffalo.EDU>
Cc: David Xu <dav...@freebsd.org>,
Mikhail Teterin <mi...@aldan.algebra.com>,
Michael Nottebrock <michaeln...@gmx.net>, r...@freebsd.org,
dav...@t2t2.com, dav...@viatech.com.cn,
freebsd-gn...@freebsd.org
Subject: Re: kern/72979: unkillable process(es) stuck in `STOP' state
Date: Tue, 26 Oct 2004 23:54:19 -0400
On Tue, Oct 26, 2004 at 10:53:05PM -0400, Ken Smith wrote:
> David,
>
> I think I have successfully tested your fix. Even if it doesn't wind
> up fixing these problems it does fix something that shouldn't happen.
>
> Please commit to RELENG_5 and RELENG_5_3. If you are too busy to do
> it can you let me know please?
Sigh. Hold off a wee bit please.
That test I sent eariler was 100% reproducible and your patch fixed
that case. But I found someone else who was having similar problems
and he reports he can still get stuck processes. I'll spend some
time trying to reproduce his case to see if I can reliably make
it happen somehow. He said with this patch it does change what
happens - the process being debugged does notice that it gets hit
with a sighup when he exits gdb but the process being debugged
doesn't die.
--
Ken Smith
- From there to here, from here to | kens...@cse.buffalo.edu
there, funny things are everywhere. |
- Theodore Geisel |
Mikhail Teterin
From: Mikhail Teterin <mi+...@aldan.algebra.com>
To: Ken Smith <kens...@cse.buffalo.edu>
Cc: David Xu <dav...@freebsd.org>,
Mikhail Teterin <mi...@aldan.algebra.com>,
Michael Nottebrock <michaeln...@gmx.net>, r...@freebsd.org,
dav...@t2t2.com, dav...@viatech.com.cn,
freebsd-gn...@freebsd.org
Subject: Re: kern/72979: unkillable process(es) stuck in `STOP' state
Date: Wed, 27 Oct 2004 07:58:44 -0400
On Tuesday 26 October 2004 10:53 pm, Ken Smith wrote:
= % gdb prog # can be anything
= gdb> break main
= gdb> run
= gdb> ^Z
= % ps
= % kill <pid of gdb>
=
= Process for prog will be left behind and can't be killed. With David's
= patch process for prog dies along with gdb.
Should not just the gdb be killed in the above scenario -- without the
explicit killing of prog?
Or does killing the tracer always imply killing the tracee? Thanks!
-mi
David Xu
From: David Xu <dav...@freebsd.org>
To: Mikhail Teterin <mi+...@aldan.algebra.com>
Cc: Ken Smith <kens...@cse.buffalo.edu>,
Mikhail Teterin <mi...@aldan.algebra.com>,
Michael Nottebrock <michaeln...@gmx.net>, r...@freebsd.org,
dav...@t2t2.com, dav...@viatech.com.cn,
freebsd-gn...@freebsd.org
Subject: Re: kern/72979: unkillable process(es) stuck in `STOP' state
Date: Wed, 27 Oct 2004 20:13:11 +0800
Mikhail Teterin wrote:
>On Tuesday 26 October 2004 10:53 pm, Ken Smith wrote:
>
>= % gdb prog # can be anything
>= gdb> break main
>= gdb> run
>= gdb> ^Z
>= % ps
>= % kill <pid of gdb>
>=
>= Process for prog will be left behind and can't be killed. With David's
>= patch process for prog dies along with gdb.
>
>Should not just the gdb be killed in the above scenario -- without the
>explicit killing of prog?
>
>Or does killing the tracer always imply killing the tracee? Thanks!
>
> -mi
>
>
>
>
>
Killing tracer should always kill tracee as well, this becauses tracer
may have changed tracees 's code or data, and if tracer dies, the tracee
will be segment fault otherwise if we let it continue.
David Xu
Ken Smith
From: Ken Smith <kens...@cse.Buffalo.EDU>
To: Mikhail Teterin <mi...@aldan.algebra.com>
Cc: Ken Smith <kens...@cse.Buffalo.EDU>,
David Xu <dav...@freebsd.org>,
Michael Nottebrock <michaeln...@gmx.net>, r...@freebsd.org,
dav...@t2t2.com, dav...@viatech.com.cn,
freebsd-gn...@freebsd.org
Subject: Re: kern/72979: unkillable process(es) stuck in `STOP' state
Date: Wed, 27 Oct 2004 14:23:40 -0400
On Tue, Oct 26, 2004 at 12:07:47PM -0400, Mikhail Teterin wrote:
> =On Tue, Oct 26, 2004 at 07:11:54PM +0800, David Xu wrote:
> =That looks both promising (given symptoms) and low-risk.
> =
> =Mikhail et. al., will you have any problems testing that?
>
> This problem was not easy to reproduce to begin with :-(
>
> I don't even know, how it can be positively claimed gone. Shall we merge
> David's fix into RELENG_5 and hope for the best?
>
> -mi
Has anyone had time to test David's patch? If yes have you had
any problems since?
I should know later today if the other person who had been reporting
problems has had them stop. He reported last night he was still able
to reproduce the problem but he hasn't had any "luck" doing it today.
David, my only real question I guess is whether perhaps P_STOPPED_SINGLE
should be added to the flags that are being turned off by your patch.
If I'm following the code right what had made P_STOPPED_TRACE special
was it's one of the three signals lumped into P_STOPPED and the code
in do_tdsignal() looks for that, as well as I'm sure the schedulers...
--
Ken Smith
- From there to here, from here to | kens...@cse.buffalo.edu
there, funny things are everywhere. |
- Theodore Geisel |
David Xu
From: David Xu <dav...@freebsd.org>
To: Ken Smith <kens...@cse.Buffalo.EDU>
Cc: Mikhail Teterin <mi...@aldan.algebra.com>,
Michael Nottebrock <michaeln...@gmx.net>, r...@freebsd.org,
dav...@t2t2.com, dav...@viatech.com.cn,
freebsd-gn...@freebsd.org
Subject: Re: kern/72979: unkillable process(es) stuck in `STOP' state
Date: Thu, 28 Oct 2004 07:19:57 +0800
Ken Smith wrote:
>On Tue, Oct 26, 2004 at 12:07:47PM -0400, Mikhail Teterin wrote:
>
>
>>=On Tue, Oct 26, 2004 at 07:11:54PM +0800, David Xu wrote:
>>=That looks both promising (given symptoms) and low-risk.
>>=
>>=Mikhail et. al., will you have any problems testing that?
>>
>>This problem was not easy to reproduce to begin with :-(
>>
>>I don't even know, how it can be positively claimed gone. Shall we merge
>>David's fix into RELENG_5 and hope for the best?
>>
>> -mi
>>
>>
>
>Has anyone had time to test David's patch? If yes have you had
>any problems since?
>
>I should know later today if the other person who had been reporting
>problems has had them stop. He reported last night he was still able
>to reproduce the problem but he hasn't had any "luck" doing it today.
>
>David, my only real question I guess is whether perhaps P_STOPPED_SINGLE
>should be added to the flags that are being turned off by your patch.
>If I'm following the code right what had made P_STOPPED_TRACE special
>was it's one of the three signals lumped into P_STOPPED and the code
>in do_tdsignal() looks for that, as well as I'm sure the schedulers...
>
>
>
P_STOPPED_SINGLE needn't be removed here, in fact, the flag should
already be
turned off at this point, if the flag is still there, then there must be a
bug, thread_single(SINGLE_EXIT) should turn it off if it returns or thread
will exit in the function.
P_STOPPED_TRACE is cleared by debugger via PT_DETACH in sys_process.c, but
if debugger dies without chance to call ptrace(PT_DETACH), then exit1() must
clear it.
David Xu
Ken Smith
From: Ken Smith <kens...@cse.Buffalo.EDU>
To: David Xu <dav...@freebsd.org>
Cc: Ken Smith <kens...@cse.Buffalo.EDU>,
Mikhail Teterin <mi...@aldan.algebra.com>,
Michael Nottebrock <michaeln...@gmx.net>, r...@freebsd.org,
dav...@t2t2.com, dav...@viatech.com.cn,
freebsd-gn...@freebsd.org
Subject: Re: kern/72979: unkillable process(es) stuck in `STOP' state
Date: Wed, 27 Oct 2004 20:24:02 -0400
--SLDf9lqlvOQaIe6s
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Thu, Oct 28, 2004 at 07:19:57AM +0800, David Xu wrote:
> P_STOPPED_SINGLE needn't be removed here, in fact, the flag should=20
> already be
> turned off at this point, if the flag is still there, then there must be a
> bug, thread_single(SINGLE_EXIT) should turn it off if it returns or thread
> will exit in the function.
> P_STOPPED_TRACE is cleared by debugger via PT_DETACH in sys_process.c, but
> if debugger dies without chance to call ptrace(PT_DETACH), then exit1() m=
ust
> clear it.
Thank you for the explanation.
Please go ahead with the MFC of your patch as-is. I'm sorry to have
been such a pain in the butt about this. If you have any questions
let re@ know please. It can go to both RELENG_5 and RELENG_5_3 at
this point.
Thanks for all your help with this.
--=20
Ken Smith
- From there to here, from here to | kens...@cse.buffalo.edu
there, funny things are everywhere. |
- Theodore Geisel |
--SLDf9lqlvOQaIe6s
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (SunOS)
iD8DBQFBgDwh/G14VSmup/YRAk6EAJsFckJuMDBworwYSIiLZQMaRhmyGwCeMdR1
KIMqigcun5omyj/Yjv8tZIA=
=/P59
-----END PGP SIGNATURE-----
--SLDf9lqlvOQaIe6s--
Ken Smith
> Can you do MFC for me ? I am behind a very slow and unreliable
> link at the time, I can only do it after I come home where I
> have a fast link after 10 hours. ;-)
All set. Thanks again for your help/patience with this.
One show-stopper down, one to go... :-)
--
Ken Smith
- From there to here, from here to | kens...@cse.buffalo.edu
there, funny things are everywhere. |
- Theodore Geisel |