TestDriven.NET crashes when debugging - object is in a zombie state

584 views
Skip to first unread message

Fabian Schmied

unread,
Aug 25, 2010, 11:06:12 AM8/25/10
to testdri...@googlegroups.com
Hi,

Has anyone experienced the problem mentioned in the subject line?

It manifests itself as follows:

Sometimes, when you choose to run a test with the debugger (Test
with/Debugger), TestDriven.NET (or, actually, the
ProcessInvocation86.exe process) crashes. The debugger, however,
doesn't get that the process has crashed and still thinks it is
attached to that process.

You can't do anything in that state, of course, you can only "stop
debugging". In which case the debugger displays a message that it
cannot detach from ProcessInvocation86.exe because that process is in
a zombie state [1]. It asks whether you want to terminate the process
instead. You have to choose "Yes" (because "No" gets you back to the
state where you can't do anything apart from stopping), then wait for
a long time (timeout?). Then the debugger finally exits. And then you
have to close and restart Visual Studio because it has locked the
assembly that was about to be run by TestDriven.NET.

This whole process is very annoying because the debugger always takes
so long to determine that it couldn't detach/kill the process, and of
course because your work is interrupted and you need to restart Visual
Studio.

Here's how I can reproduce the issue on my own PC and on my
co-worker's (both TestDriven.NET 3.0 - he has got Enterprise build
2749, I've got an extraterrestrial one; both VS 2010 on Windows Server
2008 R2, 64bit OS but TestDriven.NET configured to run with 32 bits):

- Create a failing test using NUnit (eg., a test doing: new List<int> (-3)).
- Run the test without the debugger, it should fail as expected.
- Stop the ProcessInvocation86.exe using its notification icon. (Not
sure if this is required.)
- Configure Visual Studio to _not_ break on "Thrown" CLR exceptions,
but to _do_ break on "User-unhandled" exceptions. (I think, Just My
Code must be on in order to get that distinction.)
- Run the test using the debugger. The test will be executed and the
assertion message be written to the Output window, but then
ProcessInvocation86.exe crashes - see above.

I tried to attach a second debugger to the dying
ProcessInvocation86.exe (to see why it crashed), but that doesn't work
because that process already has a debugger attached.
The issue does not occur if you configure Visual Studio to break on
"Thrown" (First-chance) CLR exceptions.

If I can do anything to help diagnose the issue, please let me know.

Regards,
Fabian

[1] The term "zombie process" is actually a term from Unix meaning a
dead-but-still-around process. In this case, the process is gone - it
has crashed, and task manager doesn't show it any longer. But the
debugger is still attached to it. At least emotionally.

Jamie Cansdale

unread,
Aug 26, 2010, 1:56:53 PM8/26/10
to testdri...@googlegroups.com, Jeff V
Hi Fabian,

I haven't been able to reproduce this issue using Windows 7 64-bit. Do
you have any other machines you could try this on? I'd be interested
to know if it does this on all Windows Server
2008 R2 machines and whether you're able to reproduce it on a Windows 7 machine.

This issue has been reported once before by Jeff V, but we were never
able to get to the bottom of it.

Regards,
Jamie.

--
http://www.testdriven.net
http://twitter.com/jcansdale
http://weblogs.asp.net/nunitaddin

> --
> You received this message because you are subscribed to the Google Groups "TestDriven.NET Users" group.
> To post to this group, send email to testdri...@googlegroups.com.
> To unsubscribe from this group, send email to testdrivenuse...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/testdrivenusers?hl=en.
>
>

Fabian Schmied

unread,
Aug 30, 2010, 8:40:26 AM8/30/10
to testdri...@googlegroups.com
Hi Jamie,

> I haven't been able to reproduce this issue using Windows 7 64-bit. Do
> you have any other machines you could try this on? I'd be interested
> to know if it does this on all Windows Server
> 2008 R2 machines and whether you're able to reproduce it on a Windows 7 machine.

I can reproduce it on two Windows Server 2008 R2 machines (64 bit),
but not on a third one. The third machine has TDD.NET 3.0 Personal
build 2749.

I've investigated a little further, and I think the question is not
about the zombie state, but about why ProcessInvocation86.exe actually
crashes. The crash is what then leaves the debugger confused with a
zombie.

Here's the event log entry for the crash (from my own system):

Faulting application name: ProcessInvocation86.exe, version:
3.0.1862.0, time stamp: 0x4c18e2ea
Faulting module name: mscorwks.dll, version: 2.0.50727.4952, time
stamp: 0x4bebd49a
Exception code: 0xc0000005
Fault offset: 0x0047761a
Faulting process id: 0x%9
Faulting application start time: 0x%10
Faulting application path: %11
Faulting module path: %12
Report Id: %13

The exception code indicates an Access Violation in mscorwks.dll.
Unfortunately, the crash did not create a minidump file (probably
because the process already had a debugger attached), so I couldn't
determine where in mscorwks.dll that access violation actually
occurred.

Note that the crash occurs _after_ the test has printed the assertion
message, but _before_ the "0 passed, 1 failed, ..." line is printed to
the test window. IIRC, that is: Since I captured that event log entry
above, I haven't been able to reproduce the problem on my own system
any longer. Although the crash used to come quite reliably before...

Fabian

Jamie Cansdale

unread,
Sep 2, 2010, 12:38:13 PM9/2/10
to testdri...@googlegroups.com
Hi Fabian,

Thanks for the update.

> I've investigated a little further, and I think the question is not
> about the zombie state, but about why ProcessInvocation86.exe actually
> crashes. The crash is what then leaves the debugger confused with a
> zombie.
>

I wonder if it might be something in the network stack that is causing
the process to crash. TestDriven.Net uses a TCP/IP channel to
communicate with its test process. In the past I've known network
security applications to cause .NET to crash. Is there any personal
firewall/security software installed on the two machines where it
crashes that isn't installed on the machine where it works?

> above, I haven't been able to reproduce the problem on my own system
> any longer. Although the crash used to come quite reliably before...
>

Is your own system running Windows Server 2008 R2 as well?

Regards,
Jamie.

On Mon, Aug 30, 2010 at 1:40 PM, Fabian Schmied

Fabian Schmied

unread,
Sep 3, 2010, 3:09:53 AM9/3/10
to testdri...@googlegroups.com
> I wonder if it might be something in the network stack that is causing
> the process to crash. TestDriven.Net uses a TCP/IP channel to
> communicate with its test process. In the past I've known network
> security applications to cause .NET to crash. Is there any personal
> firewall/security software installed on the two machines where it
> crashes that isn't installed on the machine where it works?

There is only Microsoft Forefront Client Security.

>> above, I haven't been able to reproduce the problem on my own system
>> any longer. Although the crash used to come quite reliably before...
>>
> Is your own system running Windows Server 2008 R2 as well?

Yes. All the systems I tested it on are.

It's really strange; when I wrote the original e-mail, I used to be
able to cause that problem every time I followed the steps outlined in
there. Now, however, I can't provoke the crash any longer. There must
be some additional condition that was true last week (when that bug
really made me unable to work) and now that I can't see. Some sort of
race condition made more likely by something external, perhaps?

The crash came _after_ the test completed, so the exception was
actually handled by the test runner; only then did the process crash.

When/If the issue starts occurring again, I'll try to discover more information.

Fabian

Jamie Cansdale

unread,
Sep 3, 2010, 5:43:35 AM9/3/10
to testdri...@googlegroups.com
> The crash came _after_ the test completed, so the exception was
> actually handled by the test runner; only then did the process crash.
>
I wonder if the process is crashing as VS attempts to detach the
debugger? I'm using the same command that a human would use when
detaching the debugger. If there's a bug in the VS / managed debugger
then TD.Net will be bitten by it. My only hope is to understand when
it occurs and attempt to work around the issue.

> When/If the issue starts occurring again, I'll try to discover more information.
>

Thanks, I appreciate it. This is turning out to be a difficult to nail
down heisenbug!

Regards,
Jamie.

Fabian Schmied

unread,
Sep 3, 2010, 6:53:25 AM9/3/10
to testdri...@googlegroups.com
> I wonder if the process is crashing as VS attempts to detach the
> debugger? I'm using the same command that a human would use when
> detaching the debugger. If there's a bug in the VS / managed debugger
> then TD.Net will be bitten by it. My only hope is to understand when
> it occurs and attempt to work around the issue.

Yes, at the moment, it definitely looks like that's the problem. I've
managed to reproduce the issue on a co-worker's PC and experimented a
little more.

First, to narrow down the point of time of the crash, I've checked
what exactly the output window says. The crash definitely happens
after the assertion message is written to the window, but before the
"x tests succeeded, y tests failed, ..." message is written. In fact,
instead of that message, I get a message telling me the debugger could
not detach (because the process is in a zombie state).

You say you use the same command that a human would use
(Debug.DetachAll?). If that's the case, the question is whether the
point of time where the command is sent is (sometimes) responsible for
the error.

>> When/If the issue starts occurring again, I'll try to discover more information.
>>
> Thanks, I appreciate it. This is turning out to be a difficult to nail
> down heisenbug!

I can't get another debugger to attach to the dying
ProcessInvocation86.exe, this will always cause a (different) Access
Violation, so I can't really say where exactly the access violation
occurs.

I've tried to enable unmanaged debugging on the test project (usually,
only the managed debugging engine would be used), and this,
interestingly, will cause ProcessInvocation86.exe _not_ to crash.
However, the debugger still doesn't detach (and writes a message to
the output window instead of the "x tests succeeded..." message).
When, a few seconds later, I do detach, it just works.

It seems that
- on some systems, under some conditions (maybe related to Windows
Server 2008 R2),
- when the debugger is configured not to break on exceptions but the
test case does throw an exception,
- TestDriven.NET issues the Detach (or Stop) command at a point of
time just before it writes the test summary line to the output window,
- which leads to the fact that
-- the process being debugged crashes if only managed debugging is
enabled; this causes the debugger to complain about an object in
zombie state
-- the process does not carsh but the debugger cannot detach if
unmanged debugging is enabled.

This leads me to the following speculation:
- At the point of time where TestDriven.NET calls the Detach (or Stop)
command (or shortly after),
- there's something going on in ProcessInvocation86.exe that causes
the detach command to crash the target process/the detach command not
to work correctly.

For example:
- If you're using the System.Diagnostics.Debugger class from
ProcessInvocation86.exe, there could be a race condition with (or
within) that class, for example.
- It could also be a bug in the .NET debugging infrastructure, eg. an
issue where the debugger crashes a process if it's in the middle of a
TCP operation. The bug might only occur in Windows Server 2008 R2.

Of course, even if you manage to implement a workaround, it will be
hard to test any workaround you implement, since the problem is
obviously quite random in its occurrence. (Although, on my co-worker's
PC, I can currently reproduce it every time I try.)

Regards,
Fabian

Reply all
Reply to author
Forward
0 new messages