Thread.Sleep in a NUnit test case causes the whole run to be aborted when running through NUnit test adapter

[Adrian Mustea]

Hi all.

I've been investigating this for a while, and I finally got a repro (along with an idea of what is happening).

I run NUnit tests invoked by vstest.console.exe, and using the NUnit adapter.

NUnit adapter version is 1.0, and NUnit framework is 2.6.3 (both seem to be the latest).

I have the following code:

[Test]
[Description("Thread sleep causes NUnit adapter to behave weird")]
public void NUnitAdapterBug1()
{

    Thread.Sleep(5 * 60 * 1000);
     throw new Exception("NUnit bug repro attempt");
}

I also have a TestInitialize method which just prints out 'calling TestInitialize', so I won't paste it here.

when the Thread.Sleep() line is commented out, the test runs fine, failing as expected (see screenshot 1). 

But when Thread.Sleep is enabled, even for 5 minutes, the test is just aborted somewhere during the run, and all I see is the log from TestInitialize() (see screenshot 2).

Here is how I invoke the test:

"C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" <Path_To_DLL> /Platform:x64 /Framework:Framework45 /tests:NUnitAdapterBug1 /TestAdapterPath:"E:\NuGetPackages\NUnitTestAdapter.1.0\lib"

Has anyone else encountered similar issues? Is this a timeout being hit? If so, anyone knows how to work around it ?

Thanks

[Charlie Poole]

Things to try in order to narrow this down:

1. Run under the IDE, using both Run All and also by selecting the
individual test. This will help since there are some differences in
the code path followed in each instance.

2. Mark the test with [RequiresThread]. This will force NUnit to
create a thread for the test and Join it.

3. Assuming you are using .NET 4.5, make the test async, which will
cause NUnit to wait for it to complete using a different code path.

Charlie

I think the bug is that the entire run is being completed - i.e.
considered complete by the adapter - while this one test is still
being executed.

> --
> You received this message because you are subscribed to the Google Groups
> "NUnit-Discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to nunit-discus...@googlegroups.com.
> To post to this group, send email to nunit-...@googlegroups.com.
> Visit this group at http://groups.google.com/group/nunit-discuss.
> For more options, visit https://groups.google.com/groups/opt_out.

[A.M.]

Thanks for the prompt response, Charlie. Here are the answers to your suggestions:

1. Run All - this is where I originally observed this bug. By running all tests in the lab environment (TFS). So it manifests both when running all tests and when running a test subset

2. Mark test with [RequiresThread] - this did not help, the behavior is exactly the same.

3. I will try this approach, but this will require rewriting all of our test infrastructure code, which is something we would try to avoid.

Your conclusion is correct - the adapter 'thinks' the test run is over way before the tests are actually done running.

Were you able to repro this issue ?

Thank you. 

[Charlie Poole]

Actually, I'm travelling and not in a position to repro anything. :-(
But I'm hoping somebody else will respond. Terje, are you on this
call?

Charlie

[Simone Busoli]

Can’t reproduce.

Here’s what I did

, so you might be doing something different:

• open up VS2013
  
• create a new class library project with defaults
  
• installed
  latest
  framework
   and adapter
  (the bin-deployable version)
  NuGet packages
  
• typed the following code:
  

[TestFixture]
public class Class1
{
    [SetUp]
    public void Setup()
    {
        Console.WriteLine("Setup");
    }

    [TearDown]
    public void Teardown()
    {
        Console.WriteLine("Teardown");
    }

    [Test]
    [Description("Thread sleep causes NUnit adapter to behave weird")]
    public void NUnitAdapterBug1()
    {
        Thread.Sleep(5 * 60 * 1000);
        throw new Exception("NUnit bug repro attempt");
    }
}

• Ran command (and got output)
  :
  
  

C:\Program Files (x86)\Microsoft Visual Studio 12.0>vstest.console.exe "D:\Users\Simone\Documents\Visual Studio 2013\Projects\ClassLibrary1\ClassLibrary1\bin\Debug\ClassLibrary1.dll" /Platform:x64 /Framework:Framework45 /TestAdapterPath:"D:\Users\Simone\Documents\Visual Studio 2013\Projects\ClassLibrary1\packages\NUnitTestAdapter.1.0\lib"
Microsoft (R) Test Execution Command Line Tool Version 12.0.21005.1
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Warning: Using Isolation mode to run tests as required by effective Platform:X64 and .Net Framework:Framework45 settings for test run. Use the /inIsolation parameter to suppress this warning.
Information: NUnit 1.0.0.0 executing tests is started

Information: Run started: D:\Users\Simone\Documents\Visual Studio 2013\Projects\ClassLibrary1\ClassLibrary1\bin\Debug\ClassLibrary1.dll

Information: Setup

Information: Teardown

Information: NUnit 1.0.0.0 executing tests is finished

Failed   NUnitAdapterBug1
Error Message:
   System.Exception : NUnit bug repro attempt
Stack Trace:
at ClassLibrary1.Class1.NUnitAdapterBug1() in d:\Users\Simone\Documents\Visual Studio 2013\Projects\ClassLibrary1\ClassLibrary1\Class1.cs:line 32



Standard Output Messages:
Teardown

Total tests: 1. Passed: 0. Failed: 1. Skipped: 0.
Test Run Failed.
Test execution time: 5,0131 Minutes

On Thu, Feb 6, 2014 at 1:25 AM, Adrian Mustea <amu...@gmail.com> wrote:

--

[A.M.]

Hi Simone, and thank you for stepping in. Can you please increase the Sleep to 9 minutes and try? 

I started from scratch by repeating your steps (except I use VS2012) and the 5 minutes sleep works for me too. But for 9 minutes it's failing with a warning that "NO TEST IS AVAILABLE".  Please see attached screenshot for the output I got in both cases.

Thank you.

[Simone Busoli]

Ok, give me 10 minutes ;)

--

[Simone Busoli]

I get the same result as you. I believe we are not intentionally setting any hard limit to the execution time of tests, so I can only assume that the VS test infrastructure sets that limit, which BTW is not so unreasonable. Do you really have tests taking that long to execute?

Simone

On Thu, Feb 6, 2014 at 11:23 PM, A.M. <amu...@gmail.com> wrote:

--

[Adrian Mustea]

Yes, these are UI tests, so they take a while to run. And the bigger problem is that no other tests will run after the one which fails.

Any ideas on how to dig further ? I hope to find a solution to this.

I wonder if there is a way to 'make' VS test infra work around this problem.

--
You received this message because you are subscribed to a topic in the Google Groups "NUnit-Discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nunit-discuss/fCMZ4I-rNJQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nunit-discus...@googlegroups.com.

[Simone Busoli]

I tried a google search but that didn't bring up anything interesting. I would suggest searching further and looking into the docs to write test adapters in case they mention anything pertaining timeouts and then either post on StackOverflow or get in touch directly with MS via their forums or direct contacts if you can find them. The NUnit adapter runs within the VS testing infrastructure, I'm personally not aware of an extension point which allows to extend the duration of tests, though there might well be one.

Simone

[Terje Sandstrøm]

Hi !

I will check  with the product group and come back on this.

/terje

[Buaban Buataitom]

Hi All,

I also face this problem.

My test is about checking notifications of events. The test will wait for next notification, which may appear within 5-15 minutes.

If the test doesn't see any notifications, it will sleep for 13-17 minutes and recheck again.

However, when it wake up, the test is aborted and I got this message in output window of VS.

Sleep: 943998
The thread '<No Name>' (0x1bbc) has exited with code 0 (0x0).
The thread '<No Name>' (0xe14) has exited with code 0 (0x0).
The thread '<No Name>' (0x1934) has exited with code 0 (0x0).
The thread '<No Name>' (0x2f80) has exited with code 0 (0x0).
The thread '<No Name>' (0x25cc) has exited with code 0 (0x0).
The thread '<No Name>' (0x2e6c) has exited with code 0 (0x0).
The thread '<No Name>' (0x26a0) has exited with code 0 (0x0).
The thread '<No Name>' (0x25e0) has exited with code 0 (0x0).
The thread '<No Name>' (0x2a4c) has exited with code 0 (0x0).
A first chance exception of type 'System.Threading.ThreadAbortException' occurred in mscorlib.dll
An exception of type 'System.Threading.ThreadAbortException' occurred in mscorlib.dll but was not handled in user code
The thread '<No Name>' (0x2d08) has exited with code 0 (0x0).
The thread 'TestRunnerThread' (0x1dc0) has exited with code 0 (0x0).
The thread 'EventPumpThread' (0x2044) has exited with code 0 (0x0).
System.Transactions Critical: 0 : <TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Critical"><TraceIdentifier>http://msdn.microsoft.com/TraceCodes/System/ActivityTracing/2004/07/Reliability/Exception/Unhandled</TraceIdentifier><Description>Unhandled exception</Description><AppDomain>vstest.executionengine.x86.exe</AppDomain><Exception><ExceptionType>System.AppDomainUnloadedException, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType><Message>Attempted to access an unloaded AppDomain.</Message><StackTrace>
</StackTrace><ExceptionString>System.AppDomainUnloadedException: Attempted to access an unloaded AppDomain.</ExceptionString></Exception></TraceRecord>
The thread '<No Name>' (0x2ddc) has exited with code 0 (0x0).
The thread '<No Name>' (0x1d6c) has exited with code 0 (0x0).
The thread '<No Name>' (0x2fdc) has exited with code 0 (0x0).
The program '[9556] vstest.executionengine.x86.exe: Managed (v4.0.30319)' has exited with code 0 (0x0).

Can anyone tell me how to fix this?

Platform:

VS2014 U4 / Win7 x64 / NUnit Test Adapter 1.1.0.0

I've never try this with NUnit console or NUnit GUI yet; will let you know the result soon.

Thanks.

[Terje Sandstrøm]

Hi !

This issue should be fixed in version 1.2 of the adapter. 

I checked the repro in the first post here right now, and it works.

Teh 1.2 version is on its way out to the gallery and nuget, but you can  download it now from https://github.com/nunit/nunit-vs-adapter/releases/tag/V1.2 

/terje