Use a single AppDomain for all tests - does it really work as advertised?

Showing 1-14 of 14 messages
Use a single AppDomain for all tests - does it really work as advertised? Tony 5/12/12 9:30 PM

There is a well-known problem when unit testing managed C++ code in Visual Studio: unmanaged code calling back into managed can't cross app domains and crashes unit tests, as documented here:http://social.msdn.microsoft.com/Forums/is/vststest/thread/fc7bc074-ff05-407b-b646-d9e5532c6998 and in more details here: http://lambert.geek.nz/2007/05/29/unmanaged-appdomain-callback

Using /noisolation flag is one solution but it works only when running tests outside of Visual Studio, meaning you can't debug your tests in Visual Studio.

For me this is a huge thing. Discovering that even Visual Studio 11 beta doesn't offer anything new here was a major disappointment.

So I turned to NUnit, nunit-x86.exe, version 2.6 to be precise, and started running my test with "Use a single AppDomain for all tests" checked. Disappointingly, NUnit displays message: "An unhandled System.ArgumentException was thrown while executing this test: Cannot pass a GCHandle across AppDomains." After that  nunit-x86.exe crashes.

Hope was that, given instructions on how to debug NUnit tests, which can be found on SO, I will be able to both debug and avoid GCHandle AppDomain issue. While that hope is somewhat dashed by the crash (and I didn't even attach the debugger yet) I still hope I must be doing something wrong.

Re: [nunit-discuss] Use a single AppDomain for all tests - does it really work as advertised? charlie 5/12/12 10:58 PM
"Use a single AppDomain for all tests" means that there will be two AppDomains
involved - one for NUnit itself and one for the tests. There is still
cross-domain
communication taking place between the tests and NUnit.

At a minimum, it's generally necessary that C++ tests be contained in a separate
assembly and be compiled to pure IL. Mixed assemblies won't cut it.

Here's a somewhat off-the-wall suggestion to try. Download NUnitLite 0.7 and
try running your tests using it. This is *not* something I have tried
with C++ tests
or heard about anyone trying before, but it would put *all* the code into a
single AppDomain and give you text-based output. Sorry, no gui.

You could also use VS11 with the NUnit unit test adapter, available on VS
Gallery, but I haven't found the performance to be up to snuff in the Beta.

Charlie
> --
> You received this message because you are subscribed to the Google Groups
> "NUnit-Discuss" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/nunit-discuss/-/2smLYqTQCPwJ.
> To post to this group, send email to nunit-...@googlegroups.com.
> To unsubscribe from this group, send email to
> nunit-discus...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/nunit-discuss?hl=en.
Re: Use a single AppDomain for all tests - does it really work as advertised? Tony 5/13/12 12:18 AM
The intention is not to test C++ code by using mixed assembly.
The project is by necessity a mixed mode assembly, but
the classes tested are all managed C++/CLI classes.

NUnitLite 0.7 may work, as long as I can somehow attach a debugger
before it starts executing tests and everything is in a single
AppDomain.

At this point Charlie, many thanks, but where do I find examples on
how to use classes in NUnitLite.Runner?

Quick search on Google for NUnitLite tutorial doesn't show anything
useful.

On May 13, 1:58 am, Charlie Poole <nunit...@gmail.com> wrote:
> "Use a single AppDomain for all tests" means that there will be two AppDomains
> involved - one for NUnit itself and one for the tests. There is still
> cross-domain
> communication taking place between the tests and NUnit.
>
> At a minimum, it's generally necessary that C++ tests be contained in a separate
> assembly and be compiled to pure IL. Mixed assemblies won't cut it.
>
> Here's a somewhat off-the-wall suggestion to try. Download NUnitLite 0.7 and
> try running your tests using it. This is *not* something I have tried
> with C++ tests
> or heard about anyone trying before, but it would put *all* the code into a
> single AppDomain and give you text-based output. Sorry, no gui.
>
> You could also use VS11 with the NUnit unit test adapter, available on VS
> Gallery, but I haven't found the performance to be up to snuff in the Beta.
>
> Charlie
>
>
>
>
>
>
>
> On Sat, May 12, 2012 at 9:30 PM, Tony <ton...@gmail.com> wrote:
> > There is a well-known problem when unit testing managed C++ code in Visual
> > Studio: unmanaged code calling back into managed can't cross app domains and
> > crashes unit tests, as documented
> > here:http://social.msdn.microsoft.com/Forums/is/vststest/thread/fc7bc074-f...and
Re: Use a single AppDomain for all tests - does it really work as advertised? Tony 5/13/12 1:25 AM
Ok, found example here:
http://www.nunitlite.com/index.php?p=faqs

and realized that sample VS solution comes with source, not binary
distribution.

On my way now...
Re: [nunit-discuss] Re: Use a single AppDomain for all tests - does it really work as advertised? charlie 5/13/12 9:44 AM
Sorry, the binary distribution of NUnitLite 0.7 is at
http://launchpad.net/nunitlite. The website has not been updated yet.

Charlie
Re: [nunit-discuss] Re: Use a single AppDomain for all tests - does it really work as advertised? Tony 5/13/12 10:15 AM
Looks like I will need to recompile NUnitLite as x86 module,
since it is supposed to load (eventually) a C++/CLI mixed-mode assembly which is 32 bit.

I am getting System.IO.Exception: could not load assembly X.. The given assembly name or codebase was invalid.
Re: [nunit-discuss] Re: Use a single AppDomain for all tests - does it really work as advertised? charlie 5/13/12 10:50 AM
NUnitLite (like most of NUnit) is compiled as CIL only. How it
executes depends on how you compile the exe that references it - in
this case, your test assembly.

Older documentation talks about including NUnitLite as source in your
project. You'll note that this is removed from the documentation
(README.txt) that comes with 0.7.

Charlie
> https://groups.google.com/d/msg/nunit-discuss/-/23_SUUhcyr8J.
Re: [nunit-discuss] Re: Use a single AppDomain for all tests - does it really work as advertised? Tony 5/13/12 11:32 AM
Right!

So I made my NUnitLiteRunner.exe x86 target and now in project bin\x86\Debug folder I have:
NUnitLiteRunner.exe (which is a simple console program doing not more than: new TextUI(writer).Execute(args);
then I have:
NUnitLiteTestDriverGCRoot.dll, which contains test fixture and test methods in C#, 
and finally I have: 
GCrootCode2Test.dll, which is mixed mode C++/CLI assembly.

If I start console in this folder and type:
>NUnitLiteRunner.exe NUnitLiteTestDriverGCRoot.exe

I get report about 1 test that was run and 0 failures. Yipeee!

Getting the same thing to run in debug mode and from Visual Studio 2010 seems to require a major brainpower.
Even giving full paths to everything doesn't seem to help. This will take some time as there are only a few options to
debug from Visual Studio and command line eother takes just the assembly name or a full path, thus
the problem seems trivial, but it flatly doesn't work in any of the variations, so far.

I am clearly doing something path-related wrong becaus while:

>NUnitLiteRunner.exe NUnitLiteTestDriverGCRoot.exe

command on command prompt executes the single test method with no failure, the command with -explore option

>NUnitLiteRunner.exe NUnitLiteTestDriverGCRoot.exe -explore

crashes with System.ArgumentException: the path is not of a legal form.
  
Re: [nunit-discuss] Re: Use a single AppDomain for all tests - does it really work as advertised? charlie 5/13/12 11:53 AM
Hi,

On Sun, May 13, 2012 at 11:32 AM, Tony <ton...@gmail.com> wrote:
> Right!
>
> So I made my NUnitLiteRunner.exe x86 target and now in project bin\x86\Debug
> folder I have:
> NUnitLiteRunner.exe (which is a simple console program doing not more than:
> new TextUI(writer).Execute(args);
> then I have:
> NUnitLiteTestDriverGCRoot.dll, which contains test fixture and test methods
> in C#,
> and finally I have:
> GCrootCode2Test.dll, which is mixed mode C++/CLI assembly.
>
> If I start console in this folder and type:
>>NUnitLiteRunner.exe NUnitLiteTestDriverGCRoot.exe
>
> I get report about 1 test that was run and 0 failures. Yipeee!

If you put the same main in NUnitLiteTestDriverRoot.exe, you can just
run that program and eliminate NUnitLiteRunner.exe. That's the way
NUnitLite is actually intended to be used. However, if you are using the
Main() of NUnitLiteTestDriverRoot.exe for some other purpose, then
what you're doing makes sense.

>
> Getting the same thing to run in debug mode and from Visual Studio 2010
> seems to require a major brainpower.
> Even giving full paths to everything doesn't seem to help. This will take
> some time as there are only a few options to
> debug from Visual Studio and command line eother takes just the assembly
> name or a full path, thus
> the problem seems trivial, but it flatly doesn't work in any of the
> variations, so far.

I'm not sure how you're running. What I would typically do is to make
NUnitLiteRunner.exe the Startup assembly and give it the appropriate
command-line arguments. Is that what you're doing?

>
> I am clearly doing something path-related wrong becaus while:
>
>>NUnitLiteRunner.exe NUnitLiteTestDriverGCRoot.exe
>
> command on command prompt executes the single test method with no failure,
> the command with -explore option
>
>>NUnitLiteRunner.exe NUnitLiteTestDriverGCRoot.exe -explore
>
> crashes with System.ArgumentException: the path is not of a legal form.

You may have found a bug. Is there a stack trace?

Charlie
> https://groups.google.com/d/msg/nunit-discuss/-/WlX2t5sdVu8J.
Re: [nunit-discuss] Re: Use a single AppDomain for all tests - does it really work as advertised? Tony 5/13/12 12:01 PM
The stack is:

C:\Users\Tonko\Documents\Visual Studio 2010\Projects\TestingTesting\bin>NUnitLit
eRunner.exe NUnitLiteTestDriverGCRoot -explore
NUnitLite 0.7.0 ()
Copyright 2012, Charlie Poole

Runtime Environment -
    OS Version: Microsoft Windows NT 6.1.7601 Service Pack 1
  .NET Version: 4.0.30319.17379

System.ArgumentException: The path is not of a legal form.
   at System.IO.Path.NormalizePath(String path, Boolean fullCheck, Int32 maxPath
Length)
   at System.IO.Path.GetFullPathInternal(String path)
   at System.IO.Path.GetFullPath(String path)
   at NUnitLite.Runner.CommandLineOptions.ExpandToFullPath(String path)
   at NUnitLite.Runner.CommandLineOptions.get_ExploreFile()
   at NUnitLite.Runner.TextUI.ExploreTests()
   at NUnitLite.Runner.TextUI.Execute(String[] args)

Other than that, yes, I am providing command line arguments but giving a full path,
or just the assembly name, even adding Working directory, nothing helps the runner
find and load dlls, when pressing F5 on runner exe as set as Startup Project in Visual Studio 2010.

I am greeted with "the system cannot find the file specified".
Re: [nunit-discuss] Re: Use a single AppDomain for all tests - does it really work as advertised? Tony 5/13/12 12:20 PM
BTW, if this is the only unit testing framework on the planet that allows me to
debug C++/CLI unit test code inside Visual Studio, then I am prepared to work on
VIX for it, a great, integrated VS extension with fancy UI for tests to run and test results,
and earn everlasting notoriety for myself, as well as the project.

Thus, I am more keen on having one test runner and many test fixture dlls, rather than
intergrating NUnitLite in a single test console exe.

Even if I completely understand that its "guiding spirit", considering the target devices, has
nothing to do with it.
 
On Sunday, May 13, 2012 2:53:09 PM UTC-4, charlie wrote:
Re: [nunit-discuss] Re: Use a single AppDomain for all tests - does it really work as advertised? charlie 5/13/12 12:54 PM
You seem to have found a bug. If you use -explore:filename the xml
representation of the tests will be saved in the file specified. The
intent was that leaving out the filename should send output to the
console in the form of a list of test names, but that's not working.
If you file this at http://bugs.launchpad.net/nunitlite it will help
ensure it gets fixed.

Charlie
> https://groups.google.com/d/msg/nunit-discuss/-/s57CRrMktL4J.
Re: [nunit-discuss] Re: Use a single AppDomain for all tests - does it really work as advertised? charlie 5/13/12 1:05 PM
Hi...

On Sun, May 13, 2012 at 12:20 PM, Tony <ton...@gmail.com> wrote:
> BTW, if this is the only unit testing framework on the planet that allows me
> to
> debug C++/CLI unit test code inside Visual Studio, then I am prepared to
> work on
> VIX for it, a great, integrated VS extension with fancy UI for tests to run
> and test results,
> and earn everlasting notoriety for myself, as well as the project.
>
> Thus, I am more keen on having one test runner and many test fixture dlls,
> rather than
> integrating NUnitLite in a single test console exe.
>
> Even if I completely understand that its "guiding spirit", considering the
> target devices, has
> nothing to do with it.

The original spirit of NUnitLite was to run two sorts of environments:
limited resource hardware (eg: phones) and embedded within other
applications (eg: testing VS extensions). Quite recently, I discovered
that nunitlite can be quite useful in normal desktop or web testing as
well and I tried to keep that in mind in doing the 0.7 release.

A VS extension that used NUnitLite would be an interesting idea. OTOH,
there's a great deal of virtue in simplicity. By setting up your test
assembly as an exe, with a simple main containing new
TextUI().Execute(args) you can simply run the tests, as NUnitLite
assumes the calling assembly is the one containing fixtures unless
some other assembly is specified. If you want to have multiple
assemblies containing tests, that's OK too. Just set up multiple
startup applications in Visual Studio and run them all. You get a
certain amount of speedup due to the parallel execution as a result.

Note that if you install NUnitLite into a project using NuGet, it
automatically adds a Program.cs file to the project which works as
above.

It's possible that I'll eventually produce an NUnitLite adapter for
VS11, working like the NUnit adapter but using NUnitLite. That won't
be until the next VS beta however.
> https://groups.google.com/d/msg/nunit-discuss/-/62m02ZuOvjsJ.
Re: [nunit-discuss] Re: Use a single AppDomain for all tests - does it really work as advertised? Tony 5/13/12 1:50 PM
Using NuGet and making my fixture-containing Dll a Console exe app worked.

I was able to walk through the code in debugger and my test method executed with no errors.

I am not too happy that I don't understand why other setups in Visual Studio cause me troubles, 
I know many programmers using VS would try them first, but that is a possible topic for another thread.