NUnit extension with NUnit Test Adapter

[Carsten König]

Hi,

I'm sure this is a minor issue but how do you enable/install an NUnit-extension for the NUnit test adapter in VS2012?

Background:
I'm trying to contribute to the FsCheck project where some guy already added such an extension - but I just cannot get it working with Resharper or with the NUnit-test-adapter.
I tried creating a addins directory in the corresponding resharper and vs-addin folders but I always get errors (the test-adapter basically tells me that it does not want a function returning anything but unit/void).

We are trying there to add a "Property"-Attribute that should run using the framework (FsCheck is a port from QuickCheck, and a property is a function with some arguments returning bool (mostly)).
So the guy writing the extension - derived an attribute from TestAttribute and wrote a NUnitAddin.

I have to say that I don't really get how you use those, but AFAIK the addin worked with the bundled NUnit-Testrunner (though I did not try).

Maybe some of you could give me a hint here.

Thanks.

Carsten

[Charlie Poole]

You are breaking new ground here! We have not really tested the use of addins with the adapter. In theory, you should be able to create an addins directory in the directory where the nunit.core and nunit.util assemblies are installed by the adapter. However, you may run into access problems, depending on how the adapter is installed and what permissions you have.

Charlie

Carsten

--
You received this message because you are subscribed to the Google Groups "NUnit-Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nunit-discus...@googlegroups.com.
To post to this group, send email to nunit-...@googlegroups.com.
Visit this group at http://groups.google.com/group/nunit-discuss.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Carsten König]

Hi, can you explain a bit what kind of permissions are needed?
I think I already tried that - I installed it with the VS2012 extension manager and AFAIK it is somewhere in [user]\AppData\Local\Microsoft\VisualStudio\11.0\Extensions\ (in my case it's in [user]\AppData\Local\Microsoft\VisualStudio\11.0\Extensions\cqj1nkk1.vjw)

there is extension.vsixmanifest, nunit.core, NUnit.VisualStudio.TestAdapter.dll, ..

I will recheck tomorrow but I tried adding a "addins" folder there ... did nothing for me (restarted VS ... of course)
I am rather sure that this folder is the right one, because when I replaced the nunit.core.dll with an older version VS did not find some of my tests (in this case some in a F#-module if I remember correctly).

[Charlie Poole]

First of all, you have to be able to create the folder and copy files there. Second, when running tests, nunit needs to be able to examine the directory and load assemblies from the addins folder. If you did not have that permission, you would see a security exception.

From what you describe, you don't seem to be having a security problem. NUnit will always look for the addins folder in the same location as it's nunit.core assembly (or the parent directory in the case where nunit.core is located in a directory named 'lib', which does not apply to your case). It won't look anywhere else - a limitation we would like to correct, but not applicable here either.

To see what is actually going on, you would need to run debug versions of both the adapter and the nunit assemblies. That's how I would debug it if I were not travelling on vacation right now. :-) I suggest verifying that your addin works under NUnit itself - using the GUI for example - and then filing a bug against the adapter.

Charlie

[Kurt Schelfthout]

Hi,

I've checked that our addin works fine with the "normal" runner. It doesn't show up in the addin window though, but it does run the tests correctly. Not sure if that means anything.

As for running it via the VS test adapter, I've tried copying the addin files and all their dependencies to an addins subfolder where the NUnit test adapter lives, as well as to that folder itself. I was trying to figure out using process monitor and fuslogvw what it is looking for, but to no avail.

One thing I noticed is the NUnit uses GetExecutingAssembly to determine in which path to look. Since there is shadow copying and probably some sandboxing in VS going on to load the adapter, it's very unclear what result that is going to give. Just a gut feeling I have. :)

It does seem to look in an addins folder and find our addin. Maybe something else is going wrong and the adapter is a bit more picky about the adding somehow? Hard to find out...

Best, Kurt

[Charlie Poole]

Hi Kurt,

If the addin is not listed in the GUI addins window, that means it's not loaded. Are you sure your addin is executing? Sometimes the absence of an addin simply gives a false positive. You can make sure by using the RequiredAddinAttribute on one or more tests.

There is an alternative way to load addins, which I forgot to mention earlier and which may help in the situation where the adapter is in use. In order to make it easier to test addins, NUnit checks each test assembly to see if an addin is defined within it. Any dependent assemblies must, of course, also be found. This isn't very convenient if you want to use the addin generally but it may help in the case where only one test assembly needs to use the addin.

It's very clear to me that we need some easier way (in NUnit 3.0) for addins to be created and tested!

Charlie

--

[Carsten König]

Hi,

I just try to get another go on this.

As Kurt, I get positive results if I copy the (addin)assemblies to a addins-folder within the testruner (I used the one you get with the nuget-packag NUnit.Runners.2.6.2 under tools.
If i do not add such an folder all test will break:

PropertyExamples.revIdVerbose:
Test method has non-void return type, but no result is expected

So I tried your suggestion and included the implementation of the addin (not that much to be honest) into the assembly beeing testet ... well sorry but this does not work - same error.

Now I went and included the complete code (Property definition and Addin) into the project and removed the references to the other project:
- NUnit Test runner now gives me green lights (aside from one test - this one might be wrong)
- VS2012 test runner gives is just the same :D

I feel I am there on something

[Carsten König]

NO it is even better this would work
Now only to get this working with the assemblies (instead of copying it) and we might get this still working by adding such a file with nuget during installation

[Kurt Schelfthout]

Hi Charlie,

As Carsten says, yes, we're sure the addin is working, I can see output particular to the addin in the output window. I don't know why it's not listed...the addin is written in F# and .NET 4.0, don't know if that matters (in my experience, it can matter in unexpected ways, which is why I mention it).

Looks like we have a workaround of sorts. But indeed, as addin writers we would like a much better story here. Is it possible for someone to have a look at why the addin isn't loaded from the TestAdapter addins folder? I can definitely see it probing there - it might be an easy fix. (I am guessing NUnit 3.0 is going to take a while)

That would help to decide whether we want to inflict a difficult to maintain solution on our users or not...

Best, Kurt

[Charlie Poole]

Hi Kurt,

Ah! I missed (if it was in your earlier post) that the addin is written using .NET 4.0. Since the NUnit core is built using .NET 3.5 (2.0 CLR) and since addins form an addition to the core, our assumption is that all addins are also built using the 2.0 CLR. This is admittedly a severe restriction, but one that cannot be eliminated without changing how we build NUnit.

This explains entirely why the addin won't show up in the GUI Addins dialog: it simply can't be loaded. OTOH, if you are running tests under .NET 4.0, they are executed in a separate process, where the addin can load successfully. I think that's what may be happening under the NUnit runner.

The NUnit Adapter for VS is also built using .NET 3.5, in order to allow it to run tests in any environment with CLR 2.0 or higher. Visual Studio itself determines how the tests are detected and run. Visual Studio uses two different processes for test discovery and execution, so we will need to determine how each of those is run.

Could you please file a bug against the adapter for this? We'll take a look at recognition of simple addins first, then examine the .NET 4.0 issue and finally, if necessary, check out any special issues with F#.

Although NUnit 3.0 is a bit longer term than you would like, it's really the best ultimate solution, since it uses separate builds of the framework for each CLR.

Charlie

Charlie

[Kurt Schelfthout]

Hi Charlie,

Yes, all the CLR versions are a bit of a pain. Throw in two different versions of F# as well, makes me glad I don't have to deal with a hugely popular OS project like yours :)

Anyway, I've filed https://bugs.launchpad.net/nunit-vs-adapter/+bug/1193957

Looking forward to improvements in this area.

Best, Kurt