Only can run entire suite of tests on 2.6

[nickdu]

I recently installed NUnit 2.6 on Windows 7. I have a single assembly
which contains the tests. I use the NUnit GUI. The tests appear to
execute correctly, from the NUnit perspective, if I run the entire
suite of tests. If I run just a subset of the tests some number
always fail with a module load exception which has a
SerializationException at the root of the exception chain.

I can successfully run a subset of the tests if I first run the entire
suite. However, if I exit the GUI and restart and run just a subset I
get the module load exception.

Any idea what's going on?

Also, when I get this module load exception is doesn't show up in the
results pane. Almost like it's white text on a white background. If
I use the horizontal scrollbar to scroll then I see highlighted text
show up in the results pane.

Thanks,
Nick

[Charlie Poole]

This sounds like it should be a problem with your tests. Some test is
required to be run before any others can work. Of course, such a
dependency is bad and you would want to root it out.

To verify that this is the problem, first use NUnit's own tests. You
should be able to select and run a subset with no problem.

If that works, use something like a binary search to find the problem.
Turn on checkboxes in the Gui. Select the first test you know of that
fails when run alone. See it fail. Now select in addition half of the
previously unselected tests above it. If it passes, the failing test
depends on one of those you just selected. If it still fails, then the
culprit is one of the tests you did not select. Proceed till you find
the test which causes the problem.

Warning: This will only work if the problem is one simple dependency.

Charlie

> --
> You received this message because you are subscribed to the Google Groups "NUnit-Discuss" group.
> To post to this group, send email to nunit-...@googlegroups.com.
> To unsubscribe from this group, send email to nunit-discus...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/nunit-discuss?hl=en.
>

[nickdu]

Thanks. I will try that out but I doubt that is the issue. I'm
almost positive none of the tests are dependent on the others. Also,
in this exception chain somewhere, I guess between the module load
exception and the serialization exception as I believe those are the
first and last respectfully, is an exception indicating it can't find
one of my managed assemblies which is required. The assembly is
there, obviously as it works when I run all tests.

By the way, my NUnit test assembly is targeting the .NET 4.0 runtime.
We also have some unmanaged assemblies brought into the picture by
some third party managed assemblies. Not sure if those are causing
any issues, but as I said everything works as I expect when I run all
tests.

I think it's some sort of assembly loading issue, not sure if it's
caused by NUnit or not. The reason I say this is because of the tests
that fail. They appear to be, but I have to do more work here to
verify, the methods that get past my argument validation code in my
API and then eventually call into this third party library.

Thanks,
Nick

[nickdu]

It appears my assumption is somewhat correct about which methods are
failing. It appears the .NET runtime is having a problem loading
Redbone.Framework.BusinessProcess.Common.dll. This assembly is in the
same directory as my NUnit test assembly and I have all the other
dependent assemblies there. This directory is c:\data\development
\Redbone\bpf\tests\bin\debug. However, when the fusion loader looks
for the Common assembly it's looking in the app base directory where
the NUnit GUI .exe exists. I thought NUnit somehow changed the app
base (maybe by creating another appdomain and loading the tests from
there) to the directory where the test assembly exists? My main
framework API, which all the tests make use of, gets loaded and that's
in the same directory as the test assembly. Here is the log from the
fusion log viewer:

*** Assembly Binder Log Entry (3/31/2012 @ 9:36:50 AM) ***

The operation failed.
Bind result: hr = 0x80070002. The system cannot find the file
specified.

Assembly manager loaded from: C:\Windows\Microsoft.NET
\Framework64\v4.0.30319\clr.dll
Running under executable C:\Program Files (x86)\NUnit 2.6\bin\nunit-
agent.exe
--- A detailed error log follows.

=== Pre-bind state information ===
LOG: User = redbonemobile\nickdu
LOG: DisplayName = Redbone.Framework.BusinessProcess.Common
(Partial)
WRN: Partial binding information was supplied for an assembly:
WRN: Assembly Name: Redbone.Framework.BusinessProcess.Common | Domain
ID: 1
WRN: A partial bind occurs when only part of the assembly display name
is provided.
WRN: This might result in the binder loading an incorrect assembly.
WRN: It is recommended to provide a fully specified textual identity
for the assembly,
WRN: that consists of the simple name, version, culture, and public
key token.
WRN: See whitepaper http://go.microsoft.com/fwlink/?LinkId=109270 for
more information and common solutions to this issue.
LOG: Appbase = file:///C:/Program Files (x86)/NUnit 2.6/bin/
LOG: Initial PrivatePath = NULL
LOG: Dynamic Base = NULL
LOG: Cache Base = NULL
LOG: AppName = nunit-agent.exe
Calling assembly : (Unknown).
===
LOG: This bind starts in default load context.
LOG: Using application configuration file: C:\Program Files
(x86)\NUnit 2.6\bin\nunit-agent.exe.Config
LOG: Using host configuration file:
LOG: Using machine configuration file from C:\Windows\Microsoft.NET
\Framework64\v4.0.30319\config\machine.config.
LOG: Policy not being applied to reference at this time (private,
custom, partial, or location-based assembly bind).
LOG: Attempting download of new URL file:///C:/Program Files (x86)/
NUnit 2.6/bin/Redbone.Framework.BusinessProcess.Common.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/
NUnit 2.6/bin/Redbone.Framework.BusinessProcess.Common/
Redbone.Framework.BusinessProcess.Common.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/
NUnit 2.6/bin/lib/Redbone.Framework.BusinessProcess.Common.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/
NUnit 2.6/bin/lib/Redbone.Framework.BusinessProcess.Common/
Redbone.Framework.BusinessProcess.Common.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/
NUnit 2.6/bin/addins/Redbone.Framework.BusinessProcess.Common.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/
NUnit 2.6/bin/addins/Redbone.Framework.BusinessProcess.Common/
Redbone.Framework.BusinessProcess.Common.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/
NUnit 2.6/bin/Redbone.Framework.BusinessProcess.Common.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/
NUnit 2.6/bin/Redbone.Framework.BusinessProcess.Common/
Redbone.Framework.BusinessProcess.Common.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/
NUnit 2.6/bin/lib/Redbone.Framework.BusinessProcess.Common.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/
NUnit 2.6/bin/lib/Redbone.Framework.BusinessProcess.Common/
Redbone.Framework.BusinessProcess.Common.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/
NUnit 2.6/bin/addins/Redbone.Framework.BusinessProcess.Common.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/
NUnit 2.6/bin/addins/Redbone.Framework.BusinessProcess.Common/
Redbone.Framework.BusinessProcess.Common.EXE.
LOG: All probing URLs attempted and failed.

Thanks,
Nick

[Charlie Poole]

Hi Nick,

From the error below, it's clear that NUnit itself - rather than your
tests - is trying to load the dll in question. This usually happens
when a custom exception, defined in one of your assemblies, is
uncaught in the tests. Is that a possibility in this scenario?

If this is the case, then NUnit would be failing in deserializing the
exceptioni because it can't load the assembly in which the exception
is defined. One way to find out is to temporarily copy the "missing"
assembly to the directory containing nunit-agent.exe, along with any
dependencies that may be needed.

This issue with custom exceptions may or may not be your problem, but
it's a long-standing issue and I suggest looking at it next.

BTW, in NUnit 3.0, the plan is to prevent any exceptions from
travelling across the remoting boundary by catching them and
transforming them into an xml error report.

Charlie

[nickdu]

While we do throw some custom exceptions, my initial guess is that
this is not what's going on here because some of the tests that are
failing are positive tests, ones which shouldn't be throwing any
exceptions.

I just ran another scenario to maybe shed some more light on the
issue, though it doesn't shed much more light. I have a class called
Complete which has a bunch of tests which test our Complete method.
These tests always work. I have another class called Lock which tests
our Lock method. Lock doesn't work if run by itself (e.g. only
itself) after just starting the NUnit GUI. I mentioned originally
that if I run all the tests these other ones that fail by themselves
will succeed. That's not entirely true. If I start NUnit GUI up
fresh and then run the Complete tests I can then run the Lock tests,
using the currently running NUnit GUI that ran the Complete tests,
successfully.

Here is what my guess was as to what's going on. It appears the
tests, and I mean the code in the NUnit test methods, which have code
which directly references code from the Common assembly, this is the
assembly that's having the issue of getting loaded, will always work
whether run separately or when running all tests. The tests which
don't reference code from the Common assembly won't work unless the
Common assembly has already been loaded into the process by first
running a test that referenced it. Almost as if the code is getting
jitted with an app base setting which is correct and allows the .NET
assembly loader to find the Common assembly using the usual probing
mechanism, but then executed in a different appdomain which doesn't
have the correct app base setting such that if/when Common needs to be
loaded it cannot be found.

Thanks,
Nick

[nickdu]

I did some more tests, and while I found out some new behavior I still
don't know what's going on here.

In order to test your custom exception suggestion I figured I would
take one of the tests that failed and catch all exceptions and turn
them into an InvalidOperationException, e.g like the following:

catch(Exception e)
{
throw(new InvalidOperationException(string.Format("Caught an
exception: {0}", e.ToString())));
}

I then ran just that test and much to my surprise it succeeded. So
then I commented out the try/catch I added and rebuilt and retested,
just that test, and again it succeeded. I tested, individually which
a fresh new instance of NUnit GUI, each test that failed and they all
succeeded. For lock I have tests1 - test8. When I run all lock
tests, test1 - test4 succeed and test5 - test8 fail. If I run test5 -
test8 individually with a fresh instance of NUnit GUI they all
succeed. The reason for running fresh instances of NUnit GUI is
because it seems once Common gets loaded then all tests from them on
will work as expected. Also, once loading Common fails it will
continue to fail for that instance of NUnit GUI (looks like a type
initializer, e.g. static constructor, fails and once that happens the
there's no fixing it until the appdomain is unloaded).

Thanks,
Nick

> ...
>
> read more »

[Charlie Poole]

Still, this doesn't explain why your tests have anything at all to do
with the primary AppDomain. NUnit does run all tests in a separate
AppDomain and creates a new one each time you reload the tests.

Have you turned on NUnit's internal trace and looked at whether it
tells you anything?

Charlie

[nickdu]

I haven't use NUnit's tracing yet. I just now tried turning on the
most verbose level of tracing. I looked through all the trace files
generated and there didn't seem to be any information in there which
pointed to the problem.

Still doing more testing. As I mentioned before, if I select the Lock
set of tests, and Lock is a class with a bunch of test methods, and
run that set, the first four tests, test1 - test4, succeed and the
last four, test5 - test8, fail. The first four are negative test
cases which don't make it past the first assembly implementing our API
as it validates the parameters and fails. The last four have valid
parameters and thus make it farther into our API and require this
Common assembly be loaded. This is where it fails.

Interestingly, I can execute test1 successfully and then select test5
and execute that successfully in the same instance of the NUnit GUI.
However if I select all Lock test cases it fails. Does NUnit load
things differently if you select individual tests version a set of
tests by choosing the class?

Thanks,
Nick

> ...
>
> read more »

[Charlie Poole]

Well, the absence of trace info from NUnit at least tells us that NUnit is
not aware of any problem. I guess we knew that though.

Regarding loading: NUnit does not reload any tests unless you tell it to.
You should probably turn off any automatic reload settings if you have
them set. That way you can manually control when a reload takes place.

Charlie

[nickdu]

There were trace files created with trace information in them, just
none of it seemed to point to a reason for the issue I'm having.

I don't have any automatic reload setting turned on.

When I mentioned loading it was more about how NUnit loads a class and/
or method in order to execute it. It seems odd that if I select the
Lock class and run I get different behavior than when I choose each
Lock test method individually and run them. My guess is that NUnit is
loading the code differently in those two cases.

Thanks,
Nick

> ...
>
> read more »

[Charlie Poole]

Hi Nick,

On Mon, Apr 2, 2012 at 6:04 AM, nickdu <nic...@msn.com> wrote:
> There were trace files created with trace information in them, just
> none of it seemed to point to a reason for the issue I'm having.

Right - that's what I meant. NUnit is simply considering your
problem as a test failure, not an NUnit problem worthy of logging.

> I don't have any automatic reload setting turned on.

OK

> When I mentioned loading it was more about how NUnit loads a class and/
> or method in order to execute it.  It seems odd that if I select the
> Lock class and run I get different behavior than when I choose each
> Lock test method individually and run them.  My guess is that NUnit is
> loading the code differently in those two cases.

NUnit's "Loading" involves only your test assembly. It has to happen before
the tests can be displayed in the Gui. That loading is only repeated if you
use a reload item on the menu or if you have enabled automatic reload on
run or when the assembly changes.

Loading of dependent assemblies happens according to normal .NET rules,
as the code is jitted and executed. Again, once a dependent assembly is
loaded into the test AppDomain, it stays there.

So, your problem arises when the dependent assembly is first needed by
one of the Lock tests. If you have run some other test first, which successfully
loads the assembly, then the Lock tests have no problem. This is exactly
the sort of inter-test dependency I was originally writing about - not
necessarily
a dependency that you intentionally created, but one that exists nevertheless.

I guess that gives you two possible attacks on the problem:
1. Trace how and why the Lock class cannot load the assembly.
2. Trace how and why the complete class is able to load the assembly.

What's different in these classes?

Charlie

PS: Feel free to post some code if it doesn't expose any company secrets.

[nickdu]

Yeah, I'm thinking about spending a little bit of time to create a
skelaton of our framework that has the same dependent assemblies to
see if I can repro and if so send it along or post the code.

With respect to test dependencies, I don't think this could be the
case. As I said, I can start NUnit GUI fresh and run Lock Test5
without a problem, and therefore it's not dependent on running any
other tests. However, if I start NUnit GUI fresh and select the Lock
'set' of tests and run them then Test1 - Test4 succeed and Test5 -
Test8 succeed.

Nick

> ...
>
> read more »

[Charlie Poole]

That would help. I'd definitely like to see it.

Regarding dependencies, I'd say that if running tests 1-4 has an
effect on test 5, that's a dependency by definition. It's just one that
we don't yet understand and which is probably due to the underlying
platform - or NUnit itself - rather than something you coded explicitly.

Charlie

[nickdu]

I'm going to put some effort in today to get a simple repro of this.
I hope I'm successful. By the way, I tried 2.5.10 and the same issue
occurs so it's not specific to 2.6.

Thanks,
Nick

> ...
>
> read more »

[nickdu]

The first thing I did was to create a console application which
references the assembly that contains the tests and it calls the Lock
Test3 and Test5. These are minimum set of tests I would run from
NUnit GUI from choosing the Lock class that would fail. From my
console app they both ran fine. Here is my console application:

using System;
using Redbone.Framework.BusinessProcess.Tests;

public class Application
{
public static void Main()
{
try
{
Lock @lock = new Lock();
@lock.Test3();
@lock.Test5();
Console.WriteLine("Completed both tests successfully.");
}
catch(Exception e)
{
Console.WriteLine(e.ToString());
}
}
}

Now I'll try to come up with a minimum set of code which fails in
NUnit GUI which I could send out to others.

By the way, here is the details of the exception the debugger gave me
when the exception was thrown in the nunit-agent:

System.TypeInitializationException occurred
Message=The type initializer for '<Module>' threw an exception.
Source=Redbone.Framework.BusinessProcess
TypeName=<Module>
StackTrace:
at
Redbone.Framework.BusinessProcess.ProcessService.Start(XmlQualifiedName
name, String description, Version version, String step, String key,
IRow row)
at Redbone.Framework.BusinessProcess.Tests.Lock.Test5() in C:
\data\development\Redbone\bpf\Tests\Lock.cs:line 76
InnerException: System.TypeInitializationException
Message=The type initializer for '<Module>' threw an exception.
Source=ssoDotNet
TypeName=<Module>
StackTrace:
at
<CrtImplementationDetails>.ThrowModuleLoadException(String ,
Exception )
at
<CrtImplementationDetails>.LanguageSupport.Initialize(LanguageSupport* )
at .cctor()
InnerException: <CrtImplementationDetails>.ModuleLoadException
Message=The C++ module failed to load while attempting to
initialize the default appdomain.

Source=msvcm90
StackTrace:
at
<CrtImplementationDetails>.ThrowModuleLoadException(String
errorMessage, Exception innerException)
at
<CrtImplementationDetails>.LanguageSupport.Initialize(LanguageSupport* )
at .cctor()
InnerException:
System.Runtime.Serialization.SerializationException
Message=Unable to find assembly
'Redbone.Framework.BusinessProcess.Common, Version=4.0.0.0,
Culture=neutral, PublicKeyToken=e3fe5053ddfa671d'.
Source=mscorlib
StackTrace:
Server stack trace:
at
System.Runtime.Serialization.Formatters.Binary.BinaryAssemblyInfo.GetAssembly()
at
System.Runtime.Serialization.Formatters.Binary.ObjectReader.GetType(BinaryAssemblyInfo
assemblyInfo, String name)
at
System.Runtime.Serialization.Formatters.Binary.ObjectMap..ctor(String
objectName, String[] memberNames, BinaryTypeEnum[] binaryTypeEnumA,
Object[] typeInformationA, Int32[] memberAssemIds, ObjectReader
objectReader, Int32 objectId, BinaryAssemblyInfo assemblyInfo,
SizedArray assemIdToAssemblyTable)
at
System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadObjectWithMapTyped(BinaryObjectWithMapTyped
record)
at
System.Runtime.Serialization.Formatters.Binary.__BinaryParser.Run()
at
System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(HeaderHandler
handler, __BinaryParser serParser, Boolean fCheck, Boolean
isCrossAppDomain, IMethodCallMessage methodCallMessage)
at
System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream
serializationStream, HeaderHandler handler, Boolean fCheck, Boolean
isCrossAppDomain, IMethodCallMessage methodCallMessage)
at
System.Runtime.Remoting.Channels.CrossAppDomainSerializer.DeserializeObject(MemoryStream
stm)
at
System.Runtime.Remoting.Messaging.SmuggledMethodCallMessage.FixupForNewAppDomain()
at
System.Runtime.Remoting.Channels.CrossAppDomainSink.DoDispatch(Byte[]
reqStmBuff, SmuggledMethodCallMessage smuggledMcm,
SmuggledMethodReturnMessage& smuggledMrm)
at
System.Runtime.Remoting.Channels.CrossAppDomainSink.DoTransitionDispatchCallback(Object[]
args)
Exception rethrown at [0]:
at
System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage
reqMsg, IMessage retMsg)
at
System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData&
msgData, Int32 type)
at System.AppDomain.get_Id()
at
<CrtImplementationDetails>.DoCallBackInDefaultDomain(IntPtr function,
Void* cookie)
at
<CrtImplementationDetails>.DefaultDomain.Initialize()
at
<CrtImplementationDetails>.LanguageSupport.InitializeDefaultAppDomain(LanguageSupport* )
at
<CrtImplementationDetails>.LanguageSupport._Initialize(LanguageSupport* )
at
<CrtImplementationDetails>.LanguageSupport.Initialize(LanguageSupport* )
InnerException:

Our framework makes use of SSO, some managed TIBCO assemblies. The
SSO assemblies in turn have implicit links to unmanaged assemblies.
These assemblies are in a directory which are in the path. I had a
breakpoint on ProcessService.Start() but it never hit that. It shows
Start() at the top of the exception stack. My guess is that when the
call was attempted it started to jit the code for Start() and that's
where it ran into a problem.

Thanks,
Nick

> ...
>
> read more »

[nickdu]

I believe I found the cause of the error. I have a class defined in
the Common assembly (the assembly that the appdomain says it can't
load which is in the same directory as my NUnit test assembly) which
derives from ILogicalThreadAffinitive. I have this class deriving
from ILogicalThreadAffinitive so that the class can flow across
threads in a call chain. The problem is that in NUnit we have a
process with multiple appdomains with different application base
directories so that when a call flows from the appdomain that the
NUnit test assembly is loaded calls into the other appdomain which
doesn't have the application directory set to the directory where the
NUnit test assembly exists, it can't load the Common assembly.

Thanks,
Nick

> System.Runtime.Serialization.Formatters.Binary.BinaryAssemblyInfo.GetAssemb­ly()
>                       at
> System.Runtime.Serialization.Formatters.Binary.ObjectReader.GetType(BinaryA­ssemblyInfo

> assemblyInfo, String name)
>                       at
> System.Runtime.Serialization.Formatters.Binary.ObjectMap..ctor(String
> objectName, String[] memberNames, BinaryTypeEnum[] binaryTypeEnumA,
> Object[] typeInformationA, Int32[] memberAssemIds, ObjectReader
> objectReader, Int32 objectId, BinaryAssemblyInfo assemblyInfo,
> SizedArray assemIdToAssemblyTable)
>                       at

> System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadObjectWit­hMapTyped(BinaryObjectWithMapTyped
> record)
>                       at
> System.Runtime.Serialization.Formatters.Binary.__BinaryParser.Run()
>                       at
> System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(Hea­derHandler

> handler, __BinaryParser serParser, Boolean fCheck, Boolean
> isCrossAppDomain, IMethodCallMessage methodCallMessage)
>                       at

> System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(­Stream

> serializationStream, HeaderHandler handler, Boolean fCheck, Boolean
> isCrossAppDomain, IMethodCallMessage methodCallMessage)
>                       at

> System.Runtime.Remoting.Channels.CrossAppDomainSerializer.DeserializeObject­(MemoryStream
> stm)
>                       at
> System.Runtime.Remoting.Messaging.SmuggledMethodCallMessage.FixupForNewAppD­omain()

>                       at
> System.Runtime.Remoting.Channels.CrossAppDomainSink.DoDispatch(Byte[]
> reqStmBuff, SmuggledMethodCallMessage smuggledMcm,
> SmuggledMethodReturnMessage& smuggledMrm)
>                       at

> System.Runtime.Remoting.Channels.CrossAppDomainSink.DoTransitionDispatchCal­lback(Object[]

> args)
>                    Exception rethrown at [0]:
>                       at
> System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage
> reqMsg, IMessage retMsg)
>                       at
> System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData&
> msgData, Int32 type)
>                       at System.AppDomain.get_Id()
>                       at
> <CrtImplementationDetails>.DoCallBackInDefaultDomain(IntPtr function,
> Void* cookie)
>                       at
> <CrtImplementationDetails>.DefaultDomain.Initialize()
>                       at

> <CrtImplementationDetails>.LanguageSupport.InitializeDefaultAppDomain(Langu­ageSupport* )

> ...
>
> read more »

[Charlie Poole]

Hi Nick,

Sounds like you may have found an obscure NUnit bug.

Nunit actually has code that's supposed to prevent this from happening
and even some tests to make sure it doesn't happen. But it could be
that we have missed some particular situation.

All the calls back to the gui or console runner go through an event queue,
which is written by the test thread and read by a thread that is used
only to return events to the NUnit AppDomain. So, in theory, this should
prevent any part of the thread context from flowing back to the NUnit
application.

The only place where I know this can be violated is if an exception
is thrown, but maybe there are others.

If you have time and are able to create a simple repro of this problem,
I hope you'll submit a bug.

Charlie