Random Failure

[Yann Duran]

Hi Anthony & David,

 

I'm getting some weird random failures in some tests that I've written that are using NSubstitute. Here's an example of the error:

 

Test 'TeamCentral.UnitTests.CurrentShiftSinglePresenterTests.SetupTabs_CantFillCancelCommand_VisibleReturnsFalse' failed:

System.InvalidOperationException : Stack empty.

at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource)

at System.Collections.Generic.Stack`1.Pop()

at NSubstitute.Core.CallStack.Pop()

at NSubstitute.Core.ResultSetter.SetResultForLastCall(IReturn valueToReturn, MatchArgs matchArgs)

at NSubstitute.Core.CallRouter.LastCallShouldReturn(IReturn returnValue, MatchArgs matchArgs)

at NSubstitute.Core.SubstitutionContext.LastCallShouldReturn(IReturn value, MatchArgs matchArgs)

at NSubstitute.SubstituteExtensions.Returns[T](MatchArgs matchArgs, T returnThis, T[] returnThese)

at NSubstitute.SubstituteExtensions.Returns[T](T value, T returnThis, T[] returnThese)

Presentation\CurrentShiftSinglePresenterTests.vb(145,0): at TeamCentral.UnitTests.CurrentShiftSinglePresenterTests.Setup()

825 passed, 1 failed, 0 skipped, took 43.12 seconds (NUnit 2.5.5).

It's always a "stack empty" error, but it can be for very different tests. For example as well, I ran the tests again immediately after this failure (making no changes anywhere) & there was no error. It happens very randomly & not for the same tests any time there's a failure.

 

So, wondering if I'm maybe doing something wrong, or if this is some kind of bug, or even if it's something to do with VB.NET, which is the language I use.

 

Yann

 

[David Tchepak]

Hi Yann,

Are you subbing for classes? If you try and use Returns() on a non-virtual/abstract member you can get this error (because NSub can't intercept and record those calls, it won't be able to find them on its call stack). 

That won't explain the randomness of it though. You're not running the tests in parallel are you?

Is it possible to send through some of the code? If it is something you need to keep off-list then email "me at davesquared.net". Thanks.

Regards,

David

--
You received this message because you are subscribed to the Google Groups "NSubstitute" group.
To post to this group, send email to nsubs...@googlegroups.com.
To unsubscribe from this group, send email to nsubstitute...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/nsubstitute?hl=en.

[Yann Duran]

Hi David,

 

Hmm, I wasn't aware of the necessity of "Returns" being used only on overridable classes, but I guess that makes sense when I think about it. In that case, why are my test working at all then? Most of the time they work just fine. And it's not the same tests that seem to randomly fail.

 

Nope, no parallel tests.

 

What code would help? The way I'm using NSubstitute in my setup methods, the interfaces, or the code of the classes that it's substituting for?

 

Thanks for the quick reply. It's nothing urgent, it's only a minor concern (more of a mystery really), I just thought you might like to be aware of it, if it wasn't something I was doing wrong.

 

Yann

[David Tchepak]

Not on overridable _classes_, on overridable _members_ (i.e. interface members, or virtual/abstract properties and methods on classes).

If NSub's breaking in random ways then it's urgent to me. :) We have a couple of thousand tests here and haven't run into it, but it's definitely worth looking into if you're getting a stack trace like that randomly.

If you can send me CurrentShiftSinglePresenterTests, the presenter itself and any related interfaces it references that would be great.

Also what test runner are you using? 

[Yann Duran]

Hi David,

 

Yes, I meant "members", not "classes", sorry.

 

The code you mention would actually amount to quite a lot of code, there's quite a bit of inheritance involved. I've built a reasonably large framework that supports that presenter etc.

 

For example, that presenter inherits from CurrentShiftPresenter, which inherits from ParentViewPresenter, which inherits from EntityViewPresenter, which inherits from DataViewPresenter, which inherits from BaseViewPresenter. And that's just the presenter, several of the objects being substituted also have a similar pattern.

 

So I'm not really sure its that practical, sorry.

 

And I'm using TestDriven.NET as my test runner.

[David Tchepak]

Understood. A few things you can try (assuming you have time to look into it):

1. Try running tests from NUnit gui or via the console runner. If you can script this to run it a few times and see if you can get an error that might be interesting.
2. See if you can reproduce that single stack trace. Does it ever happen if just running that one fixture, or only for all fixtures? Is there some static state somewhere that makes calls dependent on the order in which fixtures are run? (IIRC some test runners use different orders on different runs.)
3. Grab the NSubstitute source and debug, try and work out how that line could fail. When you call "sub.SomeCall().Returns(x)", all NSub does is record the "SomeCall()", then when it runs the "Returns(x)" call it looks up the last call and sets the substitute to return "x" for subsequent calls. If there is no last call registered you'll get that error. Check for times when the substitute may be called between SomeCall() and the Returns(). Check for re-entrant calls (i.e. calling back into the substitute from within SomeCall(), from Returns()). 
4. If it is mainly failing on calls to substituted delegates that would also be interesting to know (goes through a slightly different code path)
5. Not sure if it will help, but if you can send me the fixture and the single presenter (no parent classes) then I'll see if I can narrow down where it is happening.
6. We could do some screen-sharing via Skype and see if I can help debug it that way.

Other than that I can't think of any reason why your substitute is being called between the "SomeCall()" and the "Returns(x)". Maybe there's some crazy optimising where a call gets JITed out as unused??!?  (way out of my league on that one)

If you get any leads on this let me know; would love to track down why it's happening.

Cheers,

David