TestContext.CurrentContext.Result.Outcome is null when executing tests using nunit console

[David Karl]

A bit of background to this issue. We have been using Nunit with Appium to automate UI tests for an iPad app. The tests are written in dotnet using Visual Studio. The Appium tests include this TearDown:

[TearDown] public override void AfterEach() { if (TestContext.CurrentContext.Result.State != TestState.Success) { Screenshot screenshot = driver.GetScreenshot(); screenshot.SaveAsFile("Screenshots\\" + DateTime.Now.ToString("yyyyMMdd_HHmm_") + TestContext.CurrentContext.Test.Name + ".png", System.Drawing.Imaging.ImageFormat.Png); // Dismiss alerts Helpers.AcceptAlert(driver); IEnumerable<IWebElement> EditCowButtons; if ((EditCowButtons = driver.FindElements(InseminationScreen.EditCowButton)).Any()) { EditCowButtons.First().Click(); } } allPassed = allPassed && (TestContext.CurrentContext.Result.Outcome == NUnit.Framework.Interfaces.ResultState.Success); }

The tests were all working fine when invoked from Visual Studio, and we started attempting to have the tests run on checkin using our TFS build agent. We couldn't find a custom build task for TFS, so we set it up to invoke nunit-console to run the tests.

At this point the tests would run, but when the teardown was executed, it would hit a null reference exception.

After a bit of googling and debugging, we updated our Nunit packages to 3.0.1 and tried again.

The TestContext code needed to be updated, so we updated the TearDown code to:

[TearDown] public virtual void AfterEach() { if (TestContext.CurrentContext.Result.Outcome != NUnit.Framework.Interfaces.ResultState.Success) { Screenshot screenshot = driver.GetScreenshot(); screenshot.SaveAsFile("Screenshots\\" + DateTime.Now.ToString("yyyyMMdd_HHmm_") + TestContext.CurrentContext.Test.Name + ".png", System.Drawing.Imaging.ImageFormat.Png); // Dismiss alerts Helpers.AcceptAlert(driver); IEnumerable<IWebElement> EditCowButtons; if ((EditCowButtons = driver.FindElements(InseminationScreen.EditCowButton)).Any()) { EditCowButtons.First().Click(); } } allPassed = allPassed && (TestContext.CurrentContext.Result.Outcome == NUnit.Framework.Interfaces.ResultState.Success); }

Now the tests still fail with the same null reference when run from nunit-console, but now also the tests fail when invoked from Visual Studio as well. So we have gone backwards.

Is there anything obvious that we are doing wrong?

[Charlie Poole]

I notice you try to take a screenshot whenever the test result is not Success. I don't know your code but it seems likely that you would only have it available in case of a Failure result.

Outcome can have an unlimited set of values, with about eight common built-in ones.

--
You received this message because you are subscribed to the Google Groups "NUnit-Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nunit-discus...@googlegroups.com.
To post to this group, send email to nunit-...@googlegroups.com.
Visit this group at https://groups.google.com/group/nunit-discuss.
For more options, visit https://groups.google.com/d/optout.

[David Karl]

That's right. The idea is that if the test failed, I want to see what was on the screen at the point of failure. However, TestContext.CurrentContext.Result.Outcome != NUnit.Framework.Interfaces.ResultState.Success causes a null reference exception. Is there another way to determine if the test failed?

[Charlie Poole]

Well `TestContext.CurrentContext.Result.Outcome == NUnit.Framework.Interfaces.ResultState.Failure` might be a better first attempt. :-)

My point was that "not succeeding" can mean many other things besides failing. Maybe the user cancelled, maybe some unexpected (therefore untested) exception was thrown. Maybe the test was ignored or inconclusive on the fly. This is just a special case of the general rule that you don't rely on unknown states. Your negative test is succeeding for about half a dozen other results besides failure.

I can't tell you exactly what to test for, because I don't know your application, but you should only try to get the screenshot with a result for which it is produced. You'll have to analyze your code against the possible results and decide which ones to try.

This may not be the full answer, but it will be much easier to help you if you can say "With such and such a result state, I get this error."

[David Karl]

Yes, that is a fair observation; anything other than success will result in taking a screenshot. However, this is exactly the desired behaviour and isn't the cause of the problem. The problem is that TestContext.CurrentContext.Result.Outcome is null, which causes an exception. The entire TestContext object is null.

[Charlie Poole]

OK, you have not said that before - just that there was a null reference exception.

FWIW, however, I think you will eventually find that you __don't__ really want a screenshot in all cases where Outcome is not Success, so keep that in mind. For example, you may have an Error result state (different from Failure) where it isn't possible to extract a screenshot.

However, back to the problem... when you say "The entire TestContext object is null" what do you mean? Is CurrentContext null? is Result null?

Where is your TearDown located wrt the test class. Is it in it or in a base class?

What build of the framework are you using? (.Net 4.5, Portable, etc.)

I'm assuming we are continuing the discussion in the context of NUnit 3. Since that is a completely new program to NUnit V2, it will be confusing to try to analyze both at the same time.

What would be helpful, if you can create it, would be a simple test case - nothing to do with screenshots - that demonstrates that the context (or some part) of it is null. We cold work that as a bug.

FYI, tests of the context in NUnit are here: https://github.com/nunit/nunit/blob/master/src/NUnitFramework/tests/TestContextTests.cs Can you see anything substantially different about what you are doing as compared to what we test?

Charlie

[David Karl]

Sorry about the lack of clarity in my initial post. You'll have to bear with me; I'm not a developer, I'm a tester with a tiny bit of coding knowledge, so if I appear to be missing important details, let me know :)

when you say "The entire TestContext object is null" what do you mean? Is CurrentContext null? is Result null?

I'm not entirely sure but I think that shows that the TestContext is null? If not, let me know what to look for.

 

Where is your TearDown located wrt the test class. Is it in it or in a base class?

It is in a base class that is inherited by the various test classes. 

What build of the framework are you using? (.Net 4.5, Portable, etc.)

I'm hoping one of these screenshots helps answer this:

 

I'm assuming we are continuing the discussion in the context of NUnit 3. Since that is a completely new program to NUnit V2, it will be confusing to try to analyze both at the same time.

Yes, we have NUint 3 installed now. 

What would be helpful, if you can create it, would be a simple test case - nothing to do with screenshots - that demonstrates that the context (or some part) of it is null. We cold work that as a bug.

I'll have a go at this and post my findings shortly.

FYI, tests of the context in NUnit are here: https://github.com/nunit/nunit/blob/master/src/NUnitFramework/tests/TestContextTests.cs Can you see anything substantially different about what you are doing as compared to what we test?

Nothing glaringly obvious. 

[David Karl]

What would be helpful, if you can create it, would be a simple test case - nothing to do with screenshots - that demonstrates that the context (or some part) of it is null. We cold work that as a bug.

using NUnit.Framework;

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using System.Threading.Tasks;

namespace LIC.Datamate.AppiumTests

{

    [TestFixture()]

    public class X_TestContextTests

    {

        [Test]

        public void TestCanAccessItsOwnName()

        {

            Assert.That(TestContext.CurrentContext.Test.Name, Is.EqualTo("TestCanAccessItsOwnName"));

        }

    }

} 

When I execute this test, it fails with: 

Message: System.NullReferenceException : Object reference not set to an instance of an object.

[Charlie Poole]

OK, it appears that you are targetting .NET 4.5, so you should be referencing the .NET 4.5 nunit framework. You should check that by looking at where you got the reference form - see the full path to framework in the reference properties.

The only reason I can imagine for TestContext not to be set up properly is if you are running the test using some third-party runner that doesn't properly emulate NUnit execution.

That's not to say there aren't any bugs that I can't imagine. :-(

[Charlie Poole]

Your test doesn't use SetUp, but the fact that you get a null
reference in the test itself is equally surprising.

And of course it's almost identical to one of our own tests, which
passes for us every time we build.

How are you runnng the test? (Seeing the failure under nunit3-console
would be the most convincing.)

You might add a couple of more asserts before the one you have:
```C#
Assert.NotNull(TestContext.CurrentContext);
Assert.NotNull(TestContext.CurrentContext.Test);
```

Then we can see which line number fails.

Charlie

[David Karl]

Hmmm, curiously, both of those tests pass.

[David Karl]

And both of these get nullref exceptions:

        [Test]

        public void TestNameIsNotNull()

        {

            Assert.NotNull(TestContext.CurrentContext.Test.Name);

        }

        [Test]

        public void TestNameIsNull()

        {

            Assert.Null(TestContext.CurrentContext.Test.Name);

        }

This article seems to be describing the same problem, where one person added a workaround in the NUnit source code.

[Charlie Poole]

You haven't yet told me how you run your tests. This is key. If you
are using R#, for example, I don't want to talk to you. :-)

Regarding the article, it refers to NUnit 2.5.9. During the period
from 2.5.7 to 2.5.10, there were a number of bugs fixed regarding
TestContext. Now, NUnit 3.0 has an entirely different implementation
with not much of the old code left.

Do you have a stack trace for your exceptions?

[David Karl]

System.NullReferenceException was unhandled by user code

  HResult=-2147467261

  Message=Object reference not set to an instance of an object.

  Source=nunit.framework

  StackTrace:

       at NUnit.Framework.TestContext.TestAdapter.get_Name()

       at LIC.Datamate.AppiumTests.X_TestContextTests.TestNameIsNotNull() in e:\Projects\Source\Repos\LIC.Datamate\LIC.Datamate.AppiumTests\X_TestContextTests.cs:line 28

  InnerException: 

The tests are written in C# dotnet. (sorry if that doesn't answer the question).

Do you think maybe there are some artifacts from 2.5 lingering in the project?

[Charlie Poole]

The stack trace indicates that there was no initialization of
CurrentTest in the TestExecutionContext, possibly even that there is
no TestExecutionContext.

That gets me back to my question "How do you run the tests?" Make
believe I'm a new tester. You want me to run the tests in some
assembly. I ask you "How do I do that?" Tell me what steps to perform.
:-)

Charlie

[David Karl]

Okay, I'll do my best, but if I'm not cutting it, I can get my developer colleague to tag in and use some real words.

1. Open Visual Studio.

2. Find the project repository in Git and clone it.

3. Open the test project and find the class containing the test you want to run.

4. Right click on the test name and click run test.

OR. Alternatively, find the test in the Test Explorer window and run it from there.

[Charlie Poole]

OK, At step 4, It sounds as if you run the tests using Resharper,
TD.Net or some other 3rd party runner. Perhaps you know which one. The
alternative, is that you run it using the NUnit test adapter.
Hopefully, that's the NUnit test adapter 3.0.8 for NUnit 3.0.1, but if
it's showing the tests then you're good.

Basically, we try to support 3rd party runners but we have no control
over how they are written or used. Even the NUnit adapter is a
separate project from NUnit, with different people working on it, and
we need to figure out whether any issues are caused by the adapter or
by NUnit. As a tester, this should sound familiar to you. :-)

For all those reasons, I'd like you to run your tests using the nunit
console runner... nunit3-console.exe. That way, lots of variables are
removed. If it works in the console runner, then the problem may be in
some other runner you are using. If it doesn't work in the console
runner, then it could be some fundamental NUnit bug or it could be
something unusual that we have not yet seen in your tests.

Charlie

[David Karl]

Thanks Charlie. We are using the NUnit test adapter 3.0.8, and we were also getting the nullref exceptions when running the tests from the build agent (a Mac) using nunit3-console.

I'm right now running the tests from my machine using nunit3-console and I'll let you know the results.

[David Karl]

More information: when we manually run the tests using nunit3-console on both my PC and the build agent Mac, they run fine and report passing tests.

I guess it's a problem with the VS adapter then? Which might also be related to seeing two of every test in the Test Explorer?

[Charlie Poole]

Two of every test is a relatively new problem that seems to occur in
some environments under VS2015. It means you have the adapter
installed twice, once as a VS extension and once as a nuget package.
In past releases, this did not cause a problem.

Verify that the version of the tests that you run under the console
and that run under the adapter both use the same version of the nunit
framework.

[David Karl]

NUnit Console Runner 3.0.5813

Copyright (C) 2015 Charlie Poole

Runtime Environment

   OS Version: Microsoft Windows NT 6.3.9600.0

  CLR Version: 4.0.30319.34209

Test Files

    E:\Projects\Source\Repos\LIC.Datamate\LIC.Datamate.AppiumTests\bin\Release\L

IC.Datamate.AppiumTests.dll

[David Karl]

Got it! Yes, I did have VS extensions for NUnit installed. When I disabled them, the duplicate of every test disappeared, and now when I run the tests, TestContext works fine.

Thank you very much for helping me to debug this situation!

[Charlie Poole]

Glad you got it figured out.