Increased memory usage under NUnit3

[Chris Maddock]

I've just converted a large project from NUnit 2.5.7 to NUnit 3 - I'm running tests in the NUnit console.

One of the last issues I'm looking at is a big jump in memory requirements, and thus running time of our tests. 

A test suite which previous ran in ~13 mins, is now taking ~35mins, I believe due to memory limitations. (Our CI machine runs on 4GB RAM - running on a machine with more RAM available produces similar running time to previous.)

Is there anything I can do about this, or is the higher memory simply a requirement of nUnit3? (I'm aware the core of the problem may be our inefficient testing - but with 13,000 tests, I'm not looking to change that anytime soon!)

I'm investigating the --dispose-runners option, although I believe that is only at assembly level? All my tests are in a single dll, would splitting these out to multiple dll's help on any level? This is only single-threaded...for now.

[Charlie Poole]

Assuming it's actually a memory issue, --dispose-runners won't help.
If you are running tests in a single process, then there is only one
runner, in the sense that the word is intended, and it is disposed of
when the program ends. The option is primarily intended for situations
in which multiple assemblies are loaded into different agent
processes.

I suggest running some experiments to discover whether the problem is
really one of memory and if so where it occurs. I think you are
correct in thinking that the source of the problem is probably in the
tests themselves. You should be aware that a test may be leaking
memory at the time it is loaded as well as during its execution.
Simply not running a particular test will not prevent the leak if it
occurs at load time - the only way test for that is to remove the
test's code entirely as an experiment.

One thing you could try is running your tests under NUnitLite. Convert
the test assembly to an exe and give it a main as explained in our
docs. That avoids loading most of NUnit. If it results in a
significant improvement, it will tell us that the problem is at least
partly in NUnit itself.

Charlie

> --
> You received this message because you are subscribed to the Google Groups
> "NUnit-Discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to nunit-discus...@googlegroups.com.
> To post to this group, send email to nunit-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/nunit-discuss.
> For more options, visit https://groups.google.com/d/optout.

[Chris Maddock]

Thanks Charlie,

My tests load a number of files which I presumed might be the cause - I use [TestFixtureSetUp] a bit. Was wondering if the changes made there with [OneTimeSetUp] could have changed behaviour and caused problems...I don't believe I have any tear-down methods in place currently.

I'll investigate further and try nunit lite at some point over the next few days - may be in the new year at this point.

Merry Christmas - and thank you for the quick reply!

[Charlie Poole]

Hi Chris,

The change from TestFixtureSetUp to OneTimeSetUp is only a change of name.

My reference to memory use at load time refers to the point where
NUnit actually creates tests from your assembly, before any execution
(including SetUp) ever starts. Your own code may contribute to this if
you are using parameterized tests. Any methods that supply data to
generate the tests through TestCaseSource, TestFixtureSource,
ValueSource, etc. run at that point in time.

Charlie

On Mon, Dec 28, 2015 at 3:29 AM, Chris Maddock

[Chris Maddock]

Hi Charlie,

Have done some further digging, the issues appear to be caused by a change in the persistence of TestFixture classes. The tests in question load files, and thus have a lot of state, meaning out TestFixture objects are pretty substantial.

In our previous NUnit version (2.5.7) it appears that TestFixture classes went out of scope as the test suite ran:

Created 6045279
SetUp 6045279
Destroyed 6045279
Created 16914269
SetUp 16914269
Created 20203548
SetUp 20203548
Destroyed 16914269
Destroyed 20203548

Whereas in NUnit 3.0.1, it seems the TestFixtures are all preserved until the end of the run, and then destroyed:

Created 33163964

SetUp 33163964

Created 35567111

SetUp 35567111

Created 65066874

SetUp 65066874

Destroyed 65066874

Destroyed 35567111

Destroyed 33163964

This is something we can of course address at our end through [OneTimeTearDown] - but is it expected behaviour? 

Cheers - Chris.

[Charlie Poole]

Hi Chris,

It's a surprise to me and could be a bug.

Can you tell me how you instrumented the tests to get the Destroyed
info? I'm concerned that you may have done something that might
influence the result.

If that's not the case, it would indicate that we are hanging on to a
reference to the constructed user fixture, which is not what's
intended. We do maintain a reference to our internal class that wraps
your fixture, but the user class is constructed anew on each test run
- and of course in the console runner, there is only one test run per
instance.

Charlie

On Tue, Dec 29, 2015 at 10:00 AM, Chris Maddock

[Chris Maddock]

Hi Charlie,

This was the test class (copied three times over.)

[TestFixture]

    public class FakeFixture2

    {

        public FakeFixture2()

        {

            Debug.WriteLine($"Created {GetHashCode()}");

        }

        [SetUp]

        public void SetUp()

        {

            Debug.WriteLine($"SetUp {GetHashCode()}");

        }

        [Test]

        public void IsTrue()

        {

            Assert.IsTrue(true);

        }

        ~FakeFixture2()

        {

            Debug.WriteLine($"Destroyed {GetHashCode()}");

        }

[Charlie Poole]

Hi Chris,

I figured you had used a finalizer, which could conceivably interact
with NUnit in ways we don't anticipate. However, my guess is that you
didn't have a finalizer when the problem first showed up, so it's
unlikely to be the cause.

I'll see if I can duplicate this behavior myself.

Charlie

On Tue, Dec 29, 2015 at 10:56 AM, Chris Maddock

[Chris Maddock]

Hi Charlie,

Just to clarify, that's correct, I don't believe there are any finalizer's in use in out test code. Appreciate you looking into it.

Chris

[Chris Maddock]

Hi Charlie,

Just to follow-up, I made the changes to our test suite to nullify object references in a [OneTimeTearDown], and runtime halved. 15 mins as opposed to 35.

This will cover our case for the time being, please shout if I can provide any further debug information.

With thanks,

Chris

[Chris Maddock]

Hi Charlie,

Would it be possible to get some advice on how to possibly architect a fix to this issue? Internally, we've just had to start hosting our own butchered version of NUnit, as a couple of our tests suites were just eating up memory on our CI machine - but if we could get a proper fix in to the master version instead and go back to using the main packages, that would be much better!

I investigated the issue we talked about here further, and posted a summary of what I believe I've figured out on the GitHub issue below:

https://github.com/nunit/nunit/issues/1247

In terms of a proper fix - the issue seems to be that TestExecutionContext retains references. It's not immediately obvious to me as to why the Context is persisted for the entire duration of the program, could you clarify there? Putting that out of scope would seem like the nicest solution, although I presume it has other uses I haven't spotted at first glance. Otherwise, would the best solution then be to have a 'clear down' stage in the TestExecutionContext, once it has ran it's test? 

Something else I wasn't sure about is whether a clear up should happen in the OneTimeTearDown command, which seems to dispose of disposable objects, but not any others. Are other objects expected to go out of scope, or could there potentially be further calls to them later in run time?

                IDisposable disposable = context.TestObject as IDisposable;
                if (disposable != null && Test is IDisposableFixture)
                    disposable.Dispose();

Thanks for your on-going help with the issues we've encountered during the upgrade. When we got round to setting up our butchered version yesterday, we're now running a test suite that used to take 16-18 minutes in 4 minutes, so it's been completely worthwhile there!!

Cheers,

Chris